Communications Infrastructure Committee Update for LAN Managers Friday, May 27, 2011

Initial Building List  OT – Done* Clark - Done* Shepardson - Done* ARBL - Done A/Z - Done NESB – In process Engineering VTH Atmospheric Science – * Single GE for now April 15, 2011IAC Update2

10G Connectivity Proposals One upgrade in process, as part of a funded NSF proposal Evaluating 5 additional requests for high- speed connectivity Goal is to make a funding decision very soon April 15, 2011IAC Update3

Operational Management Discussion Issues and Difficulties Responding to problems/outages Typical examples of problems Edict from VPIT Open Discussion April 15, 2011IAC Update4

Issues and Difficulties Switch Consistency (Type and Configuration) Documentation (Change mgmt., ports, vlans) Security (Physical and Virtual) Resources – Hardware inventory – People – delegation of responsibilities (configurations, responding to outages, etc.) Multi-departmental occupancy of buildings New technologies – Radiation Safety Project – VoIP – exacerbated by all of the above April 15, 2011IAC Update5

Issues and Difficulties (Cont’d) 900+ switches 120+ MDFs/IDFs 40,000+ systems Video, Environmental Controls, Power Meters, Voice, Police/Fire, Security Panels, Servers, Wireless, Private College LANs, etc… Main campus, West Campus, South Campus, Remote sites comprising – 213 Subnets – 60 Subnet managers

Response Issues w/ Current Model Typically, 90% of time (2-4 hours, average) is spent figuring out what changed, then a quick fix (put it back the way it was) 10% of the time, problem is evident but very difficult to find the source and/or remedy “Flying blind” is very difficult We need to develop and maintain accurate cabling records – E911, CALEA Act Compliance, etc. April 15, 2011IAC Update7

Small sample and examples of concerns Environmental controls offline due to multicast storm, After hours response by Facilities, April 2011 Looped network takes down environmental controls requiring reconfiguration of switches and further monitoring after hours by Facilities, NOC, April 2011 Firewall not configured to be on network and thus not firewalling, March, 2011 Several building networks down along with environmental controls due to dual- homed server misbehaving. Discovered hubs and other devices as part of debug process. March, 2011 VOIP and Wireless down due to cables being moved to wrong switch ports. March, 2011 Compromised machines swamp entire LAN affecting police/fire, March 2011 Departmental servers down, nobody knew location of switches affecting servers nor configuration of switches thus requiring NOC to respond, find, fix, March, 2011 Unit carrying police/fire notifications moved to wrong port on switch. After hours response by NOC, Facilities, Feb Lost communication on large campus network due to switches being plugged into each other incorrectly. Feb Network device with duplicate IP takes down a College’s central server, Feb 2011 Emergency fiber run had to be made to bypass departmental network device incapable of handling vlans for cardkey access, Feb, 2011

Directive, from VPIT Must ‘harden’ our network for future applications, including VoIP April 15, 2011IAC Update9

Discussion Operational management of switches is becoming more complex, and more critical – QoS, 802.1X, VoIP, documentation, life & safety, etc. Need to “harden” the network, both physically and operationally How can we do this, improving integrity of the network without impeding day-to-day functionality our users require? Virtual vs. Physical Demarc April 15, 2011IAC Update10