The Journey Toward 24/7 IT Monitoring University of North Carolina at Greensboro Design and Build of Network Operations Center Copyright Thomas M. Sheriff, This work is the intellectual property of the author. Permission is granted for this material to be shared for non- commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

The 2004 Enrollment 11,497 undergraduate students. 3,217 graduate students. 14, 714 total headcount. Largest freshman class (2,158) Residence halls at capacity! (approx. 3,800) Approx employees (Faculty/Staff)

The Data Network End to end Cisco network (IP only) 700 network switches Approx. 25,000 ports. 50 buildings connected to the campus network

The Server Farm Two Machine rooms –9 VMS servers –63 UNIX servers –30 Netware servers –71 Windows servers

The Enterprise Applications Banner Student, Alumni, Financial Aid SCT FRS and HRS Lotus Domino Blackboard Pipeline Novell file and print services Windows based web services

The Motivation Be ahead of the client –Clients often notified IT of some outage Collect monitoring knowledge in single location –Monitoring being carried out in isolated IT enclaves Clearing house for enterprise events –Response to events differ throughout IT

The Goal Improve customer service and communication Encourage a team approach to event management Delivery of concise information to management

The Method Develop a project within the IT Project Management model Three phase approach

The 3 Phase Approach Specifications –Define the physical characteristics of the NOC Design –Draw upon Physical Plant and Vendors for design development Build –Facilitate renovation –Facilitate infrastructure installation –Bring facility to operational state

The Process Weekly meetings with project team –Developed NOC specifications document –Developed monitoring priorities Planning sessions with Physical Plant Visits from vendors Visits to working NOC’s

The Obstacles Other projects gaining priority Lack of funds Staffing challenges Marginally acceptable physical space Renovation costs

The Progress Improved customer service Improved service response time Centralized IT troubleshooting location Monitoring with low cost/no cost tools Monitoring centralized Low cost functional NOC

The Hardware 18 Dell SX280 2 Belkin KVM switches 2 Samsung giant monitors 8 Dell GX models 3 Sun workstations 6 EDP console units –Rack cubes –Phone mounts 10 Cisco switches –All VLAN’s present –Out of band networks

The Software What’s Up Gold –140 servers monitored ICMP (ping), DNS, NTP, HTTP, FTP, TELNET Custom services defined as needed –700 switches Maps automatically generated and imported via XML –MAN and WAN routers monitored MRTG – trend analysis (daily/weekly/monthly) –Critical traffic paths –Router CPU usage –Domino connection counts –Environmental conditions APC Enterprise - Switch closet UPS monitoring

The Lessons Learned A NOC can be built without breaking the budget Adequate staffing resources should be identified in advance Pooling monitoring tools requires tact A completed NOC will take years

The Road Ahead Design of NOC for renovated building Incorporating enterprise monitoring tools Providing enterprise management tools CA or BMC enterprise tools?

The following slides contain photographs of the Network Operations Center at UNCG.

The following slides are screen images of MRTG and What’s Up Gold as used at the UNCG NOC.