8/20/2015How to Build a NOC Pacific NorthWest GigaPop November 29, 2007 How to Build a NOC.

Slides:



Advertisements
Similar presentations
Steve Lewis J.D. Edwards & Company
Advertisements

Top-Down Network Design Chapter Nine Developing Network Management Strategies Copyright 2010 Cisco Press & Priscilla Oppenheimer.
Information Technology Disaster Recovery Awareness Program.
Common NOC Practices 4/05/2007 The Quilt NOC Common Practice Panel April 4, 2007.
OVERVIEW TEAM5 SOFTWARE The TEAM5 software manages personnel and test data for personal ESD grounding devices. Test and personnel data may be viewed/reported.
Copyright 2007 CENIC and PNWGP TransitRail: Nationwide Commodity Peering Program February 12, 2007 Minneapolis, MN.
End to End Uptime Monitoring
5/15/2015How to Build a NOC Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007 How to.
IT Technical Support Policies and Procedures South Nottingham College.
Documenting the Existing Network - Starting Points IACT 418 IACT 918 Corporate Network Planning.
Facilities Management and Planning Horace Bomar, Director November 1, 2011.
Wireless Directions University of California, Davis Wireless Technology Team February, 2001.
Downtown Seattle Transit Tunnel (DSTT)
© 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac.
Chapter 13 Network Management and Operations. Agenda Objectives Scope Functions Software Security Physical Facility Staffing.
GLOCO – Integrated Corporate Portal Final Presentation Presented by Team 3 1 Team 3 Members: Joyce Torres Kenneth Kittredge Pamela Fisher Ruzhena Saltisky.
1 Disaster Recovery Planning & Cross-Border Backup of Data among AMEDA Members Vipin Mahabirsingh Managing Director, CDS Mauritius For Workgroup on Cross-Border.
Data Centers and IP PBXs LAN Structures Private Clouds IP PBX Architecture IP PBX Hosting.
1 Customer Network Operations Center. 2 Agenda Overview Benefits Value Additional Services Questions Monitoring Options.
Service Blueprint: conferencing and collaboration Web/KioskService Point/Call CenterPresentation SpacesGroup Studies Teaching Spaces (T&V, Creativity,
What is Business Analysis Planning & Monitoring?
Introduction to Information and Computer Science Networks Lecture a This material (Comp4_Unit7a) was developed by Oregon Health and Science University,
Connect. Communicate. Collaborate Place your organisation logo in this area End-to-End Coordination Unit Toby Rodwell, Network Engineer, DANTE TNLC, 28.
Organizing Information Technology Resources
Circuit Services - IPTV Christian Todorov Internet2 Fall Member Meeting October 9, 2007.
Working with the LiveOps CSC (Customer Support Center)
Current Job Components Information Technology Department Network Systems Administration Telecommunications Database Design and Administration.
Working with the LiveOps Help Desk
Requirements Review – July 21, Requirements for CMS Patricia McBride July 21, 2005.
SCSC 311 Information Systems: hardware and software.
So You Have Assigned the SLA, Now What? Dave McGaugh Network Architecture, PNWGP Quilt Fiber Workshop - 13/Oct/2006 St Louis, MO.
Next Generation Peering for Next Generation Networks Jacqueline Brown Executive Director International Partnerships Pacific Northwest Gigapop CANS2004,
Youth Ambassadors & Site Liaison Training. How does the partnership between my site and Connecting Generations work? The AmeriCorps member is the bridge.
Steve Peck Service Desk Manager Global Research NOC At Indiana University.
Technical Policy and Standards Andy Gorton – Senior Architect: Institutional Networks.
Existing Solution 1 Proactive – SLR 6 hour schedule activities for managed services Stop light report is posted and reviews any open tickets for.
Unit 8a Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Component 8 Installation and Maintenance of Health IT.
WebEx Cloud Connected Audio Enterprise
Session 7 - Maintenance - contract and day-to-day Maintenance Support Presenter  Grenville Powell (Managing Director - of Shokaz Integrated Computing.
Connect. Communicate. Collaborate Place your organisation logo in this area The PERT – Evolution from a Centralised to a Federated Organization Toby Rodwell.
Business Data Communications, Fourth Edition Chapter 11: Network Management.
© 2006 Cisco Systems, Inc. All rights reserved.Cisco Public 1 Version 4.0 Gathering Network Requirements Designing and Supporting Computer Networks – Chapter.
Based on work by DoIT Network Services, UW-Madison The Network and the Role of Tools January 6, 2006 Ron Kraemer, Deputy CIO.
Terri Lahey Control System Cyber-Security Workshop October 14, SLAC Controls Security Overview Introduction SLAC has multiple.
Oncology Patient Enrollment Network OPEN Support Services Lucille Patrichuk OPEN Implementation Manager OPEN Conference September 18, 2008.
Cctv Management Solution - CMS From – Intelitech Solutions Pvt. Limited. [ ITSPL Group ]
1 Local Readiness Team Lead Meeting June 6, 2007.
Adoption and Use of Electronic Medical Records (in Federally Qualified Health Centers) and Supporting an ASP Community Care Network of Virginia, Inc.
Working with the LiveOps Help Desk
GLOCO – Integrated Corporate Portal Final Presentation Presented by Team 3 1 Team 3 Members: Joyce Torres Kenneth Kittredge Pamela Fisher Ruzhena Saltisky.
Information Technology Services Strategic Directions Approach and Proposal “Charting Our Course”
How to Build a NOC. Identify Customers –Who are your customers? Understand Customer Expectations –What are your user expectations? –SLA’s? Support Service.
Mount Auburn Hospital Application Support Transition MIS Committee Overview October 14, 2015 Bob Todd.
Component 4: Introduction to Information and Computer Science Unit 7: Networks & Networking Lecture 1 This material was developed by Oregon Health & Science.
1 Network Quarantine At Cornell University Steve Schuster Director, Information Security Office.
Staff Assessment Technology Services Department Palmyra Area School District.
A Service-Based SLA Model HEPIX -- CERN May 6, 2008 Tony Chan -- BNL.
Griffith University Kevin Grant Manager, Database Management Services Information Services.
Installation and Maintenance of Health IT Systems Unit 8a Troubleshooting; Maintenance and Upgrades; and Interaction with Vendors, Developers, and Users.
Information Technology Division Customer Service Support Center.
© 2014 Level 3 Communications, LLC. All Rights Reserved. Proprietary and Confidential. Simple, End-to-End Performance Management Application Performance.
 How to Reach Us  Service Support Levels  Support Definitions & Standards  Response Levels  Point of Contact  Our Escalation Processes  Expect a.
Anytime, Anywhere Access Benefits Functionality Work Order Administration Dispatch Work Order Work Order Details New Work Order Additional Functionality.
Copyright 2007 CENIC and PNWGP
«My future profession»
State Coordinator Intervention
Momi Ford, Support Center Manager Indiana University
Copyright 2007 CENIC and PNWGP
Relate to Clients on a business level
Presentation transcript:

8/20/2015How to Build a NOC Pacific NorthWest GigaPop November 29, 2007 How to Build a NOC

9/11/2007How to Build a NOC Key Elements to Building a NOC Identify Customers –Who are your customers? Understand Customer Expectations –What are your user expectations? –SLA’s? Support Service Offerings –Besides networking, what other services are being offered?

9/11/2007How to Build a NOC Key Elements to Building a NOC Monitor and Troubleshoot Service Issues –How large and complex is your network –What level troubleshooting and/or monitoring will your NOC do? –How will you communicate outages and planned work to customers?

9/11/2007How to Build a NOC Determine Appropriate Staffing Levels –What will be the service hours? Not all NOCs need to be 7x24x365 What about holidays? Weekends? On-call? –Do your SLA's require in-person staffing? –Do you have after hours service response requirements? –What level of staff needs to be present, and when? –What will be the means of responding to issues when NOC is not staffed 24x7? Key Elements to Building a NOC

9/11/2007How to Build a NOC Organizational Structure –What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs? –What will be your escalation practices and policies? –To what group does the NOC report? –What other groups report there, and what is your organizational relationship to other key groups? –Who will write and update procedures, training manuals, etc.? Key Elements to Building a NOC

9/11/2007How to Build a NOC Location and Design of NOC Facility –How much space do you need? –What is your facility like? –How do you want to arrange your staff? Separate offices? "War room"? NOC Funding –How will your organization be funded? Key Elements to Building a NOC

9/11/2007How to Build a NOC NOC Tools –How will you track customer information? (Database needs, CRM?) –How will you monitor and troubleshoot? Tools, specifically. – Are you writing any of your own tools? – Who will maintain your applications? – How will you track trouble tickets? Key Elements to Building a NOC

9/11/2007How to Build a NOC Reporting –What reports will you issue? –How will you measure the data? Key Elements to Building a NOC

9/11/2007How to Build a NOC NOC Evolution What factors may determine operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level

8/20/2015How to Build a NOC October 3, 2007 Building a NOC Pacific NorthWest GigaPop Case Study

9/11/2007How to Build a NOC Customers and Expectations Our customers are in the Pacific Northwest and Pacific Rim. We track customer information in a database designed and maintained locally. Time zones have driven our need for 7x24. We cover 15 time zones which also makes scheduling planned maintenance difficult.

9/11/2007How to Build a NOC Customers and Expectations Escalation policies also drive our need for on-call schedules and on-site personnel. Our customers expect 48 hours notice prior to work, unless it is an emergency. Outages are communicated via an application we have built locally. We also want to know when our customers have planned work.

9/11/2007How to Build a NOC Customers and Expectations Expectations are included in the contract and located on the PNWGP web page. We prefer to have the NOC as the primary customer contact point for our organization in order to maintain quality of service when a ticket moves between groups.

9/11/2007How to Build a NOC Supported Services In addition to network monitoring for PNWGP, our NOC monitors: UW campus connectivity, including two hospitals (both layer 2 and layer 3, and wired as well as wireless, many UPSs), Approximately 500 sites throughout WA state for the K-20 network, Pacific Wave, and Transit Rail.

9/11/2007How to Build a NOC Supported Services We do not do any host, server, or customer application monitoring. We do not do desktop support. We do not reset passwords or arrange services such as and web. We do not monitor data center security alarms such as for fire or flood.

9/11/2007How to Build a NOC Monitoring and Troubleshooting  PNWGP: approx. 20 customers, 5 node sites, MPLS, BGP, VPN  Campus: 115,000 connected devices, ~100 remote locations, ~7,000 wireless APs, ~4,000 layer 2 switches, ~150 routers.

9/11/2007How to Build a NOC Monitoring and Troubleshooting  Over 450 Washington state K-20 sites monitored.  Transit Rail, approx. 5 node sites,BGP and ISIS status.

9/11/2007How to Build a NOC Monitoring and Troubleshooting  PNWGP troubleshoots with the customer on routing problems, latency, and loss of connectivity.  IS-IS and IGP status is monitored.  We manage DWDM, Sonet, and MPLS circuits.  Complexity is increased with escalation paths being different depending on what isn’t working.

9/11/2007How to Build a NOC Monitoring and Troubleshooting  Outages and planned events are sent via announcement in a standard format.  We include the date/time of the work or when the outage began.  If the customer’s connectivity is entirely down, we also call or page them.  Updates are sent at predefined intervals for large events, or when we have a change in status.

9/11/2007How to Build a NOC Staffing We are 7x24 with full-time staff. Weekends we only have one person covering each day, so vacations and sick time are problems. Holidays are covered by on-site and on- call staff. On-call consists of a 7-day period and rotates among all NOC staff on a regular schedule.

9/11/2007How to Build a NOC Staffing Tier 1 are student staff, also called Network Analysts. Tier 2 are full-time staff, most are titled Network Specialists. Tier 3 are full-time staff, titled Network Engineers.

9/11/2007How to Build a NOC Staffing We advertise to all customers an on-site 7x24 staff for immediate response to outages. Our SLAs indicate when a problem should be escalated beyond the oncall staff to a manager, director, and higher at any time of the day or week.

9/11/2007How to Build a NOC Staffing Day shift M-F requires that we have engineers (Tier 3) staff in the building. Off-hours we have Tier 2 staff always present in the building. They will escalate as necessary, either due to outage severity or complexity, or at a customer’s request.

9/11/2007How to Build a NOC Staffing To maximize staffing efficiencies, 2 nd and 3 rd shift personnel report to Computer Operations managers, rather than the NOC manager. These staff provide all of the same support and have the same training/access as the daytime Tier 2 staff in the NOC.

9/11/2007How to Build a NOC Staffing We have on-call lists with a primary and secondary person for backup when NOC staff is not on-site. A separate call list exists for escalation to managers. Engineering and other service groups are also available on-call 7x24x365.

9/11/2007How to Build a NOC Organizational Structure PNWGP has recently adopted a tiered approach for staffing. NOC Network Engineers are available after 6pm M-F and weekends only via on- call. One manager, reporting to the Director of Operations.

9/11/2007How to Build a NOC Organizational Structure Our escalation practices and policies are based on length or severity of outage. At predetermined intervals additional management levels are notified of severe outages in order to help with escalation at other organizations (telcos), or to keep peers updated at the affected sites.

9/11/2007How to Build a NOC Organizational Structure Many people in the NOC work to provide and update training materials; SMEs write procedures. We use a wiki to maintain our documentation. We were able to have a tech writing expert give us training on writing effective procedural documents and re-organize our wiki.

9/11/2007How to Build a NOC NOC Location Currently, our NOC has some single- occupant offices and some multiple (2-3). We are all on the same floor. We find that being located near the Network Engineering team is quite helpful for urgent escalations. Next year we will be moving into a new location and we do not yet have details on what that will look like.

9/11/2007How to Build a NOC NOC Location Consider lighting and noise control with shared offices. How many monitors will each person need? Will you use a large central monitor for some things? Provide an impromptu meeting space for collaboration on big events.

9/11/2007How to Build a NOC NOC Location Conference bridges greatly enhance collaboration across geographic distances whether working on outages or events.

9/11/2007How to Build a NOC NOC Funding UW funding comes primarily from the state tax payers of Washington,and UW Medical Centers. Funding also comes from the PNWGP and customers and WA state K-20 network. If you recharge you will need a business and billing model. Or will you use time and materials?

9/11/2007How to Build a NOC Tools A very useful tool is live chat or IM for coordinating efforts no matter where your office is. Our customer information is tracked in a home-grown database which has grown and morphed over a dozen years. New needs such as SLAs and layer 1 info now require significant investment in upgrades.

9/11/2007How to Build a NOC Tools Our monitoring system – surprise – is also “homegrown”. We monitor interface state and IP reachability; performance and protocol state connectivity will soon be integrated into our “event system” (NMS) Automated tools can page the appropriate group to notify them of outages or threshold conditions.

9/11/2007How to Build a NOC Tools We have a separate Tools team (with 10 staff members) who design, write, implement, and maintain tools. This allows us to have full-featured and robust tools. One trade-off is fewer “one-off” tools for specific or isolated issues.

9/11/2007How to Build a NOC Tools PNWGP uses Request Tracker, RT, an open-source application to track trouble tickets. Weekly reports are generated for our Directors by sector, severity, and type. Monthly reports are generated by sector for billing purposes.

9/11/2007How to Build a NOC Reporting Key metrics we track include: 1.Ticket numbers by sector for billing 2.Phone call volumes 3.Duration of outages 4.Root Cause Analysis for high-impact events Outage time is measured by duration of the customer impact. After Action Review and Follow-up is conducted for serious events.

9/11/2007How to Build a NOC Reporting Monthly report is ed to the customer for traffic sent to/from their site. Our internal reporting includes “operational impacts” to groups under our main organization. How do you measure your NOC’s success? Response times? Reduced calls?

9/11/2007How to Build a NOC NOC Evolution Factors that have determined operational changes for our organization have been increased size, complexity and number of networks monitored; Need to respond to outages 24 hours/day with on-site personnel rather than paging; Skill and responsibility levels have increased significantly, and continue to do so.