Performance Management (Best Practices) REF: Document ID 15115

Introduction Performance Management involves optimization of network response time and management of consistency and quality of individual and overall network services The most important service is the need to measure the user/application response time. For most users, response time is the critical performance success factor.

Background (1) Performance problems often correlate with capacity of resources (CPU, RAM, Bandwidth). – In networks, this is typically bandwidth and data that must wait in queues before it can be transmitted through the network. – In voice applications, this wait time almost certainly impacts users because factors such as delay and jitter affect the quality of the voice call.

Performance management issues User performance Application performance Capacity planning Proactive fault management It is important to note that with newer application like video and voice performance management is the key success

Performance management process flow (1)

Performance management process flow (2) 1 develop a network management concept of operation – Define the required features : Services, Scalability objectives – Define availability and network management objectives – Define performance SLAs and Metrics – Define SLA

Performance management process flow (3) 2 Measure Performance – Gather network baseline data – Measure availability – Measure response time – Measure accuracy – Measure utilization – Capacity planning

Performance management process flow (4) 3 perform a proactive fault analysis – Use threshold for proactive fault management – Network management implementation – Network operation metrics

Performance management process flow (5)

Develop a network management concept of operation The purpose of this document is to describe the overall desired system characteristics from an operational standpoint The focus of this document is to form the long range operational planning activities for network management and operation. It also provides guidance for the development of all subsequent definition documentation, such as service level agreements.

Define the required features: Services, Scalability Objectives Define services objectives : – To describe what the objectives that networks and services are supposed to be – This step requires that you understand applications, basic traffic flows, user and site counts, and required network services. Define scalability objectives: – to help network engineers design networks that meet future growth requirement and not experience resource constraint (media capacity, number of routes and etc)

Define availability and network management Objectives (1) Defining availability objectives is to explain the level of services needed (service level requirements) This helps to ensure the solution meets end availability requirements It might lead to – categorize different class of service for each availability requirement – Higher availability objective might necessitate increased redundancy and support procedures

Define availability and network management objectives (2) Define manageability objectives to ensure that overall network management does not lack management functionality It might lead to – Have understand the process and tools used for organization – Uncover all important MIB or network tool information required to support a potential network – Have training required to support the new network service

Define performance SLAs and Metrics Performance SLAs and metrics help define and measure the performance of new network solutions to ensure they meet performance requirements. The performance SLAs should include the average expected volume of traffic, peak volume of traffic, average response time and maximum response time allowed

Define SLAs (1) SLA (Service Level Agreement) – Customer (Enterprise), SLM (Service Level Management) - Provider SLA include definitions for problem types and severity and help desk responsibilities – Escalation path, time before escalation at each tier support level – Time to start work on the problem – Time to close target based on priority – Service to provide in the area of capacity planning, hardware replacement

Performance management process flow

Measure Performance Gather Network Baseline data – Perform a baseline of the network before and after a new solution deployment – A typical router/switch baseline report includes capacity issues related to CPU, memory, buffer, link/media utilization, throughput – Application baseline: bandwidth used by app per time period

Measure availability Availability is the the measure of time for which a network system or application is available to a user – Coordinate the help desk phone calls with the statistics collected from managed devices – Check scheduled outages – Etc

Measure Response Time Network response time is the time required to travel between two points Simple level – pings from the network management station to key points I the network. (not accuracy) Server-centric polling : SAA (Service Assurance Agent) on router (Cisco) to measure response time to a destination device Generate traffic that resembles the particular application or technology of interest

Measure accuracy Accuracy is the measure of interface traffic that does not result in error and can be expressed in term of percentage Accuracy = 100 – error rate Error rate = ifInErrors * 100 / (ifInUcastPkts + IfInNUcastPkts)

Measure Utilization (1) Utilization measure the use of a particular resource over time Percentage in which the usage of a resource is compared with its maximum operational capacity High utilization is not necessarily bad Sudden jump in utilization can indicate unnormal condition

Measure Utilization (2) Input utilization = ifInOctets *8*100/(time in second)*ifSpeed Output Utilization ifOutOctets *8*100/(time in second)*ifSpeed

Capacity planning The following are potential areas for concern: – CPU – Backplane or I/O – Memory – Interface and pip sizes – Queuing, latency and jitter – Speed and distance – Application characteristics

Performance management process flow

Perform a Proactive fault analysis One method to perform fault management is through the use of RMON alarms and event groups Distributed management system that enables polling at a local level with aggregation of data at a manager to manager

Use threshold for proactive fault management (1/2) Threshold is the point of interest in specific data stream and generate event when threshold is triggered 2 classes of threshold for numeric data – Continuous threshold apply to continuous or time series data such as data stored in SNMP counter or gauges – Discrete threshold apply to enumerated objects or discrete numeric data such as Boolean objects

Use threshold for proactive fault management (2/2) 2 different forms of continuous threshold – Absolute :use with gauges – Relative (delta): use with counter Step to determine threshold – 1 select the objects – 2 select the devices and interfaces – 3 determine the threshold values for each object or interface – 4 determine the severity for the event generated by each threshold

Network management implementation The organization should have an implemented network management system. SNMP/RMON or other network management system tools

Network operation metrics (1/2) Number of problems that occurs by call priority Minimum, maximum and average time to close in each priority Breakdown of problems by problem type (hardware, software crash, configuration, power user error)

Network operation metrics (2/2) Breakdown of time to close for each problem type Availability by availability or SLA How often you met or missed SLA requirements

Configuration Management (Best Practice) Ref.: Document ID 15111

High Level process flow for Configuration Management

Create Standards (1) Create Standards helps reduce network complexity, the amount of unplanned downtime and exposure to network impact events

Create Standards (2) Following standards for optimal network consistency – Software version control and management – IP addressing standard and management – Naming convention and Domain Name System/ DHCP assignment – Standard Configuration and Descriptors – Configuration Upgrade procedure – Solution Templates

Software Version Control and Management (1) Software version control is the practice of deploying consistent software versions on similar network devices – Limit amount of software defects and interoperability issues – Reduce the risk of unexpected behavior with user interfaces

Software Version Control and Management (2) Following steps for Software version control – Determine device classifications based on chassis, stability and new feature requirements – Target individual software versions for similar-device classification – Test, validate and pilot chosen software versions – Document successful version as standard for similar-device classification – Consistently deploy or upgrade all similar devices to standard software version

IP Address Standards and Management (1) IP address management is the process of allocating, recycling and documenting IP address and subnets in a network – It reduces the opportunity for overlapping or duplicate subnets, wasted IP address space, complexity

IP Address Standards and Management (2) We should standard subnet size for standard application – Subnet size of building – Subnet size of WAN link – Subnet size of Branch site – Subnet size of Loopback The subnet block should promote IP summarization (contiguous IP ) Create standards for IP assignment – Router should be the first available address – Switch may be the next available address – Dynamic address should be followed by fixed address Finally document standard you developed and IP allocation

Naming Convention and DNS/DHCP Assignment (1) Consistent, structure use of naming conventions and DNS for devices helps – Create a consistent point to routers for all network management information related to a device – Reduce the opportunity for duplicate IP address – Creates simple identification of a device showing location, device type and purpose – Improve inventory management by providing a simpler method to identify network devices

Naming Convention and DNS/DHCP Assignment (2) On router, it is strongly recommended to use loopback interface as the primary management interface – Loopback interface can be used for trap, SNMP and syslog Individual interface can have name convention that identifies the device, location, purpose and interface

Naming Convention and DNS/DHCP Assignment (3) It is also recommended to identify DHCP ranges and adding them to the DNS including location of the user Example: “dhcp-bldg-c21-10” to “dhcp-bldg- c21-253” which identifies IP address in building C, second floor wiring closet 1

Standard Configuration and Descriptors (1) Standard Configuration applies to protocol and media configuration as well as global configuration command Descriptors are interface commands used to describe an interface It is recommended to create standard configurations for each device classification – Router, LAN switch, WAN Switch, ATM switch

Standard Configuration and Descriptors (2) Each standard configuration contain the global, media, and protocol configuration command Global configuration – Password, vty, banners – SNMP configuration, Network Time Protocol (NTP) Media configuration – ATM, Frame Relay, Fast Ethernet configuration Protocol Configuration – Routing protocol – Access control list – QoS configuration

Standard Configuration and Descriptors (3) Descriptors are developed by creating a standard format that applies to each interface The descriptor includes – the purpose and location of the interface – Other devices and location connected to the interface – Circuit identifier

Standard Configuration and Descriptors (4) It is recommended – to keep standard configuration parameters in a standard configuration file – downloading the file to each new device prior to protocol and interface configuration We should document the standard configuration file including an explanation of each global configuration parameter and why it is important RME (Cisco Resource Manager Essentials)

Configuration Upgrade Procedure (1) Upgrade procedures ensure that software and hardware upgrades occur smoothly with minimal downtime Upgrade procedures include – vendor verification – Vendor installing references such as release notes – Upgrade methodologies or steps – Configuration guideline – Testing requirement

Solution Templates (1) Solution templates are used to define modular network solutions A network module may be a wiring closet, a WAN field office or an access concentrator It is used to ensure that similar deployment can be carried out in exactly the same way – can reduce risk level to the organization

Solution Templates (2) Specific details of the solution template – Hardware and hardware modules including memory, flash, power and card layouts – Logical topology including port assignment – Software versions including firmware versions – All non-standard, non-devices specific configuration, VLAN configuration, access lists, switching paths, spanning tree parameters and etc – Out of band management requirement – Cable requirement – Installation requirement including environmental, power and rack location

Maintain Documentation (1)

Maintain Documentation (2) It is recommended to use the following network documentation critical success factor – Current device, link and end user inventory – Configuration version control system – TACACS (Terminal Access Controller Access- Control System) configuration log – Network topology documentation

Validate and Audit Standards (1)

Validate and Audit Standards (2) We can use configuration management performance indicators to measure configuration management success Configuration management performance indicators – Configuration integrity checks – Devices, protocol and media audits – Standards and documentation review

Configuration integrity checks It should evaluate the overall configuration of the network its complexity and consistency and potential issues For cisco network, it is recommended to use Netsys configuration validating tool.

Device, Protocol and Media Audits It is used to check consistency in software versions, hardware devices and modules, protocol and media and naming convention Ciscowork RME is a configuration tool that can audit and report on hardware versions modules and software versions

Standards and Documentation review It is done to ensure that the information is accurate and up to date The audit should include reviewing current documentation recommending changes or additions and approving new standards Following documents should be reviewed on a quarterly basis – Standard configuration definition – Solution templates including recommended hardware configuration – Current standard software versions – Upgrade procedures for all devices and software version – Topology documentation – Current templates – IP address management