1 Root-Cause Network Troubleshooting Optimizing the Process Tim Titus CTO, PathSolutions

2 Business disconnect Why is troubleshooting so hard? Troubleshooting methodology Tool selection Finding the root-cause Achieving Total Network Visibility Agenda

3 You’re responsible for the entire network Most network engineers know less about their network’s health and performance than their user community You can’t manage what you can’t measure -- Peter Drucker Business Disconnect

4 Business Reasons Networks are getting more complex Less staff remains to support the network Technical Reasons Proper methodology is not utilized Wrong tools are employed Why is Troubleshooting so Hard?

5 What graduates a junior level Engineer to a senior level Engineer is their troubleshooting methodology Troubleshooting Methodologies

6 “Do something to try to fix the problem” Reboot the device Change the network settings Replace hardware Re-install the OS Bad Methodology

7 Collect information Verify Original Problem is Solved and no new problems exist Create hypothesis Test hypothesis Implement fix Document fix Notify users Undo changes Good Methodology

8 Types of Tools Cable Testers Packet analyzers/capture Application Performance Monitoring (APM) Flow collectors SNMP Collectors Tool Selection

9 Results 4.3db of loss NEXT detected Cable Tester Actual VoIP Call You have information about Layer 1 on one link in the network Using a cable tester to solve a call quality problem Cable Testers

10 Good for: Confirming physical issues on one link in the network Bad for: Finding physical issues on the network Determining application usage Finding bandwidth limitations Finding device limitations Cable Testers

11 Results of VoIP Call Latency: 127ms Jitter: 87ms Packet loss: 8.2% Packet Capture Actual VoIP Call You have confirmation that there is a problem, but no idea which device or link caused the packet loss Using a sniffer to solve a call quality problem Packet Capture

12 Good for: Confirming packet loss (Are we missing packets?) Confirming packet contents issues (No QoS tagging on packets when there should be) Determining application-level issues (Source and destination IP and ports used for a session) Bad for: Finding physical, data-link, or network issues Finding bandwidth limitations Finding device limitations Packet Capture

13 Results of Simulation Latency: 127ms Jitter: 87ms Packet loss: 8.2% Agent Simulated VoIP Call You have knowledge of the experience across the network, but no understanding of the source or cause of the problem. Using APM to determine performance through the network Agent Application Performance Monitoring

14 Good for: Measuring user experience across the network (Are we having problems right now?) Bad for: Finding physical, data-link, or network issues Finding bandwidth limitations Finding device limitations Application Performance Monitoring

15 Results of Flow SourceIP: :80 DestinationIP: :3411 Packets: 251 Bytes: 19,386 Flow Collector Actual VoIP Call You have knowledge of a transfer across the network, but no recognition if there were any problems with the transfer. Using a flow collector to determine usage of the network Flow Record Flow Collectors

16 Good for: Determining communications across the network Who is using a link? When do they use it? What do they use it for? Bad for: Finding physical, data-link, or network issues Finding bandwidth limitations Finding device limitations Flow Collectors

17 Results of Collection WAN link is overloaded at 2:35pm SNMP Collector Actual VoIP Call You have data about conditions on some parts of the network, but no analysis of the problem or correlation to events Collecting information from switches and routers to discover faults SNMP Collectors

18 Good for: Tracking packet loss per interface/device (Are we dropping packets on a link? why?) Monitoring device and link resource limitations (Are we over-utilizing a link? Is the router CPU pegged?) Bad for: Determining who is using the network Finding application layer problems SNMP Collectors

19 Step 1: Identify the involved endpoints and where they are connected into the network Poor Quality VoIP Call Finding the Root-Cause

20 Step 2: Identify the full layer-2 path through the network from the first phone to the second phone Finding the Root-Cause

21 Step 3: Investigate involved switch and router health (CPU & Memory) for acceptable levels Finding the Root-Cause

22 Step 4: Investigate involved interfaces for: VLAN assignment DiffServe/QoS tagging Queuing configuration 802.1p Priority settings Duplex mismatches Cable faults Half-duplex operation Broadcast storms Incorrect speed settings Over-subscription TRANSIENT PROBLEM WARNING: If the error condition is no longer occurring when this investigation is performed, you may not catch the problem Finding the Root-Cause

23 In a perfect world, you want: Monitoring of:  Every switch, router, and link in the entire infrastructure  All error counters on the interfaces  QoS configuration and performance Continuous collection of information Automatic layer-1, 2, and 3 mapping from any IP endpoint to any other IP endpoint Problems identified in plain-English for rapid remediation This is what PathSolutions TotalView does Optimizing the Methodology

24 Install TotalView Result: One location is able to monitor all devices and links in the entire network for performance and errors All Switches and Routers are queried for information Deployment

25 Broad: All ports on all routers & switches Continuous: Health collected every 5 minutes Deep: 18 different error counters collected and analyzed Network Prescription engine provides plain- English descriptions of errors: Total Network Visibility® “This interface is dropping 12% of its packets due to a cable fault”

26 Establish Baseline of Network Health 7% Loss from cabling fault 12% Loss from Alignment Errors 11% Loss from Jumbo Frame Misconfiguration 28% Loss from Duplex mismatch Results Within 12 Minutes

27 Repair Issues 7% Loss from cabling fault 12% Loss from Alignment Errors 28% Loss from Duplex mismatch Results Within 12 Minutes 11% Loss from Jumbo Frame Misconfiguration

28 11:32am 100% Transmit utilization 15% Loss from discards Latency & Jitter penalty incurred 7:56am 18% Loss from Cable Fault 12:02pm 12% Loss from Collisions Path Analysis Report

29 Demo

30 Don’t turtle your network

31 With it, you will always have an easy way to map out your network on any white board! Free Network Equipment Magnet Set

32 Must be present to win Hourly iPad mini giveaways Booth #1951 Today’s Show Times 1:00 2:00 3:00 4:00

33 Special Guest at 1pm: Brian Chee on Station Aloha One of our more interesting implementations. Located 3 miles beneath the ocean surface. How exactly do you manage its performance? Attend this special session and discover what it’s like to manage our most unique network!

34