1. Network Troubleshooting
Accessing the WAN – Chapter 8

2. Objectives Establish a network baseline
Describe troubleshooting methodologies and troubleshooting tools
Describe the common issues that occur during WAN implementation
Troubleshoot enterprise network implementation issues

3. Establish a Network Baseline
Explain the importance of network documentation

4. Establish a Network Baseline
When you document your network, you may have to gathr information directly from routers and switches.
Commands that are useful to the network documentation process include:
The ping command is used to test connectivity with neighboring devices. Pinging to other PCs in the network also initiates the MAC address auto-discovery process.
The telnet command is used to log in remotely to a device for accessing configuration information.
The show ip interface brief is used to display the up or down status and IP address of all interfaces.
The show ip route command is used to display the routing table in a router to learn the directly connected neighbors, more remote devices (through learned routes), and the routing protocols.
The show cdp neighbor detail command is used to obtain detailed information about directly connected Cisco neighbor devices.

5. Establish a Network Baseline
Explain the purpose for measuring normal network performance when creating a baseline

6. Establish a Network Baseline
Describe the steps for establishing a network baseline

7. Describe Troubleshooting Methodologies and Troubleshooting Tools
When you document your network, you may have to gather information directly from routers and switches.
Commands that are useful to the network documentation process include:
The ping command is used to test connectivity with neighboring devices. Pinging to other PCs in the network also initiates the MAC address auto-discovery process.
The telnet command is used to log in remotely to a device for accessing configuration information.
The show ip interface brief is used to display the up or down status and IP address of all interfaces.
The show ip route command is used to display the routing table in a router to learn the directly connected neighbors, more remote devices (through learned routes), and the routing protocols.
The show cdp neighbor detail command is used to obtain detailed information about directly connected Cisco neighbor devices.

8. Describe Troubleshooting Methodologies and Troubleshooting Tools
There are three main methods for troubleshooting:
Bottom-Up Troubleshooting Method
In bottom-up troubleshooting you start with the physical components of the network and move up through the layers.
Bottom-up troubleshooting is a good approach to use when the problem is suspected to be a physical one.
Top-Down Troubleshooting Method
In top-down troubleshooting your start with the end-user applications and move down the layers of the OSI model.
Use this approach for simpler problems or when you think the problem is with a piece of software.
Divide-and-Conquer Troubleshooting Method
In divide-and-conquer troubleshooting you start by collecting user experience of the problem, document the symptoms and then, using that information, make an informed guess as to which OSI layer to start your investigation.
For example, if users can't access the web server and you can ping the server, then you know that the problem is above Layer 3.
If you can't ping the server, then you know the problem is likely at a lower OSI layer.

9. Describe Troubleshooting Methodologies and Troubleshooting Tools
To quickly resolve network problems, take the time to select the most effective troubleshooting method.
Use the process shown in the figure to help you select the most efficient troubleshooting method.
For example: Two IP routers are not exchanging routing information. The last time this type of problem occurred it was a protocol issue. So you choose the divide-and-conquer troubleshooting method.

10. Gathering Symptoms Step 1. Analyze existing symptoms
Analyze symptoms gathered from the trouble ticket or users to form a definition of the problem.
Step 2. Determine ownership
If problem is within your system, move onto next stage.
If the problem is outside the boundary of your control, for example, lost Internet connectivity you need to contact an administrator for the external system.
Step 3. Narrow the scope
Determine if the problem is at the core, distribution, or access layer of the network.
Step 4. Gather symptoms from suspect devices
Use knowledge and experience to determine if the problem is a hardware or software problem.
Step 5. Document symptoms
Sometimes the problem can be solved using the documented symptoms. If not, begin the isolating phase of the general troubleshooting process.

11. Gathering Symptoms
Use the Cisco IOS commands to gather symptoms about the network.
Although the debug command is an important tool for gathering symptoms it generates a large amount of console message traffic and the performance of a network device can be noticeably affected.
Make sure you warn network users that a troubleshooting effort is underway and that network performance may be affected.
Remember to disable debugging when you are done.

12. Describe Troubleshooting Methodologies and Troubleshooting Tools
NMS Tools
Network management system (NMS) tools include device-level monitoring, configuration, and fault management tools.
Network monitoring software graphically displays a physical view of network devices, allowing network managers to monitor remote devices without physically checking them.
Examples are CiscoView, HP Openview, Solar Winds, and What's Up Gold.
Knowledge Bases
On-line network device vendor knowledge bases have become indispensable sources of information.
When vendor-based knowledge bases are combined with Internet search engines like Google, a network administrator has access to a vast pool of experience-based information.

13. Software Troubleshooting Tools
Baselining Tools
For example they can help you draw network diagrams, help you to keep network software and hardware documentation up-to-date and help you to cost-effectively measure baseline network bandwidth use.
Many tools for automating the network documentation and baselining process are available.
The figure shows a screen chapter of the SolarWinds LAN surveyor and CyberGauge software.
Protocol Analyzers
A protocol analyzer decodes the various protocol layers in a recorded frame and presents this information in a relatively easy to use format.
The figure shows a screen capture of the Wireshark protocol analyzer.
Most protocol analyzers can filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device can be captured.

14. Hardware Troubleshooting Tools
Network Analysis Module
A network analysis module (NAM) can be installed in Cisco Catalyst 6500 series switches and Cisco 7600 series routers to provide a graphical representation of traffic.
Digital Multimeters
Digital multimeters (DMMs) are test instruments that are used to directly measure electrical values of voltage, current, and resistance.
Cable Testers
Cabling testers can be used to detect broken wires, crossed-over wiring, shorted connections, and improperly paired connections.
These devices can be inexpensive continuity testers, moderately priced data cabling testers, or expensive time-domain reflectometers (TDRs).
TDRs are used to test the distance to a break in a cable.
TDRs used to test fiber optic cables are known as optical time-domain reflectometers (OTDRs).

15. Hardware Troubleshooting Tools
Cable Analyzers
Cable analyzers are multifunctional handheld devices that are used to test and certify copper and fiber cables for different services and standards.
The more sophisticated tools include advanced troubleshooting diagnostics that measure distance to performance defect (NEXT, RL), identify corrective actions, and graphically display crosstalk and impedance behavior.
Portable Network Analyzers
Portable devices that are used for troubleshooting switched networks and VLANs.
By plugging the network analyzer in anywhere on the network, a network engineer can see the switch port to which the device is connected and the average and peak utilization.
The analyzer can also be used to discover VLAN configuration, identify top network talkers, analyze network traffic, and view interface details.

16. Describe the Common Issues that Occur During WAN Implementation
WAN technologies function at the lower three layers of the OSI reference model.
A communications provider normally owns the data links that make up a WAN.
The links are made available to subscribers for a fee and are used to interconnect LANs or connect to remote networks.
WAN data transfer speed (bandwidth) is considerably slower than the common LAN bandwidth.
The charges for link provision are the major cost element, therefore the WAN implementation must aim to provide maximum bandwidth at acceptable cost.

17. Describe the Common Issues that Occur During WAN Implementation
WAN connectivity is important to business and expensive, these are the steps for designing or modifying a WAN:
Step 1. Locate LANs - Establish the source and destination endpoints that will connect through the WAN.
Step 2. Analyze traffic - Know what data traffic must be carried, its origin, and its destination.
Step 3. Plan the topology - A high requirement for availability requires extra links that provide alternative data paths for redundancy and load balancing.
Step 4. Estimate the required bandwidth - Traffic on the links may have varying requirements for latency and jitter.
Step 5. Choose the WAN technology - Suitable link technologies must be selected.
Step 6. Evaluate costs - When all the requirements are established, installation and operational costs for the WAN can be determined and compared with the business need driving the WAN implementation.

18. WAN Traffic Considerations
The table in the figure shows the wide variety of traffic types and their varying requirements of bandwidth, latency, and jitter that WAN links are required to carry.
To determine traffic flow conditions and timing of a WAN link, you need to analyze the traffic characteristics specific to each LAN that is connected to the WAN.

19. Describe the Common Issues that Occur During WAN Implementation
Describe the considerations for designing a WAN topology

20. Describe the Common Issues that Occur During WAN Implementation
Describe common WAN implementation issues

21. Describe the Common Issues that Occur During WAN Implementation
Describe the recommended steps for troubleshooting a WAN

22. Troubleshoot Enterprise Network Implementation Issues
Explain how network diagrams are used for troubleshooting

23. To isolate problems at the physical layers
Check for bad cables or connections
Verify that the cable is properly connected and is in good condition. Your cable tester might reveal an open wire.
Check that the correct cabling standard is adhered to throughout the network
Verify that the proper cable is being used. For example, in the figure, the Fluke meter detected that a cable was good for Fast Ethernet, it is not qualified to support 1000BASE-T.
Check that devices are cabled correctly
Check that cables are connected to their correct ports.
This is where having a neat and organized wiring closet saves you a great deal of time.
Verify proper interface configurations
Check that all switch ports are set in the correct VLAN and, speed, and duplex settings are correctly configured.
Check operational statistics and data error rates
Use Cisco show commands to check for statistics such as collisions and input and output errors.

24. Symptoms of Data Link Layer Problems
Common symptoms at the data link layer include:
No functionality or connectivity at network layer or above
Some Layer 2 problems can stop the frames across a link.
Network is operating below baseline performance levels
There are two types of suboptimal Layer 2 operation:
Frames take an illogical path to their destination but do arrive. An example of a problem which could cause frames to take a suboptimal path is a poorly designed Layer 2 spanning-tree.
Some frames are dropped. An extended or continuous ping also reveals if frames are being dropped.
Excessive broadcasts
Excessive broadcasts result from one of the following:
Poorly programmed or configured applications
Large Layer 2 broadcast domains
Underlying network problems, such as STP loops.
Console messages
In some instances, a router recognizes a Layer 2 problem has occurred and sends alert messages to the console.
The most common console is line protocol down message.

25. Troubleshooting Layer 3 Problems
In most networks, static routes are used in combination with dynamic routing protocols.
Improper configuration of static routes can lead to less than optimal routing and, in some cases the network to become unreachable.
Here are some possible problems involving routing protocols:
General network issues
Often a change in the topology, such as a down link, may have affects on other areas that might not be obvious at the time.
Connectivity issues
Check for any equipment problems, cabling, and ISP problems.
Neighbor issues
Check if there are any problems with the routers forming neighbor.
Topology database
Check the topology table, for any missing or unexpected entries.
Routing table
Check the routing table for anything missing or unexpected routes.
Use debug commands to view routing updates and maintenance.

26. Transport Layer Troubleshooting: Access List Issues
1. Selection of traffic flow
ACL must be applied to the correct interface, and correct traffic direction must be selected to function properly.
If the router is running both ACLs and NAT, the order in which each of these technologies is applied is important:
Inbound traffic is processed by the inbound ACL before being processed by outside-to-inside NAT.
Outbound traffic is processed by the outbound ACL after being processed by inside-to-outside NAT.
2. Order of access control elements
The elements ACL should be from specific to general.
3. Implicit deny all
Forgetting about this implicit access control element may be the cause of an ACL misconfiguration.
4. Addresses and wildcard masks
Complex wildcard masks provide significant improvements in efficiency, but are more subject to configuration errors.
The address and wildcard mask to select the first 15 host addresses in either the or network.

27. Transport Layer Troubleshooting: Access List Issues
5. Selection of transport layer protocol
When configuring ACLs, it is important that only the correct transport layer protocols [TCP, UDP] be specified.
6. Source and destination ports
Address and port information for traffic generated by a replying host is the mirror address and port from the source host.
7. Use of the established keyword
If the keyword is applied to an outbound ACL, unexpected results may occur.
8. Uncommon protocols
Uncommon protocols that are gaining popularity are VPN and encryption protocols.
Troubleshooting Access Control Lists
A useful command for viewing ACL operation is the log keyword on ACL entries.
This keyword instructs the router to place an entry in the system log whenever that entry condition is matched.

28. Application Layer Overview
The most widely known application layer protocols:
Telnet - Enables users to establish terminal session connections with remote hosts.
HTTP - Supports the exchanging of text, graphic, sound, video, and other multimedia files on the web.
FTP - Performs interactive file transfers between hosts.
TFTP - Performs basic interactive file transfers typically between hosts and networking devices.
SMTP - Supports basic message delivery services.
POP - Connects to mail servers and downloads .
Simple Network Management Protocol (SNMP) - Collects management information from network devices.
DNS - Maps IP addresses to the names assigned to network devices.
Network File System (NFS) - Enables computers to mount drives on remote hosts and operate them as if they were local drives.

29. Symptoms of Application Layer Problems
A problem at the application layer can result in unreachable or unusable resources when the physical, data link, network, and transport layers are functional.
It is possible to have full network connectivity, but the application simply cannot provide data.
Another type of problem at the application layer occurs when the physical, data link, network, and transport layers are functional, but the data transfer and requests for network services from a single network service or application do not meet the normal expectations of a user.
A problem at the application layer may cause users to complain that the network or the particular application that they are working with is sluggish or slower than usual when transferring data or requesting network services.

30. Troubleshooting Application Layer Problems
The steps for troubleshooting application layer problems are:
Step 1. Ping the default gateway.
If successful, Layer 1 and Layer 2 services are functioning properly.
Step 2. Verify end-to-end connectivity.
If Layers 1-3 functioning properly, the issue exist at a higher layer.
Step 3. Verify access list and NAT operation.
If the ACLs and NAT are functioning as expected, the problem must lie in a higher layer.
Step 4. Troubleshoot upper layer protocol connectivity.
Upper layer protocol, such as FTP, HTTP, or Telnet ride on top of the basic IP transport but are subject to protocol-specific problems relating to packet filters and firewalls.
Troubleshooting an upper layer protocol connectivity problem requires understanding the process of the protocol.
This information is usually found in the latest RFC for the protocol or on the developer web page.

31. Correcting Application Layer Problems
The steps for correcting application layer problems are:
Step 1: Make a backup.
Ensure that a valid configuration has been saved.
Step 2: Make an initial configuration change.
Make only one change at a time.
Step 3: Evaluate each change and its results.
If the results of any problem-solving steps are unsuccessful, immediately undo the changes.
Step 4: Determine if the change solves the problem.
Verify that the change actually resolves the problem without introducing any new problems.
If the problem is not solved, undo all the changes.
Step 5: Stop when the problem is solved.
Step 6: If necessary, get assistance from outside resources.
This may be a co-worker, a consultant, or TAC.
Step 7: Document.
Once the problem is resolved, document the solution.

32. Summary Network Baseline Network documentation should include:
How a network is expected to perform under normal conditions
Network documentation should include:
Network configuration table
End-system configuration table
Network topology diagram
Planning for the 1st baseline
Determine what type of data to collect
Identify devices and ports of interest
Determine baseline duration

33. Summary 3 stages of the troubleshooting process
Gather symptoms
Isolate problem
Correct problem
3 main methods for troubleshooting a network
Bottom up
Top down
Divide & conquer

34. Summary Software troubleshooting tools Hardware troubleshooting tools
Cisco view
Solar winds
HP Open view
Hardware troubleshooting tools
Network analysis mode
Digital multi-meters
Cable testers
Network analyzer

35. Summary Common WAN implementation issues include
QoS
Reliability
Security
Latency
Confidentiality
Public or Private
Using a layered approach to troubleshooting aids in
isolating and solving the problem