1. © Synergon Informatika Rt., 1999 Troubleshooting

2. 2 Support Resources for Troubleshooting

3. 3 Topics  Test Equipment and Network Management  Cisco Connection Online (CCO)  A Problem-Solving Model  Cisco IOS commands  Error message logging

4. 4 Test Equipment and Network Management  Cable testers  Network Monitors  Protocol Analyzers  CiscoWorks  TrafficDireector  Other: HP OpenView, MRTG, SNMPc, etc.

5. 5 Cisco Connection Online  www.cisco.com  CCO Documentation  Software Center  Case Management  Bug Navigator, Bug Alerts and Bug Watcher

6. 6 Cisco IOS Commands  clear clear counters, tables, etc.  show snapshot problems with interfaces,media or network problem  ping test the reachability of a protocol on a targeted remote router or other device

7. 7 Cisco IOS commands (contd.)  trace determine a step-by-step routing path to remote routers or devices  debug check the flow of protocol traffic problems, protocol bugs or misconfigurations  telnet make a telnet (layer 7) connection to remote routers or other devices

8. 8 A Problem-Solving Model  Define Problem  Gather Facts  Consider Possibilities  Create Action Plan  Implement Action Plan  Observe Results  Problem Resolved  Document Facts

9. 9 IOS Clear commands  clear counters Reset counters to zero  clear ip route Reset ip routing table  clear ip arp cache Clear ARP cache

10. 10 IOS Show commands  show cdp neighbors (detail) display information about the networks directly conneted to the router  show controllers display information memory management and error counters for interface cards  show interfaces display statistics for the network interfaces  show interface ethernet display information about an Ethernet interface on the router

11. 11 IOS Show commands (cont.)  show ip arp display the entries in the ARP cache for the router  show logging display the state of syslog error and event logging  show version / show hardware display the configuration of the system harware, software version, the names and sources of configuration file and the boot images

12. 12 IOS Show commands (cont.)  show ip interface display the entries in the ARP cache for the router  show ip protocols display the parameters and current state of all active IP routing protocols  show ip route display the routing table

13. 13 IOS Show commands (cont.)  show ip traffic display the the statistics that the router has gathered about its protocol processes  show ip access-list display the the content of current access lists

14. 14 IOS Show commands (cont.)  show stack monitors the stack utilization of processes and interrupt routines  show buffer display statistics for the buffer pools on the router  show memory show statistics about the router’s memory

15. 15 IOS Show commands (cont.)  show protocols display the parameters and current state of all active routing protocols  show processes (cpu / memory) display information the active processes  show tech-support Several of the commands of interest to TAC

16. 16 IOS Show commands (cont.)  show running-config display the running configuration in the RAM  show startup-config display the startup configuration in the NVRAM

17. 17 IOS Ping (Packet Internet Groper) command  application of the ICMP echo message and reply funcyion  tests Layer 3 connection (network)  standard ping in user or privileged mode  extended ping only in extended mode  command: ping 10.10.10.10 or ping

18. 18 IOS Trace command  Tests the step-by-step paths that the packets sent out a router interface follow when travelling to their destination  in user mode and privileged mode  Command: router#trace

19. 19 IOS Debug commands  Provide a wealth of information about the traffic being seen (or not seen) on an interface, error messages generated by nodes on the network, protocol-specific diagnostic packets, and other useful troubleshooting data.  Use process switching  Use to isolate problems, not to monitor network  Undo the debug after troubleshooting  Generate large amounts of output, occasional output, lines of text, infromation in field format

20. 20 IOS Debug commands (cont.)  service timestamps adds time stamps to debug output  No debug all, focus debug with specific argument  terminal monitor to see debug output from a telnet  Commands: router#debug ?  show debug Display the actually running debug commands  Undeb all / no debug x Turn off debug command(s)

21. 21 IOS Telnet command  In user and privileged mode  Make a connection (port 23) into remote router or other device  It tests all the OSI Layers  Command: router#telnet 10.10.10.10

22. 22 Error Message Logging  It can be enable message logging to different destination  Logging to buffer (default)  Logging to console (default)  Logging to remote console (telnet)  Logging to Syslog server  Commands: logging on – enables or disables message logging to all destinations except the console terminal logging console [level] – to limit messages logged to the console based on severity

23. 23 Error Message Logging (cont.)  Commands: logging buffered – copies logging messages to an internal buffer instead of writing them to the console terminal logging monitor [level] – to limit messages logged to the terminal lines based on severity logging [ip-address] – identifies a syslog server host to receive logging messages logging trap [level] – limits the logging messages sent to the syslog servers show logging – display the state of syslog error and event logging

24. 24 OSPF Specific show and debug commands DD

25. 25 Multiaccess Network (cont.)  Source router tells DR on 224.0.0.6  DR tells others on 224.0.0.5  Other routers flood LSAs Link-State Change New Router DR BDR X X LSA

26. 26 Configuring OSPF

27. 27 OSPF Basic Configuration Commands Router (config) # router ospf process-id  Enables on OSPF routing process Router (config-router ) # network address wildcard-mask area area-id Selects participating interfaces

28. 28 OSPF Basic Configuration Example router ospf 63 network 172.16.5.3 0.0.0.0 area 1 network 172.16.0.0 0.0.255.255 area 0 network 192.168.10.5 0.0.0.0 area 1 router ospf 63 network 172.16.5.3 0.0.0.0 area 1 network 172.16.0.0 0.0.255.255 area 0 network 192.168.10.5 0.0.0.0 area 1 172.16.5.3 192.168.10.5 172.16.1.1 172.16.3.1 T0 E0 E1 E2 Area 1 Area 0

29. 29 OSPF Virtual Link Command Router (config-router) # Area area-id virtual-link router-id  Creates a virtual link

30. 30 OSPF Virtual Link Example R2: router ospf 63 area 1 virtual-link 192.168.10.5 R2: router ospf 63 area 1 virtual-link 192.168.10.5 Area 1 Area 0 Area 3 Router ID 192.168.10.5 Router ID 192.168.20.123 R1: router ospf 100 area 1 virtual-link 192.168.20.123 R1: router ospf 100 area 1 virtual-link 192.168.20.123

31. 31 Vendor Interoperability Assigns a cost to an outgoing interface May be required for interoperability Use default cost between Cisco devices Traffic Non-Cisco Cisco Router (config-if ) # Ip ospf cost cost

32. 32 Using Route Summarization, Stub Areas, and Redistribution

33. 33 OSPF Route Summarization Minimizes routing table entries Localizes impact of a topology change Area 1 ABRs Summarization Area 0 Backbone

34. 34 Route Summarization (cont.) Interarea (IA) summary link carries mask One entry can represent several subnet Area 1 Area 0 ABR Summarization O 131.108.4.0255.255.255.0 O 131.108.8.0255.255.255.0 O 131.108.12.0255.255.255.0 O 131.108.16.0255.255.255.0 O 131.108.20.0255.255.255.0 O 131.108.24.0255.255.255.0 O 131.108.28.0255.255.255.0 IA 131.108.16.0255.255.240.0 Routing Table for C Routing Table for B ABC

35. 35 Route Summarization Issue Existing Subnet Mask = 2521111 11 00Valid 131.108.12.0255.255.255.0Address = 120000 11 00 3rd Subnet Summary Route Mask = 2401111 0000Invalid Subnet 131.108.12.0255.255.240.0Address = 120000 1100zero Some addresses may need reallocating

36. 36 Route Summarization Commands Router (config-router ) # Area area-id range address mask  Consolidates IA router on an ABR Router (config-router ) # Summary-address address mask Consolidates external routes on an ASBR

37. 37 Route Summarization Example Area 0 Area 1 Area 2 172.16.96.0 - 172.16.127.0 255.255.255.0 172.16.64.0 - 172.16.95.0 255.255.255.0 172.16.32.0 - 172.16.63.0 255.255.255.0 172.16.32.1 172.16.64.1 172.16.127.1 172.16.96.1 Interface Addresses (255.255.255.0 mask) Interface Addresses (255.255.255.0 mask) R1# router ospf 100 network 172.16.32.0 0.0.31.255 area 1 network 172.16.96.0 0.0.31.255 area 0 area 0 range 172.16.96.0 255.255.224.0 area 1 range 172.16.32.0 255.255.224.0 R1# router ospf 100 network 172.16.32.0 0.0.31.255 area 1 network 172.16.96.0 0.0.31.255 area 0 area 0 range 172.16.96.0 255.255.224.0 area 1 range 172.16.32.0 255.255.224.0 R2# router ospf 100 network 172.16.32.0 0.0.31.255 area 2 network 172.16.96.0 0.0.31.255 area 0 area 0 range 172.16.96.0 255.255.224.0 area 2 range 172.16.64.0 255.255.224.0 R2# router ospf 100 network 172.16.32.0 0.0.31.255 area 2 network 172.16.96.0 0.0.31.255 area 0 area 0 range 172.16.96.0 255.255.224.0 area 2 range 172.16.64.0 255.255.224.0 R1 R2

38. 38 Stub Areas Hide external routes, reduce database Consolidate external links - 0.0.0.0 Area 1 Area 0 Area 2 0.0.0.0 BGP Stub Area 172.20.64.0 - 172.20.95.0 255.255.255.0 External AS

39. 39 Stub Area Restrictions External AS 0.0.0.0 Area 2 single exit point or, if multiple exit points, suboptimal path acceptable Transit area for virtual links disallowed An ASBR cannot be internal to stube Single Exit Point

40. 40 Totally Stubby Areas Block external and summary routes Know only intra-area and default routes Area 1 Area 0 Area 2 0.0.0.0 BGP Stub Area 172.20.64.0 - 172.20.95.0 255.255.255.0 External AS Summary (IA) Route

41. 41 OSPF Stub Area Commands Router (config-router ) # Area area-id stub [no-summary]  Creates a stub network Router (config-router ) # Area area-id default-cost cost Specifies cost for default route sent into stub area

42. 42 OSPF Stub Area Example External AS Area 0 Stub Area 2 192.168.15.2 192.168.15.1 192.168.14.1 E0S0 R3 R4 R3# interface Ethernet 0 ip address 192.168.14.1 255.255.255.0 interface Serial 0 ip address 192.168.15.1 255.255.255.252 router ospf 100 network 192.168.14.0 0.0.0.255 area 0 network 192.168.15.0 0.0.0.255 area 2 area 2 stub R3# interface Ethernet 0 ip address 192.168.14.1 255.255.255.0 interface Serial 0 ip address 192.168.15.1 255.255.255.252 router ospf 100 network 192.168.14.0 0.0.0.255 area 0 network 192.168.15.0 0.0.0.255 area 2 area 2 stub R4# interface Serial 0 ip address 192.168.15.2 255.255.255.252 router ospf 15 network 192.168.15.0 0.0.0.255 area 2 area 2 stub R4# interface Serial 0 ip address 192.168.15.2 255.255.255.252 router ospf 15 network 192.168.15.0 0.0.0.255 area 2 area 2 stub

43. 43 OSPF Totally Stubby Example External AS Area 0 Stub Area 2 192.168.15.2 192.168.15.1 192.168.14.1 E0S0 R3 R4 R3# router ospf 100 network 192.168.14.0 0.0.0.255 area 0 network 192.168.15.0 0.0.0.255 area 2 area 2 stub no-summary area 2 stub default-cost 20 R3# router ospf 100 network 192.168.14.0 0.0.0.255 area 0 network 192.168.15.0 0.0.0.255 area 2 area 2 stub no-summary area 2 stub default-cost 20 R4# router ospf 15 network 192.168.15.0 0.0.0.255 area 2 area 2 stub R4# router ospf 15 network 192.168.15.0 0.0.0.255 area 2 area 2 stub

44. 44 OSPF Route Redistribution RIP IGRP Enhanced IGRP IS-IS OSPF BGP EGP Allows routing-information exchange between OSPF and other routing protocols

45. 45 OSPF redistribution Command Router (config-router ) # Redistribute protocol [ process-id ] [ metric value ] [ metric-type value ] [ subnet ]  Redistributes routes from OSPF into other routing protocols (and vice versa)

46. 46 OSPF Redistribution Example 1 RIP OSPF Area 0 172.16.62.1 172.16.9.1 S0 S1 R1 172.16.8.1172.16.63.1 Redistribution between RIP and OSPF router ospf 109 network 172.16.62.0 0.0.0.255 area 0 network 172.16.63.0 0.0.0.255 area 0 redistribute rip subnets metric-type 1 metric 20 router rip network 172.16.0.0 passive-interface serial 0 passive-interface serial 1 default-metric 10 redistribute ospf match internal external 1 external 2 router ospf 109 network 172.16.62.0 0.0.0.255 area 0 network 172.16.63.0 0.0.0.255 area 0 redistribute rip subnets metric-type 1 metric 20 router rip network 172.16.0.0 passive-interface serial 0 passive-interface serial 1 default-metric 10 redistribute ospf match internal external 1 external 2

47. 47 Redistribution Example 2 “Back door” creates potential loop RIP OSPF Area 0 R1 R2 R3 172.16.9.1 172.16.8.1 router ospf 109 network 172.16.62.0 0.0.0.255 area 0 network 172.16.63.0 0.0.0.255 area 0 redistribute rip subnets metric-type 1 metric 20 distribute-list 11 out rip access-list 11 permit 172.16.8.0 0.0.7.255 router ospf 109 network 172.16.62.0 0.0.0.255 area 0 network 172.16.63.0 0.0.0.255 area 0 redistribute rip subnets metric-type 1 metric 20 distribute-list 11 out rip access-list 11 permit 172.16.8.0 0.0.7.255

48. 48 Verifying OSPF Operation

49. 49 Show ip ospf interface Command Verifies interfaces are in correct areas Router# show ip ospf interface e 0 Ethernet 0 is up, line protocol is up Internet Address 203.250.14.1 255.255.255.0, Area 0.0.0.0 Process ID 10, Router ID 203.250.13.41, Network Type BROADCAST, Cost: 10 Transmit Delay is 1 sec, State BDR, Priority l Designated Router (ID) 203.250.15.1, Interfece address 203.250.14.2 Backup Designated router (ID) 203.250.13.41, interface address 203.250.14.1 Timer intervale configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 0:00:02 Neighbor Count is 3, Adjacent neighbor count is 3 Adjacent with neighbor 203.250.15.1 (Designated Router) Loopback0 is up, line protocol is up Internet address 203.250.13.41 255.255.255.255, Area 1 Process ID 10, Router ID 203.250.13.41, Network Type LOOPBACK, Cost: 1 Loopback interface is treated as a stub Host Router# show ip ospf interface e 0 Ethernet 0 is up, line protocol is up Internet Address 203.250.14.1 255.255.255.0, Area 0.0.0.0 Process ID 10, Router ID 203.250.13.41, Network Type BROADCAST, Cost: 10 Transmit Delay is 1 sec, State BDR, Priority l Designated Router (ID) 203.250.15.1, Interfece address 203.250.14.2 Backup Designated router (ID) 203.250.13.41, interface address 203.250.14.1 Timer intervale configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 0:00:02 Neighbor Count is 3, Adjacent neighbor count is 3 Adjacent with neighbor 203.250.15.1 (Designated Router) Loopback0 is up, line protocol is up Internet address 203.250.13.41 255.255.255.255, Area 1 Process ID 10, Router ID 203.250.13.41, Network Type LOOPBACK, Cost: 1 Loopback interface is treated as a stub Host

50. 50 Show ip ospf Command Router # show ip ospf Routing Process "ospf 1" with ID 2.2.2.2 Supports only single TOS (TOS0) routes SPF schedule delay 5 secs, Hold time between two SPFs 10 secs Number of areas in this router is 1 Area 23 Number of interfaces in this area is 3 Area has no authentication SPF algorithm executed 19 times Area ranges are Link State Update Interval is 0:30:00 and due in 0:04:55 Link State Age Interval is 0:20:00 and due in 0:04:55 Router # show ip ospf Routing Process "ospf 1" with ID 2.2.2.2 Supports only single TOS (TOS0) routes SPF schedule delay 5 secs, Hold time between two SPFs 10 secs Number of areas in this router is 1 Area 23 Number of interfaces in this area is 3 Area has no authentication SPF algorithm executed 19 times Area ranges are Link State Update Interval is 0:30:00 and due in 0:04:55 Link State Age Interval is 0:20:00 and due in 0:04:55 Displays general information about the OSPF routing process

51. 51 show ip ospf database Command Router# show ip ospf database OSPF Router with ID (3.3.3.3) (Process ID 1) Router Link States (Area 23) Link IDADVAgeSeq#ChecksumLink Routercount 3.3.3.33.3.3.3780x800000320x80865 4.4.4.44.4.4.416910x800000280xEllC1 2.2.2.22.2.2.216930x800000300x835E5 1.1.1.11.1.1.116960x800000260x80A11 Net Link States (Area 23) Link IDADV RouterAgeSeg#Checksum 150.100.4.2 4.4.4.416910x800000300x2FCE 150.100.1.2 2.2.2.216930x800000240xFB29 Router# show ip ospf database OSPF Router with ID (3.3.3.3) (Process ID 1) Router Link States (Area 23) Link IDADVAgeSeq#ChecksumLink Routercount 3.3.3.33.3.3.3780x800000320x80865 4.4.4.44.4.4.416910x800000280xEllC1 2.2.2.22.2.2.216930x800000300x835E5 1.1.1.11.1.1.116960x800000260x80A11 Net Link States (Area 23) Link IDADV RouterAgeSeg#Checksum 150.100.4.2 4.4.4.416910x800000300x2FCE 150.100.1.2 2.2.2.216930x800000240xFB29

52. 52 show ip protocol Command Router> show ip protocol Routing Protocol is "ospf 300" Sendinq updates every 0 seconds Invalid after 0 seconds, hold down 0, flushed after 0 Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Redistributing: ospf 300 Routing for Networks: 183.8.0.0/0.0.255.255 144.253.100.0/0.0.0.255 Routing Information Sources: GatewayDistance Last Update 14d.253.100.0110 6d21 183.8.128.121100:17:32 192.3.63.1921100:17:33 153.50.193.11100:17:33 183.8.6d.1301106d19 183.8.64.1281100:17:33 133.3.4.01100:17:33 131.108.100.31100:17:33 Distance: (default is 1l0) - - More - - Router> show ip protocol Routing Protocol is "ospf 300" Sendinq updates every 0 seconds Invalid after 0 seconds, hold down 0, flushed after 0 Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Redistributing: ospf 300 Routing for Networks: 183.8.0.0/0.0.255.255 144.253.100.0/0.0.0.255 Routing Information Sources: GatewayDistance Last Update 14d.253.100.0110 6d21 183.8.128.121100:17:32 192.3.63.1921100:17:33 153.50.193.11100:17:33 183.8.6d.1301106d19 183.8.64.1281100:17:33 133.3.4.01100:17:33 131.108.100.31100:17:33 Distance: (default is 1l0) - - More - -

53. 53 Other OSPF show Commands Router# show ip ospf virtual-links Displays parameters about OSPF virtual links Router# show ip ospf neighbor detail Displays neighbor information per interface Router# show ip ospf border-routers Displays routes to the ABR and ASBR

54. 54 Summary OSPF is a scalable, standards-based link-state routing protocol OSPF features: Hierarchical design VLSM support Vendor interoperability Route summarization Route redistribution