Real World Troubleshooting with Wireshark

Agenda Packet Capture Family of Tools Where to Capture Filters Setting Up Your Environment Problem Solving Where to Get Help

Packet Capture Tools Steelhead tcpdump / embedded Shark Pilot / Shark / vShark Wireshark (via WinPcap) tshark WinPcap / WinDump

Wireshark UI Tour

Pilot UI Tour Ribbons Source Panel View ToolTip Main Events Workspace 200+ Searchable Views Fast Troubleshooting via Select-able / Drill-able Charts Efficient Operation via Robust Context Menus Two Click Export to Wireshark with Filtering Enables Very Fast & Efficient Problem Resolution Timeline

tshark C:\Program Files\Wireshark>tshark –help Usage: tshark [options] ... Capture interface: -i <interface> name or idx of interface (def: first non-loopback) -f <capture filter> packet filter in libpcap filter syntax -s <snaplen> packet snapshot length (def: 65535) -p don't capture in promiscuous mode -B <buffer size> size of kernel buffer (def: 1MB) -y <link type> link layer type (def: first appropriate) -D print list of interfaces and exit -L print list of link-layer types of iface and exit Capture stop conditions: -c <packet count> stop after n packets (def: infinite) -a <autostop cond.> ... duration:NUM - stop after NUM seconds filesize:NUM - stop this file after NUM KB files:NUM - stop after NUM files ………..

Where & What to Capture It Depends Capture Liberally WAN Opt ADC Issue App Issue Capture Liberally set liberal snaplen Filter Aggressively Where: Points of Interest or Transition Proxies, ADC, FW, Client, Server, etc. Steelheads: SFE & CFE LAN/WAN Stingray: Both frontend & backend int. Client & Server What: Healthy vs Problem LAN vs WAN FW vs No FW

Why get a healthy baseline capture?

Filters Capture Filters Display Filters ether host 00:04:13:00:09:a3 host 10.10.10.10 and host 10.10.15.15 src net 192.169.33.0 mask 255.255.255.0 src net 192.168.33.0/24 tcp len <= 70 tcp tcp host 10.10.10.10 tcp port 7800 tcp portrange 20-25 tcp host 10.10.10.10 port 80 ip6 vlan 223 not, and, or !, &&, || Parenthesis available to group values, but need be escaped when used at cli Display Filters ip.addr == 10.10.10.10 !ip.addr == 192.168.32.16 ip.addr==192.168.1.10 && ip.addr==192.168.1.20 tcp tcp.flags.syn tcp.port == 80 tcp.analysis.flags tcp.options.rvbd.probe tcp.options.rvbd.probe.prober == <SH_IP> tcp.options.rvbd.trpy tcp.options.scps vlan.id==<#>

Capture Options v1.8 can capture from multiple interfaces Setup a liberal capture filter Note in Wireshark 1.8.2 you can now capture from multiple interfaces.

Display Filters

Display Filters – Right Click

Demonstrate Filtering

Lab: Capture Filtering Write filters to capture the following: tcp traffic btwn test machine 10.10.10.3 and Class B subnet 10.10.15.0 HTTP traffic for test machine 192.168.33.15 Traffic on either VLAN 55 or 222

Lab: Capture Filtering (Answers) tcp host 10.10.10.3 and net 10.10.15.0/16 tcp port 80 and host 192.168.33.15 vlan 55 or vlan 222

Lab: Display Filtering Write filters to display the following: Analyze all vlans except 1 TCP traffic with possible problems Find all TCP SYN packets Show Steelhead probes Steelhead Full Transparency traffic over WAN All Probes from Client Steelhead 192.168.32.61 Show all SCPS probes

Lab: Display Filtering (Answers) !vlan.id==1 tcp.analysis.flags tcp.flags.syn==1 tcp.options.rvbd.probe tcp.options.rvbd.trpy tcp.options.rvbd.probes.prober==192.168.32.61 tcp.options.scps

Setting Up Your Environment Setup Profiles for Different Use Cases WAN Opt Application Troubleshooting Web VoIP Wireless VLAN analysis

Customize Your Profiles Coloring Rules – “Butt Uglies” Customize Columns (Edit > Prefs) Pkt Length, DSCP, Winsize, Pkt Annotations Disable TCP Reassembly Disable Relative Seq# Set Time for your use case Create Filtering Buttons

Demonstrate Custom Profiles

Lab: Setup Your Rvbd Profile Create New Profile Customize Columns Add pkt len, dscp, delta time, pkt info… Create List of Common Expressions tcp.options.rvbd.probe, .trpy, tcp.port==708… Create Filter Buttons Most used expressions Customize Coloring Rules identify packets of most interest (probes, trpy, oob slice…) Shortcut: Use provided Riverbed profile.

Isolate the Problem by OSI Layer

Problem Solving MTU Issues / Fragmentation Speed / Duplex Autodiscovery / NAT / OOB Slice, etc Zone Based Firewall Retrans / Loss / Queue Depth Issues QoS Trace Analysis HTTP Analysis SSL Analysis Authentication Integration, etc

Lab Network Client CFE FW .1 .1 .10 Pri .62 Inpath .2 A: .254 RTR 192.168.34.0/24 10.10.10.0/24 Client CFE FW (NAT) .1 .1 .10 Pri .62 Inpath .2 A: .254 RTR 192.168.32.0/24 B .254 Svr (USB) Switch SFE Hub .253 Pri .60 Inpath .61 Wireshark (Eth0)

Check directly connected NICs Speed / Duplex Check Steelhead NICs Check directly connected NICs If applications are slower then before Steelheads, it is most likely a Layer 2/3 issue. Useful filters: tcp.analysis.flags tcp.analysis…. Leverage IO Graphs or TCP Conversation List to visualize

MTU / Fragmentation Prior to deploying Steelheads it can be useful to baseline the environment look for fragmentation: ip.flags.df MSFT: ping –l 1480 <target> Look for optimized connections that setup, but then stall, fail or are very slow. Look for large pkts leaving the source, but never getting to the destination. Get LAN/WAN captures from both Steelheads Start with display filter tcp.analysis.flags Use tcp.analysis.flags in IO Graphs for a visual with the ticks to pkts Duplicate ACKs followed by a RST is a common signature. UsingWiresharkToSolveRealWorldProblems_CapFiles ip_frag_source.pcap

Autodiscovery / NAT / OOB Slice Packet Richocet / Layer 3 Issue Probe Stripping Incorrect Inpath Rules OOB Connection Dropping/Resetting in-path FT+R rule and in-path peering oobtransparency mode full = spoofing with client port of 708 in-path peering oobtransparency mode none = oob port 7800

Retrans / Loss / Queue Depth Issues See following slides….

Queue Comparison (from Linktropy Mini UI)

Short Queue – Wireshark Graph

Proper Queue – Wireshark Graph

Transparency Analysis Diving Deeper into Rvbd Inner Channel Analysis Find traffic from a specific Steelhead tcp.options.rvbd.trpy.dst.ip == <inpath IP> tcp.options.rvbd.trpy.src.ip == <inpath IP>

Transparency – Steelhead Headers

When in doubt disable HTTP optimization Step into HTTP optimization HTTP Analysis When in doubt disable HTTP optimization For single server OR whole HTTP blade? Step into HTTP optimization Walk before you Run Determine the variable that triggers the issue Document, consider options, move forward You’re not along, leverage the Rvbd Team Use Logs, Fiddler & Wireshark together

Watch for improper tagging Watch for oobslice in wrong Class of Svc QoS Trace Analysis Add DSCP to Column Watch for improper tagging Classification / Reclassification Impact? Watch for oobslice in wrong Class of Svc Watch for oobslice dropping under heavy congestion

Lab: PCAP Analysis Lab 1 Problem: Steelheads aren’t optimizing traffic. Goal: Determine they aren’t optimizing. Files: Lab1_Steelheads_Not_Optimizing_CONUS_SIDE_WAN.pcap Lab1_Steelheads_Not_Optimizing_OCONUS_SIDE_WAN.pcap

Lab: PCAP Analysis Lab 2 Problem: Steelhead optimization is slower than expected over satcom link. Goal: Determine what may be impacting performance. File: Lab2_Steelheads_Slower_than_Expected_cfe_wan0_0.cap0

Goal: Determine the issue, and next steps. File: PCAP Analysis Lab 3 Problem: SCPS between a Steelhead and TurboIP is having problems, and some transfers aren’t completing. Goal: Determine the issue, and next steps. File: Lab3_Steelheads_with_SCPS_to_TurboIP_Issue_4Mbps_720ms_0BER.cap

Transfer Failure issue – misconfigured? Is this a bug or not? Transfer Failure issue – misconfigured? Unusual Inner Traffic issue or normal? HTTP misconfigured or broken? This was a bug. Details are in slide notes. Transfer Failure = 8.0 bug with SCPS – related to time stamping Unusual Inner Traffic = run away connection pooling bug in 7.0.3 HTTP bug = SSL proxy bug when satcom delay is present filter: tcp.port==58950 SFE HTTP GET pkt 376, 200 OK pkt 423 CFE HTTP Get pkt 296 , but no corresponding 200 OK, first part seen in session pkt 303

Where to Get Help Docs: Videos: http:splash.riverbed.com http://www.wireshark.org/docs/ http://www.wiresharktraining.com/ http://www.wiresharkbook.com/ Videos: YouTube WireShark Videos thetechfirm Wireshark Videos www.wiresharktraining.com - Laura Chappell https://wcnaportal.com/ Tony Fortunato – www.ilovemytool.com & Wireshark postings Sharkfest Videos & Presentations http:splash.riverbed.com Books 24x7 (requires login) Practical Packet Analysis