1. Real World Troubleshooting with Wireshark

2. Agenda
Packet Capture Family of Tools
Where to Capture
Filters
Setting Up Your Environment
Problem Solving
Where to Get Help

3. Packet Capture Tools
Steelhead tcpdump / embedded Shark
Pilot / Shark / vShark
Wireshark (via WinPcap)
tshark
WinPcap / WinDump

4. Wireshark UI Tour

5. Pilot UI Tour Ribbons Source Panel View ToolTip Main Events Workspace
200+
Searchable
Views
Fast Troubleshooting via Select-able / Drill-able Charts
Efficient Operation via Robust Context Menus
Two Click Export to Wireshark with Filtering
Enables Very Fast & Efficient Problem Resolution
Timeline

6. tshark
C:\Program Files\Wireshark>tshark –help Usage: tshark [options] ... Capture interface: -i <interface> name or idx of interface (def: first non-loopback) -f <capture filter> packet filter in libpcap filter syntax -s <snaplen> packet snapshot length (def: 65535) -p don't capture in promiscuous mode -B <buffer size> size of kernel buffer (def: 1MB) -y <link type> link layer type (def: first appropriate) -D print list of interfaces and exit -L print list of link-layer types of iface and exit Capture stop conditions: -c <packet count> stop after n packets (def: infinite) -a <autostop cond.> ... duration:NUM - stop after NUM seconds filesize:NUM - stop this file after NUM KB files:NUM - stop after NUM files ………..

7. Where & What to Capture It Depends Capture Liberally
WAN Opt
ADC Issue
App Issue
Capture Liberally
set liberal snaplen
Filter Aggressively
Where:
Points of Interest or Transition
Proxies, ADC, FW, Client, Server, etc.
Steelheads: SFE & CFE LAN/WAN
Stingray: Both frontend & backend int.
Client & Server
What:
Healthy vs Problem
LAN vs WAN
FW vs No FW

8. Why get a healthy baseline capture?

9. Filters Capture Filters Display Filters ether host 00:04:13:00:09:a3
host and host
src net mask
src net /24
tcp len <= 70
tcp
tcp host
tcp port 7800
tcp portrange 20-25
tcp host port 80
ip6
vlan 223
not, and, or
!, &&, ||
Parenthesis available to group values, but need be escaped when used at cli
Display Filters
ip.addr ==
!ip.addr ==
ip.addr== && ip.addr==
tcp
tcp.flags.syn
tcp.port == 80
tcp.analysis.flags
tcp.options.rvbd.probe
tcp.options.rvbd.probe.prober == <SH_IP>
tcp.options.rvbd.trpy
tcp.options.scps
vlan.id==<#>

10. Capture Options v1.8 can capture from multiple interfaces
Setup a liberal capture filter
Note in Wireshark you can now capture from multiple interfaces.

11. Display Filters

12. Display Filters – Right Click

13. Demonstrate Filtering

14. Lab: Capture Filtering
Write filters to capture the following:
tcp traffic btwn test machine and Class B subnet
HTTP traffic for test machine
Traffic on either VLAN 55 or 222

15. Lab: Capture Filtering (Answers)
tcp host and net /16
tcp port 80 and host
vlan 55 or vlan 222

16. Lab: Display Filtering
Write filters to display the following:
Analyze all vlans except 1
TCP traffic with possible problems
Find all TCP SYN packets
Show Steelhead probes
Steelhead Full Transparency traffic over WAN
All Probes from Client Steelhead
Show all SCPS probes

17. Lab: Display Filtering (Answers)
!vlan.id==1
tcp.analysis.flags
tcp.flags.syn==1
tcp.options.rvbd.probe
tcp.options.rvbd.trpy
tcp.options.rvbd.probes.prober==
tcp.options.scps

18. Setting Up Your Environment
Setup Profiles for Different Use Cases
WAN Opt
Application Troubleshooting
Web
VoIP
Wireless
VLAN analysis

19. Customize Your Profiles
Coloring Rules – “Butt Uglies”
Customize Columns (Edit > Prefs)
Pkt Length, DSCP, Winsize, Pkt Annotations
Disable TCP Reassembly
Disable Relative Seq#
Set Time for your use case
Create Filtering Buttons

20. Demonstrate Custom Profiles

21. Lab: Setup Your Rvbd Profile
Create New Profile
Customize Columns
Add pkt len, dscp, delta time, pkt info…
Create List of Common Expressions
tcp.options.rvbd.probe, .trpy, tcp.port==708…
Create Filter Buttons
Most used expressions
Customize Coloring Rules
identify packets of most interest (probes, trpy, oob slice…)
Shortcut: Use provided Riverbed profile.

22. Isolate the Problem by OSI Layer

23. Problem Solving
MTU Issues / Fragmentation
Speed / Duplex
Autodiscovery / NAT / OOB Slice, etc
Zone Based Firewall
Retrans / Loss / Queue Depth Issues
QoS Trace Analysis
HTTP Analysis
SSL Analysis
Authentication Integration, etc

24. Lab Network Client CFE FW .1 .1 .10 Pri .62 Inpath .2 A: .254 RTR
/24
/24
Client
CFE FW
(NAT) .1 .1
.10
Pri .62
Inpath .2
A: .254
RTR
/24
B .254
Svr (USB)
Switch
SFE
Hub
.253
Pri .60
Inpath .61
Wireshark
(Eth0)

25. Check directly connected NICs
Speed / Duplex
Check Steelhead NICs
Check directly connected NICs
If applications are slower then before Steelheads, it is most likely a Layer 2/3 issue.
Useful filters:
tcp.analysis.flags
tcp.analysis….
Leverage IO Graphs or TCP Conversation List to visualize

26. MTU / Fragmentation
Prior to deploying Steelheads it can be useful to baseline the environment
look for fragmentation: ip.flags.df
MSFT: ping –l 1480 <target>
Look for optimized connections that setup, but then stall, fail or are very slow.
Look for large pkts leaving the source, but never getting to the destination.
Get LAN/WAN captures from both Steelheads
Start with display filter tcp.analysis.flags
Use tcp.analysis.flags in IO Graphs for a visual with the ticks to pkts
Duplicate ACKs followed by a RST is a common signature.
UsingWiresharkToSolveRealWorldProblems_CapFiles
ip_frag_source.pcap

27. Autodiscovery / NAT / OOB Slice
Packet Richocet / Layer 3 Issue
Probe Stripping
Incorrect Inpath Rules
OOB Connection Dropping/Resetting
in-path FT+R rule and
in-path peering oobtransparency mode full = spoofing with client port of 708
in-path peering oobtransparency mode none = oob port 7800

28. Retrans / Loss / Queue Depth Issues
See following slides….

29. Queue Comparison (from Linktropy Mini UI)

30. Short Queue – Wireshark Graph

31. Proper Queue – Wireshark Graph

32. Transparency Analysis
Diving Deeper into Rvbd Inner Channel Analysis
Find traffic from a specific Steelhead
tcp.options.rvbd.trpy.dst.ip == <inpath IP>
tcp.options.rvbd.trpy.src.ip == <inpath IP>

33. Transparency – Steelhead Headers

34. When in doubt disable HTTP optimization Step into HTTP optimization
HTTP Analysis
When in doubt disable HTTP optimization
For single server OR whole HTTP blade?
Step into HTTP optimization
Walk before you Run
Determine the variable that triggers the issue
Document, consider options, move forward
You’re not along, leverage the Rvbd Team
Use Logs, Fiddler & Wireshark together

35. Watch for improper tagging Watch for oobslice in wrong Class of Svc
QoS Trace Analysis
Add DSCP to Column
Watch for improper tagging
Classification / Reclassification Impact?
Watch for oobslice in wrong Class of Svc
Watch for oobslice dropping under heavy congestion

36. Lab: PCAP Analysis
Lab 1
Problem: Steelheads aren’t optimizing traffic.
Goal: Determine they aren’t optimizing.
Files:
Lab1_Steelheads_Not_Optimizing_CONUS_SIDE_WAN.pcap
Lab1_Steelheads_Not_Optimizing_OCONUS_SIDE_WAN.pcap

37. Lab: PCAP Analysis
Lab 2 Problem: Steelhead optimization is slower than expected over satcom link. Goal: Determine what may be impacting performance. File: Lab2_Steelheads_Slower_than_Expected_cfe_wan0_0.cap0

38. Goal: Determine the issue, and next steps. File:
PCAP Analysis
Lab 3
Problem: SCPS between a Steelhead and TurboIP is having problems, and some transfers aren’t completing.
Goal: Determine the issue, and next steps.
File:
Lab3_Steelheads_with_SCPS_to_TurboIP_Issue_4Mbps_720ms_0BER.cap

39. Transfer Failure issue – misconfigured?
Is this a bug or not?
Transfer Failure issue – misconfigured?
Unusual Inner Traffic issue or normal?
HTTP misconfigured or broken?
This was a bug. Details are in slide notes.
Transfer Failure = 8.0 bug with SCPS – related to time stamping
Unusual Inner Traffic = run away connection pooling bug in 7.0.3
HTTP bug = SSL proxy bug when satcom delay is present
filter: tcp.port==58950
SFE HTTP GET pkt 376, 200 OK pkt 423
CFE HTTP Get pkt 296 , but no corresponding 200 OK, first part seen in session pkt 303

40. Where to Get Help Docs: Videos: http:splash.riverbed.com
Videos:
YouTube WireShark Videos
thetechfirm Wireshark Videos
- Laura Chappell
Tony Fortunato – & Wireshark postings
Sharkfest Videos & Presentations
Books 24x7 (requires login)
Practical Packet Analysis