SHARKFEST ‘10 | Stanford University | June 14–17, 2010 To the Terabyte and Beyond! Leveraging Pilot and Wireshark to Analyze Truly Massive Packet Traces June 17, 2010 Loris Degioanni CTO | CACE Technologies SHARKFEST ‘10 Stanford University June 14-17, 2010

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Packet Aquisition

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Capture Card Dedicated card is essential – No network stack overhead – Minimizes copies – Optimizes locality – Filtering capability in the card normally not really useful Unless in some unusual conditions, the application wants to see everything PCI bus is the only resource that card filtering optimizes Any tap nowadays can do basic filtering – Small packets is the worst condition CACE Turbocap – Hybrid between home-built and off the shelf – No unnecessary features (who needs filtering?) – Affordable price

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 CPU Bottlenecks – CPU clock (expensive) – Number of CPUS (cheap) Multi-threading hard to leverage when capturing and processing network packets – Network monitoring is intrinsically sequential Locking is evil – Doing things more than once is better than locking At 10Gbps, cache coherency is a big deal Small packets is the worst condition

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Disk Bottlenecks – Single disk write speed – Number of spindles – Raid Controller – Big packets is the worst condition Solid State? Not a good idea yet – Single disk performance is not really the bottleneck – Cost is an important factor when you build a system with tens of disks – Reliability not as proven as the old magnetic disks

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Disk write speed based on position

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 I can capture a lot of packets. Now what? Read of packets must be non-disruptive! Even if I stop the capture process, since I was writing at full speed, reading the data is going to take around the same time of writing it – Read needs to be localized – I need high level visibility to reach the point I need Indexing

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Standalone card vs. kit A network card nowadays is not enough to build a functional packet capture system.

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Indexing While capturing, on a Shark Appliance capture job On a trace file, after the fact Summary of the network traffic – Volume, talkers and protocol information – Coordinated with the packet store – “Netflow on steroids” Designed to be extremely efficient in terms of disk usage Coordinated with the packet store

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Indexing Index file Time intervals File Positions Time index pcap file Index entry Packet