Securing and Monitoring 10GbE WAN Links Steven Carter Center for Computational Sciences Oak Ridge National Laboratory

Disclaimer Oak Ridge National Laboratory does not endorse any particular product. This presentation merely details our experience and chosen course of action (i.e. I am not a patsy for Force10).

Requirements Wire rate intrusion detection (i.e. 20Gb/s) Little or no latency Low administrative/development overhead Flexible (used for IDS and protocol monitoring) Scalable (We have 5+ 10G links that we would like to monitor) Affordable

Approaches Divide and Conquer: Use a piece of network equipment (e.g. Juniper Router) to divide the stream of packets by some attribute (e.g. destination port) into smaller, more easily handled streams for processing.

Approaches (Cont.) Host intensive: Send the full (or possibly filtered) stream to the host CPU for inspection. NIC intensive: The NIC does the packet inspection.

The Contenders Intel, Neterion, Chelsio 10G NICs Endace DAG 6.2SE Force10 P-Series (formally MetaNetworks)

Initial Pros/Cons Standard 10G NICS Inexpensive Single host unable to keep up with full rate, full duplex connection Endace DAG 6.2SE Offload allows single host to inspect more traffic (~13Gb/s), but you need a beefy host. Timestamps Only available with 1310nm optics Expensive

Initial Pros/Cons (cont) Force10 P-Series Less expensive Compete offload Scalable Can block packets if used in-line Supports too few snort rules (700 shared between 2 channels) Long compile time PCI Bus (1Gb/s b/w the card and the host)

Initial Test Setup HostHost P-Series DAG Switch HostHost Switch Optical TapPort Mirror HostHost Simulated Nefarious Traffic Saturating Traffic (~10Gb/s)

DAG Results Circular Buffer started overflowing ~5Gb/s (could likely be tuned better) Not a generic network interface (Either use the provided dag* utilities or a special version of libpcap) Only one tool can be used at a time

P-Series Results Able to handle full rate (~10Gb/s) Interface presented as generic interface (i.e. can run Bro, Snort, and tcpdump simultaneously) Supports too few snort rules (700 shared between 2 channels)... you have to choose well Long compile time (long test cycles)

Our Decision The DAG 6.2SE is way too expensive for what you get. We could not afford to use it on 5+ links The Force10 P-Series had the best strategy and would scale best to fit our needs. Although the card doubled in price, the next generation is slated to have stateful firewall features, more real estate, and a PCI-X (should be PCIe) interface. This makes for a very cost effective, flexible, firewall, IPS, and protocol analysis solution.

Working Around the Rule Limitation Send known low-rate traffic (ICMP, DNS, HTTP, etc.) to the host CPU to be compared against full complement of Snort rules. Send the first few packets of every connection to the host CPU to be compared against full compliment of Snort rules (either via state register or through the API). Use the rules on the card for high-rate traffic.

Final Setup 3U Dual 2.8Ghz Opteron 8 GB RAM 3TB of internal RAID 5 storage 2 P-Series cards (room for a third)

Final Testing BorderRouter Host P-SeriesHost Switch Saturating Traffic (~9Gb/s) “Real” Internet Traffic

Conclusion The Force10 P-Series takes a good approach to the problem. It allows us to secure and monitor several 10G links for a reasonable price. The next generation is even more promising allowing the merging of IPS with firewalling capabilities.

Questions?