Defeating Large Scale Attacks: Technology and Strategies for Global Network Monitoring The NetViewer Experiment PAVG in collaboration with Networking Systems R. Kamath, E. Jang, D. Luckham

2 Project Goals Detect system misuse on a global level User re-configurable and flexible Hierarchical organization of monitors Correlation of distributed monitors Monitor activity from diverse sources Monitor at multiple levels of abstraction

3 Stanford NetViewer Experiment Uses Stanford Rapide Toolset Uses Complex Event Processing technology Uses Talarian’s SmartSockets TM middleware for distributed processing FOR MORE INFO...

4 NetViewer Experiment setup

5 SUNet Campus Network Undergrad Education Business School Admin Host 1 Computer Center 1 Computer Center 2 Admin Host 2 Stanford Hospital Grad. Education Redundancy Gateway Redundancy Gateway Redundancy Gateway Redundancy Gateway Core Gateway Core Gateway Internet To FlowCollector

6 Complex Event Processing Accept network ‘events’ from any source –CISCO NetFlow FlowCollector, tcpdump Correlates events based on content and temporal relationship between events Event Processing Agents (EPAs) connected in an Event Processing Network (EPNs) Both post-mortem and real-time processing

7 Event Processing Agents (EPAs) -- Loggers and Filters Loggers –Convert external data into events –E.g. CISCO FlowCollector logs to events Filters –Select a subset of events based on pattern –E.g. Only connections from Stanford hosts

8 EPAs-- Maps and Viewers Maps –Search for patterns in input events –Generate appropriate output events –E.g. look for IP scans and generate alarms Viewers –Graphical display of data in events –Tables, Bar Graphs

9 RapNet User interface RapNet –Graphical Interface to NetViewer tool –Easy access to EPA and EPN library –Easy re-configuration of EPAs –Easy modification of EPNs –Construct new EPNs using EPAs

10 NetViewer running under RapNet

11 Hierarchical monitoring Two types of hierarchy –Abstraction hierarchy NetViewer monitors data at different abstraction levels –Topological hierarchy NetViewers at different locations NetViewers at different levels communicate using SmartSockets middleware General case: arbitrary network of monitors

12 Network Abstraction Hierarchy Application layer –Host-based monitoring –Data exchanged by SMTP, TELNET, FTP, HTTP protocols Transport layer –Data exchanged by TCP/IP suite of protocols Network layer –Router-based monitoring –IP and UDP packets

13 Topological Hierarchy -- multiple gateways example Distributed processing of data Each NetViewer at level 1 monitors data from a different gateway Results (e.g. top 10 IPs) from level 1 NetViewers sent to level2 NetViewers Level 2 NetViewers correlate the results of level 1 NetViewers –E.g. compute top 10 IPs over all gateways

14 Distributed monitoring on SUNet Admin host Core gateway Admin host Press gateway SmartSockets over SUNet Sender running NetViewer 1 Sender running NetViewer 2 Receiver running NetViewer 3

15 Current Status -- EPAs Library of Event Processing Agents (EPAs) –Traffic categories Web, Mail, DNS, ftp … –Scan Detectors IP scan, Port scan –Policy violation detectors Access to restricted hosts Access to restricted ports on hosts –Traffic event filters Web, Mail, Hosts, Networks

16 Current Status -- EPNs Library of Viewers –Tables –Bar graphs –Pie charts Library of Event Processing Networks (EPNs) –Network of EPAs –Graphical viewers to display results

17 Research Directions Hierarchical monitoring –Data sources from different layers –Correlation of results from multiple NetViewers Accept more input formats Distributed processing –Assign individual EPAs within a NetViewer to run on different machines Expand EPA library –Work on mail spam detection

18 Experiment results on SUNet NetViewer used to process router logs –Real-time performance of about 1000 log records/sec Generated traffic statistics –Top IPs by packets or bytes –Classification of traffic into categories such as internal/external, web/mail/DNS etc. Intrusion detection –Detected IP and port scans –Well-known attack signature e.g. finger attack

19 Related projects -- CIDF Correlates information from multiple intrusion detectors –Reduces false alarms –Prioritizes network warnings Part of the DARPA Common Intrusion Detection Framework (CIDF) –Multiple intrusion detectors in cyber battlefield FOR MORE INFO...

20 Overview of the CIDF project Goal Experiment with semantic interoperability of different components in CIDF Groups Involved Group A: produces GIDOs, questions, detailed English description of the events, and the answers to the questions. Group B: gets 10 scenarios and produces 10 GIDOs describing the scenarios. Group C: gets the questions and high level scenarios from B and builds the code. Then, gets 10 GIDOs and produces text answers to the questions - Stanford belongs to group C.

21 Processing GIDOs with CEP agents Make each GIDO an event Use (and fix) our existing cidfLogger Separate event processing agent called “Qagent” Provides flexible way of handling GIDOs

22 Qagent Finds an answer from a given GIDO and a query pattern. Qagent traverses the tree to find all the possible paths that can lead to the answer. The question is fed to the program as a text file with two sections: –The input file may contain a text description –Patterns to be searched from the tree. The pattern lines are preceded with Implemented in C++ (I.e. not map language) –Easier tree traversal –File input

23 Pattern Language Lists of SID separated by comma. Answer is the subtree after the last SID Attack,AttackSpecifics,IPV4Address “#true” or “#false” to get the sibling SID rather than child SID of the last SID for the answer. ByMeansOf,Attack#true ‘^’ to indicate that the SID is one of the base SID that applies to all other parts of the pattern ^And,^Copy,Outcome,ReturnCode?success=FileSource,File Name

24 Examples Event1 Brief description: This is an attack that began on Monday, May 24, at 12:44. What is the certainty of this Attack,Certainty ( Attack ( Initiator ( IPV4Address ) ) ( Target ( IPV4Address ) ) ( AttackSpecifics ( Certainty 100 ) ( Severity 50 ) ( AttackID f ) ) ( When ( BeginTime Mon May 24 12:44: PDT ) ( EndTime Mon May 24 12:44: PDT ) )

25 Team Members Rajesh Kamath David Luckham Eunhei Jang John Kenney James Vera