A Machine Learning Approach to Detecting Attacks by Identifying Anomalies in Network Traffic A Dissertation by Matthew V. Mahoney Major Advisor: Philip K. Chan

Overview Related work in intrusion detection Approach Experimental results –Simulated network –Real background traffic Conclusions and future work

Limitations of Intrusion Detection Host based (audit logs, virus checkers, system calls (Forrest 1996)) –Cannot be trusted after a compromise Network signature detection (SNORT (Roesch 1999), Bro (Paxson 1998)) –Cannot detect novel attacks –Alarms occur in bursts Address/port anomaly detection (ADAM (Barbara 2001), SPADE (Hoagland 2000), eBayes (Valdes & Skinner 2000)) –Cannot detect attacks on public servers (web, mail)

Anomaly SignatureNetwork Host User System BSM Virus Detection SNORT Bro Audit Logs Firewalls SPADE ADAM eBayes Network Protocol Anomaly Detection Intrusion Detection Dimensions Model Data Method

Problem Statement Detect (not prevent) attacks in network traffic No prior knowledge of attack characteristics Model of normal traffic IDS Training – no known attacks Test data with attacksAlarms

Approach 1.Model protocols (extend user model) 2.Time-based model of “bursty” traffic 3.Learn conditional rules 4.Batch and continuous modeling 5.Test with simulated attacks and real background traffic

Approach 1. Protocol Modeling User model (conventional) –Source address for authentication –Destination port to detect scans Protocol model (new) –Unusual features (more likely to be vulnerable) –Client idiosyncrasies –IDS evasion –Victim’s symptoms after an attack

Example Protocol Anomalies AttackHow detected Category Teardrop – overlapping IP fragments crashes target IP fragments Unusual feature Sendmail – buffer overflow gives remote root shell Lower case mail Idiosyn- crasy FIN scan (portsweep) - FIN packets not logged FIN with- out ACK Evasion ARPpoison – Forged replies to ARP-who-has Interrupt- ed TCP Victim symptoms

Approach 2 -Non-Poisson Traffic Model (Paxson & Floyd, 1995) Events occur in bursts on all time scales Long range dependency No average rate of events Event probability depends on –The average rate in the past –And the time since it last occurred

Time-Based Model If port = 25 then word1 = HELO or EHLO Anomaly: any value never seen in training Score = tn/r –t = time since last anomaly for this rule –n = number of training instances (port = 25) –r = number of allowed values (2) Only the first anomaly in a burst receives a high score

Example Training = AAAABBBBAA Test = AACCC C is an anomaly r/n = average rate of training anomalies = 2/10 (first A and first B) t = time since last anomaly = 9, 1, 1 Score (C) = tn/r = 45, 5, 5

Approach 3. Rule Learning 1.Sample training pairs to suggest rules with n/r = 2/1 2.Remove redundant rules, favoring high n/r 3.Validation: remove rules that generate alarms on attack-free traffic

Learning Step 1 - Sampling PortWord1Word2Word3 80GET/HTTP/1.0 80GET/index.htmlHTTP/1.0 If port = 80 then word1 = GET word3 = HTTP/1.0 If word3 = HTTP/1.0 and word1 = GET then port = 80

Learning Step 2 – Remove Redundant Rules (Sorted by n/r) R1: if port = 80 then word1 = GET (n/r = 2/1, OK) R2: word1 = HELO or GET (n/r = 3/2, OK) R3: if port = 25 then word1 = HELO (n/r = 1/1, remove) R4: word2 = pascal, /, or /index.html (n/r = 3/3, OK) PortWord1Word2Word3 25HELOpascalMAIL 80GET/HTTP/1.0 80GET/index.htmlHTTP/1.0

Learning Step 3 – Rule Validation Training (no attacks) – Learn rules, n/r Validation (no attacks) – Discard rules that generate alarms Testing (with attacks) TrainValidateTest

Approach 4. Continuous Modeling No separate training and test phases Training data may contain attacks Model allows for previously seen values Score = tn/r + t i /f i –t i = tine since value i last seen –f i = frequency of i in training, f i > 0 No validation step

Implementation ModelDataCon- ditions Valid- ation Score PHADPacket headers NoneNotn/r ALADTCP streams Server, port Notn/r LERADTCP streams LearnedYestn/r NETADPacket bytes ProtocolYestn/r + t i /f i

Example Rules (LERAD) /1 if SA3=172 then SA2 = /1 if SA2=016 then SA3 = /1 if F1=.UDP then F3 = /1 if F1=.UDP then F2 = /1 if F3=. then F1 =.UDP /1 if F3=. then DUR = /1 if DA0=100 then DA1 = /1 if W6=. then W7 = /1 if W5=. then W6 = /1 if W4=. then W8 = /1 if W4=. then W5 = /1 if DA1=118 then W /1 if DA1=118 then SA1 = /1 if SP=520 then DP = /1 if SP=520 then W /1 if DP=520 then DA1 = /1 if DA1=118 SA1=112 then LEN = /2 if F2=.AP then F1 =.S.AS /1 if then DP = /6 if then DA1 = /6 if then F1 =.UDP.S.AF.ICMP.AS.R /1 if W3=.HELO then W /1 if F1=.S W3=.HELO then DP = /1 if DP=25 W5=.MAIL then W3 =.HELO

1999 DARPA IDS Evaluation (Lippmann et al. 2000) 7 days training data with no attacks 2 weeks test data with 177 visible attacks Must identify victim and time of attack SunOSSolarisLinuxWinNT IDS Victims Internet (simulated) Attacks

Attacks Detected at 10 FA/Day

Unlikely Detections Attacks on public servers (web, mail, DNS) detected by source address Application server attacks detected by packet header fields U2R (user to root) detected by FTP upload

Unrealistic Background Traffic Source Address, client versions (too few clients) TTL, TCP options, TCP window size (artifacts) Checksum errors, “crud”, invalid keywords and values (too clean) r Time Simulated Real

5. Injecting Real Background Traffic Collected on a university departmental web server Filtered: truncated inbound client traffic only IDS modified to avoid conditioning on traffic source SunOSSolarisLinuxWinNT IDS Internet (simulated and real) Attacks Real web server

Mixed Traffic: Fewer Detections, but More are Legitimate

Detections vs. False Alarms (Simulated and Combined Traffic)

Results Summary Original 1999 evaluation: 40-55% detected at 10 false alarms per day NETAD (excluding U2R): 75% Mixed traffic: LERAD + NETAD: 30% At 50 FA/day: NETAD: 47%

Contributions 1.Protocol modeling 2.Time based modeling for bursty traffic 3.Rule learning 4.Continuous modeling 5.Removing simulation artifacts

Limitations False alarms – Unusual data is not always hostile Rule learning requires 2 passes (not continuous) Tests with real traffic are not reproducible (privacy concerns) Unlabeled attacks in real traffic –GET /MSADC/root.exe?/c+dir HTTP/1.0 –GET /scripts/..%255c%255c../winnt/system32/cmd.exe?/c+dir

Future Work Modify rule learning for continuous traffic Add other attributes User feedback (should this anomaly be added to the model?) Test with real attacks

Acknowledgments Philip K. Chan – Directing research Advisors – Ryan Stansifer, Kamel Rekab, James Whittaker Ongoing work –Gaurav Tandon – Host based detection using LERAD (system call arguments) –Rachna Vargiya – Parsing application payload –Hyoung Rae Kim – Payload lexical/semantic analysis –Muhammad Arshad – Outlier detection in network traffic DARPA – Providing funding and test data