1. Malicious Code Detection and Security Applications
Prof. Bhavani Thuraisingham
The University of Texas at Dallas
September 8, 2008
Lecture #5

2. Outline Data mining overview
Intrusion detection and Malicious code detection (worms and virus)
Digital forensics and UTD work
Algorithms for Digital Forensics

3. What is Data Mining? Information Harvesting Knowledge Mining
Knowledge Discovery
in Databases
Data Archaeology
Data Dredging
Database Mining
Knowledge Extraction
Data Pattern Processing
Information Harvesting
Siftware
The process of discovering meaningful new correlations, patterns, and trends by sifting through large amounts of data, often previously unknown, using pattern recognition technologies and statistical and mathematical techniques
(Thuraisingham, Data Mining, CRC Press 1998)

4. What’s going on in data mining?
What are the technologies for data mining?
Database management, data warehousing, machine learning, statistics, pattern recognition, visualization, parallel processing
What can data mining do for you?
Data mining outcomes: Classification, Clustering, Association, Anomaly detection, Prediction, Estimation, . . .
How do you carry out data mining?
Data mining techniques: Decision trees, Neural networks, Market-basket analysis, Link analysis, Genetic algorithms, . . .
What is the current status?
Many commercial products mine relational databases
What are some of the challenges?
Mining unstructured data, extracting useful patterns, web mining, Data mining, security and privacy

5. Data Mining for Intrusion Detection: Problem
An intrusion can be defined as “any set of actions that attempt to compromise the integrity, confidentiality, or availability of a resource”.
Attacks are:
Host-based attacks
Network-based attacks
Intrusion detection systems are split into two groups:
Anomaly detection systems
Misuse detection systems
Use audit logs
Capture all activities in network and hosts.
But the amount of data is huge!

6. Misuse Detection
Misuse Detection

7. Problem: Anomaly Detection

8. Our Approach: Overview
Training
Data
Class
Hierarchical
Clustering (DGSOT)
SVM Class Training
Testing
DGSOT: Dynamically growing self organizing tree
Testing Data

9. Hierarchical clustering with SVM flow chart
Our Approach: Hierarchical Clustering
Our Approach
Hierarchical clustering with SVM flow chart

10. Results Training Time, FP and FN Rates of Various Methods Methods
Average
Accuracy
Total Training Time
Average FP
Rate (%)
Average FN
Random Selection
52%
0.44 hours 40 47
Pure SVM
57.6%
17.34 hours
35.5 42
SVM+Rocchio Bundling
51.6%
26.7 hours
44.2 48
SVM + DGSOT
69.8%
13.18 hours
37.8
29.8

11. Introduction: Detecting Malicious Executables using Data Mining
What are malicious executables?
Harm computer systems
Virus, Exploit, Denial of Service (DoS), Flooder, Sniffer, Spoofer, Trojan etc.
Exploits software vulnerability on a victim
May remotely infect other victims
Incurs great loss. Example: Code Red epidemic cost $2.6 Billion
Malicious code detection: Traditional approach
Signature based
Requires signatures to be generated by human experts
So, not effective against “zero day” attacks

12. State of the Art in Automated Detection
Automated detection approaches:
Behavioural: analyse behaviours like source, destination address, attachment type, statistical anomaly etc.
Content-based: analyse the content of the malicious executable
Autograph (H. Ah-Kim – CMU): Based on automated signature generation process
N-gram analysis (Maloof, M.A. et .al.): Based on mining features and using machine learning.

13. Our New Ideas (Khan, Masud and Thuraisingham)
Content -based approaches consider only machine-codes (byte-codes).
Is it possible to consider higher-level source codes for malicious code detection?
Yes: Diassemble the binary executable and retrieve the assembly program
Extract important features from the assembly program
Combine with machine-code features

14. Feature Extraction Binary n-gram features
Sequence of n consecutive bytes of binary executable
Assembly n-gram features
Sequence of n consecutive assembly instructions
System API call features
DLL function call information

15. The Hybrid Feature Retrieval Model
Collect training samples of normal and malicious executables.
Extract features
Train a Classifier and build a model
Test the model against test samples

16. Hybrid Feature Retrieval (HFR)
Training

17. Hybrid Feature Retrieval (HFR)
Testing

18. Feature Extraction
Binary n-gram features
Features are extracted from the byte codes in the form of n- grams, where n = 2,4,6,8,10 and so on.
Example:
Given a 11-byte sequence: abcdef012345,
The 2-grams (2-byte sequences) are: 0123, 2345, 4567, 6789, 89ab, abcd, cdef, ef01, 0123, 2345
The 4-grams (4-byte sequences) are: , , ab,...,ef and so on....
Problem:
Large dataset. Too many features (millions!).
Solution:
Use secondary memory, efficient data structures
Apply feature selection

19. Feature Extraction
Assembly n-gram features
Features are extracted from the assembly programs in the form of n-grams, where n = 2,4,6,8,10 and so on.
Example:
three instructions
“push eax”; “mov eax, dword[0f34]” ; “add ecx, eax”;
2-grams
(1) “push eax”; “mov eax, dword[0f34]”;
(2) “mov eax, dword[0f34]”; “add ecx, eax”;
Problem:
Same problem as binary
Solution:
Same solution

20. Feature Selection Select Best K features
Selection Criteria: Information Gain
Gain of an attribute A on a collection of examples S is given by

21. Experiments Dataset Dataset1: 838 Malicious and 597 Benign executables
Collected Malicious code from VX Heavens (
Disassembly
Pedisassem ( )
Training, Testing
Support Vector Machine (SVM)
C-Support Vector Classifiers with an RBF kernel

22. Results HFS = Hybrid Feature Set BFS = Binary Feature Set
AFS = Assembly Feature Set

23. Results HFS = Hybrid Feature Set BFS = Binary Feature Set
AFS = Assembly Feature Set

24. Results HFS = Hybrid Feature Set BFS = Binary Feature Set
AFS = Assembly Feature Set

25. Future Plans System call: seems to be very useful.
Need to Consider Frequency of call
Call sequence pattern (following program path)
Actions immediately preceding or after call
Detect Malicious code by program slicing
requires analysis

26. Data Mining for Buffer Overflow Introduction
Goal
Intrusion detection.
e.g.: worm attack, buffer overflow attack.
Main Contribution
'Worm' code detection by data mining coupled with 'reverse engineering'.
Buffer overflow detection by combining data mining with static analysis of assembly code.

27. Background What is 'buffer overflow'?
A situation when a fixed sized buffer is overflown by a larger sized input.
How does it happen?
example:
char buff[100];
gets(buff);
memory
buff
Stack
Input string

28. Background (cont...) Then what? buff Stack ........ char buff[100];
gets(buff);
memory
buff
Stack
Return address overwritten
Attacker's code
memory
buff
Stack
New return address points to this memory location

29. Background (cont...) So what? Program may crash or
The attacker can execute his arbitrary code
It can now
Execute any system function
Communicate with some host and download some 'worm' code and install it!
Open a backdoor to take full control of the victim
How to stop it?

30. Background (cont...) Stopping buffer overflow Preventive approaches
Detection approaches
Finding bugs in source code. Problem: can only work when source code is available.
Compiler extension. Same problem.
OS/HW modification
Capture code running symptoms. Problem: may require long running time.
Automatically generating signatures of buffer overflow attacks.

31. CodeBlocker (Our approach)
A detection approach
Based on the Observation:
Attack messages usually contain code while normal messages contain data.
Main Idea
Check whether message contains code
Problem to solve:
Distinguishing code from data

32. Severity of the problem
It is not easy to detect actual instruction sequence from a given string of bits

33. Our solution Apply data mining.
Formulate the problem as a classification problem (code, data)
Collect a set of training examples, containing both instances
Train the data with a machine learning algorithm, get the model
Test this model against a new message

34. CodeBlocker Model

35. Feature Extraction

36. Disassembly We apply SigFree tool
implemented by Xinran Wang et al. (PennState)

37. Feature extraction What is an n-gram? -Sequence of n instructions
Features are extracted using
N-gram analysis
Control flow analysis
What is an n-gram?
-Sequence of n instructions
Traditional approach:
-Flow of control is ignored
2-grams are: 02, 24, 46,...,CE
Assembly program
Corresponding IFG

38. Feature extraction (cont...)
Control-flow Based N-gram analysis
What is an n-gram?
-Sequence of n instructions
Proposed Control-flow based approach
-Flow of control is considered
2-grams are:
02, 24, 46,...,CE, E6
Assembly program
Corresponding IFG

39. Feature extraction (cont...)
Control Flow analysis. Generated features
Invalid Memory Reference (IMR)
Undefined Register (UR)
Invalid Jump Target (IJT)
Checking IMR
A memory is referenced using register addressing and the register value is undefined
e.g.: mov ax, [dx + 5]
Checking UR
Check if the register value is set properly
Checking IJT
Check whether jump target does not violate instruction boundary

40. Putting it together Why n-gram analysis?
Intuition: in general, disassembled executables should have a different pattern of instruction usage than disassembled data.
Why control flow analysis?
Intuition: there should be no invalid memory references or invalid jump targets.
Approach
Compute all possible n-grams
Select best k of them
Compute feature vector (binary vector) for each training example
Supply these vectors to the training algorithm

41. Experiments Dataset Real traces of normal messages
Real attack messages
Polymorphic shellcodes
Training, Testing
Support Vector Machine (SVM)

42. Results CFBn: Control-Flow Based n-gram feature
CFF: Control-flow feature

43. Novelty, Advantages, Limitations, Future
We introduce the notion of control flow based n-gram
We combine control flow analysis with data mining to detect code / data
Significant improvement over other methods (e.g. SigFree)
Advantages
Fast testing
Signature free operation
Low overhead
Robust against many obfuscations
Limitations
Need samples of attack and normal messages.
May not be able to detect a completely new type of attack.
Future
Find more features
Apply dynamic analysis techniques
Semantic analysis

44. Analysis of Firewall Policy Rules Using Data Mining Techniques
Firewall is the de facto core technology of today’s network security
First line of defense against external network attacks and threats
Firewall controls or governs network access by allowing or denying the incoming or outgoing network traffic according to firewall policy rules.
Manual definition of rules often result in in anomalies in the policy
Detecting and resolving these anomalies manually is a tedious and an error prone task
Solutions:
Anomaly detection:
Theoretical Framework for the resolution of anomaly;
A new algorithm will simultaneously detect and resolve any anomaly that is present in the policy rules
Traffic Mining: Mine the traffic and detect anomalies

45. Traffic Mining Firewall Log File Mining Log File Using Frequency
To bridge the gap between what is written in the firewall policy rules and what is being observed in the network is to analyze traffic and log of the packets– traffic mining
Network traffic trend may show that some rules are out- dated or not used recently
Firewall
Log File
Mining Log File
Using Frequency
Filtering
Rule
Generalization
Generic Rules
Identify Decaying
&
Dominant Rules
Edit
Firewall Rules
Firewall
Policy Rule

46. Anomaly Discovery Result
Traffic Mining Results
1: TCP,INPUT, ,ANY,*.*.*.*,80,DENY
2: TCP,INPUT,*.*.*.*,ANY,*.*.*.*,80,ACCEPT
3: TCP,INPUT,*.*.*.*,ANY,*.*.*.*,443,DENY
4: TCP,INPUT, ,ANY,*.*.*.*,22,DENY
5: TCP,INPUT,*.*.*.*,ANY,*.*.*.*,22,ACCEPT
6: TCP,OUTPUT, ,ANY,*.*.*.*,22,DENY
7: UDP,OUTPUT,*.*.*.*,ANY,*.*.*.*,53,ACCEPT
8: UDP,INPUT,*.*.*.*,53,*.*.*.*,ANY,ACCEPT
9: UDP,OUTPUT,*.*.*.*,ANY,*.*.*.*,ANY,DENY
10: UDP,INPUT,*.*.*.*,ANY,*.*.*.*,ANY,DENY
11: TCP,INPUT, ,ANY, ,22,DENY
12: TCP,INPUT, ,ANY, ,80,DENY
13: UDP,INPUT,*.*.*.*,ANY, ,ANY,DENY
14: UDP,OUTPUT, ,ANY, *,ANY,DENY
15: TCP,INPUT,*.*.*.*,ANY, ,22,ACCEPT
16: TCP,INPUT,*.*.*.*,ANY, ,80,ACCEPT
17: UDP,INPUT, *.*,53, ,ANY,ACCEPT
18: UDP,OUTPUT, ,ANY, *.*,53,ACCEPT
Rule 1, Rule 2: ==> GENRERALIZATION
Rule 1, Rule 16: ==> CORRELATED
Rule 2, Rule 12: ==> SHADOWED
Rule 4, Rule 5: ==> GENRERALIZATION
Rule 4, Rule 15: ==> CORRELATED
Rule 5, Rule 11: ==> SHADOWED
Anomaly Discovery Result

47. Worm Detection: Introduction
What are worms?
Self-replicating program; Exploits software vulnerability on a victim; Remotely infects other victims
Evil worms
Severe effect; Code Red epidemic cost $2.6 Billion
Goals of worm detection
Real-time detection
Issues
Substantial Volume of Identical Traffic, Random Probing
Methods for worm detection
Count number of sources/destinations; Count number of failed connection attempts
Worm Types
worms, Instant Messaging worms, Internet worms, IRC worms, File- sharing Networks worms
Automatic signature generation possible
EarlyBird System (S. Singh -UCSD); Autograph (H. Ah-Kim - CMU)

48. Email Worm Detection using Data Mining
Task:
given some training instances of both “normal” and “viral” s,
induce a hypothesis to detect “viral” s.
We used:
Naïve Bayes
SVM
Outgoing s
The Model
Test data
Feature extraction
Classifier
Machine Learning
Training data
Clean or Infected ?

49. Assumptions Features are based on outgoing emails.
Different users have different “normal” behaviour.
Analysis should be per-user basis.
Two groups of features
Per (#of attachments, HTML in body, text/binary attachments)
Per window (mean words in body, variable words in subject)
Total of 24 features identified
Goal: Identify “normal” and “viral” s based on these features

50. Feature sets Per email features Binary valued Features
Presence of HTML; script tags/attributes; embedded images; hyperlinks;
Presence of binary, text attachments; MIME types of file attachments
Continuous-valued Features
Number of attachments; Number of words/characters in the subject and body
Per window features
Number of s sent; Number of unique recipients; Number of unique sender addresses; Average number of words/characters per subject, body; average word length:; Variance in number of words/characters per subject, body; Variance in word length
Ratio of s with attachments

51. Data Mining Approach Classifier Clean/ Infected Test instance
SVM
infected?
Naïve Bayes
Test instance
Clean?
Clean

52. Data set Collected from UC Berkeley.
Contains instances for both normal and viral s.
Six worm types:
bagle.f, bubbleboy, mydoom.m,
mydoom.u, netsky.d, sobig.f
Originally Six sets of data:
training instances: normal (400) + five worms (5x200)
testing instances: normal (1200) + the sixth worm (200)
Problem: Not balanced, no cross validation reported
Solution: re-arrange the data and apply cross-validation

53. Our Implementation and Analysis
Naïve Bayes: Assume “Normal” distribution of numeric and real data; smoothing applied
SVM: with the parameter settings: one-class SVM with the radial basis function using “gamma” = and “nu” = 0.1.
Analysis
NB alone performs better than other techniques
SVM alone also performs better if parameters are set correctly
mydoom.m and VBS.Bubbleboy data set are not sufficient (very low detection accuracy in all classifiers)
The feature-based approach seems to be useful only when we have
identified the relevant features
gathered enough training data
Implement classifiers with best parameter settings

54. Digital Forensics and UTD Work
Machines are infected through unauthorized intrusions, worms and viruses
Therefore data has to be acquired from the machine, we skip this step as we get the data from open source web sites
We then apply our analysis tools based on data mining
Our current research at UTD is focusing mainly on “Botnets” and also to some extent “Honeypots”.
We are also conducting research on “Active Defense” – trying to find out the adversary is upto.

55. Algorithms for Digital Forensics
GUIDE&dl=GUIDE&idx=J79&part=periodical&WantType=per iodical&title=Communications%20of%20the%20ACM