1. Detecting Malicious Executables
Mr. Mehedy Masud (PhD Student)
Prof. Latifur Khan
Prof. Bhavani Thuraisingham
Department of Computer Science
The University of Texas at Dallas
Lecture#7

2. Introduction: Detecting Malicious Executables
What are malicious executables?
Harm computer systems
Virus, Exploit, Denial of Service (DoS), Flooder, Sniffer, Spoofer, Trojan etc.
Exploits software vulnerability on a victim
May remotely infect other victims
Incurs great loss. Example: Code Red epidemic cost $2.6 Billion
Malicious code detection: Traditional approach
Signature based
Requires signatures to be generated by human experts
So, not effective against “zero day” attacks

3. State of the Art: Automated Detection
Automated detection approaches:
Behavioural: analyse behaviours like source, destination address, attachment type, statistical anomaly etc.
Content-based: analyse the content of the malicious executable
Autograph (H. Ah-Kim – CMU): Based on automated signature generation process
N-gram analysis (Maloof, M.A. et .al.): Based on mining features and using machine learning.

4. New Ideas
Content -based approaches consider only machine-codes (byte-codes).
Is it possible to consider higher-level source codes for malicious code detection?
Yes: Diassemble the binary executable and retrieve the assembly program
Extract important features from the assembly program
Combine with machine-code features

5. Feature Extraction Binary n-gram features Assembly n-gram features
Sequence of n consecutive bytes of binary executable
Assembly n-gram features
Sequence of n consecutive assembly instructions
System API call features
DLL function call information

6. The Hybrid Feature Retrieval Model
Collect training samples of normal and malicious executables.
Extract features
Train a Classifier and build a model
Test the model against test samples

7. Hybrid Feature Retrieval (HFR)
Training

8. Hybrid Feature Retrieval (HFR)
Testing

9. Feature Extraction Binary n-gram features Example: Problem: Solution:
Features are extracted from the byte codes in the form of n-grams, where n = 2,4,6,8,10 and so on.
Example:
Given a 11-byte sequence: abcdef012345,
The 2-grams (2-byte sequences) are: 0123, 2345, 4567, 6789, 89ab, abcd, cdef, ef01, 0123, 2345
The 4-grams (4-byte sequences) are: , , ab,...,ef and so on....
Problem:
Large dataset. Too many features (millions!).
Solution:
Use secondary memory, efficient data structures
Apply feature selection

10. Feature Extraction Assembly n-gram features Example: Problem:
Features are extracted from the assembly programs in the form of n-grams, where n = 2,4,6,8,10 and so on.
Example:
three instructions
“push eax”; “mov eax, dword[0f34]” ; “add ecx, eax”;
2-grams
(1) “push eax”; “mov eax, dword[0f34]”;
(2) “mov eax, dword[0f34]”; “add ecx, eax”;
Problem:
Same problem as binary
Solution:
Same solution

11. Feature Selection Select Best K features
Selection Criteria: Information Gain
Gain of an attribute A on a collection of examples S is given by

12. Experiments Dataset Disassembly Training, Testing
Dataset1: 838 Malicious and 597 Benign executables
Dataset2: 1082 Malicious and 1370 Benign executables
Collected Malicious code from VX Heavens (
Disassembly
Pedisassem ( )
Training, Testing
Support Vector Machine (SVM)
C-Support Vector Classifiers with an RBF kernel

13. Results HFS = Hybrid Feature Set BFS = Binary Feature Set
AFS = Assembly Feature Set

14. Results HFS = Hybrid Feature Set BFS = Binary Feature Set
AFS = Assembly Feature Set

15. Results HFS = Hybrid Feature Set BFS = Binary Feature Set
AFS = Assembly Feature Set

16. Future Plans System call: seems to be very useful.
Need to Consider Frequency of call
Call sequence pattern (following program path)
Actions immediately preceding or after call
Detect Malicious code by program slicing
requires analysis

17. Buffer Overflow Attack Detection
Mohammad M. Masud,
Latifur Khan,
Bhavani Thuraisingham
Department of Computer Science
The University of Texas at Dallas

18. Introduction Goal Main Contribution Intrusion detection.
e.g.: worm attack, buffer overflow attack.
Main Contribution
'Worm' code detection by data mining coupled with 'reverse engineering'.
Buffer overflow detection by combining data mining with static analysis of assembly code.

19. Background What is 'buffer overflow'? How does it happen?
A situation when a fixed sized buffer is overflown by a larger sized input.
How does it happen?
example:
char buff[100];
gets(buff);
memory
buff
Stack
Input string

20. Background (cont...) Then what? buff Stack ........ char buff[100];
gets(buff);
memory
buff
Stack
Return address overwritten
Attacker's code
memory
buff
Stack
New return address points to this memory location

21. Background (cont...) So what? It can now How to stop it?
Program may crash or
The attacker can execute his arbitrary code
It can now
Execute any system function
Communicate with some host and download some 'worm' code and install it!
Open a backdoor to take full control of the victim
How to stop it?

22. Background (cont...) Stopping buffer overflow Preventive approaches
Detection approaches
Finding bugs in source code. Problem: can only work when source code is available.
Compiler extension. Same problem.
OS/HW modification
Capture code running symptoms. Problem: may require long running time.
Automatically generating signatures of buffer overflow attacks.

23. CodeBlocker (Our approach)
A detection approach
Based on the Observation:
Attack messages usually contain code while normal messages contain data.
Main Idea
Check whether message contains code
Problem to solve:
Distinguishing code from data

24. Severity of the problem
It is not easy to detect actual instruction sequence from a given string of bits

25. Our solution Apply data mining.
Formulate the problem as a classification problem (code, data)
Collect a set of training examples, containing both instances
Train the data with a machine learning algorithm, get the model
Test this model against a new message

26. CodeBlocker Model

27. Feature Extraction

28. Disassembly We apply SigFree tool
implemented by Xinran Wang et al. (PennState)

29. Feature extraction Features are extracted using N-gram analysis
Control flow analysis
What is an n-gram?
-Sequence of n instructions
Traditional approach:
-Flow of control is ignored
2-grams are: 02, 24, 46,...,CE
Assembly program
Corresponding IFG

30. Feature extraction (cont...)
Control-flow Based N-gram analysis
What is an n-gram?
-Sequence of n instructions
Proposed Control-flow based approach
-Flow of control is considered
2-grams are:
02, 24, 46,...,CE, E6
Assembly program
Corresponding IFG

31. Feature extraction (cont...)
Control Flow analysis. Generated features
Invalid Memory Reference (IMR)
Undefined Register (UR)
Invalid Jump Target (IJT)
Checking IMR
A memory is referenced using register addressing and the register value is undefined
e.g.: mov ax, [dx + 5]
Checking UR
Check if the register value is set properly
Checking IJT
Check whether jump target does not violate instruction boundary

32. Feature extraction (cont...)
Why n-gram analysis?
Intuition: in general, disassembled executables should have a different pattern of instruction usage than disassembled data.
Why control flow analysis?
Intuition: there should be no invalid memory references or invalid jump targets.

33. Putting it together Compute all possible n-grams Select best k of them
Compute feature vector (binary vector) for each training example
Supply these vectors to the training algorithm

34. Experiments Dataset Training, Testing Real traces of normal messages
Real attack messages
Polymorphic shellcodes
Training, Testing
Support Vector Machine (SVM)

35. Results CFBn: Control-Flow Based n-gram feature
CFF: Control-flow feature

36. Novelty / contribution
We introduce the notion of control flow based n-gram
We combine control flow analysis with data mining to detect code / data
Significant improvement over other methods (e.g. SigFree)

37. Advantages 1) Fast testing 2) Signature free operation 3) Low overhead
4) Robust against many obfuscations

38. Limitations Need samples of attack and normal messages.
May not be able to detect a completely new type of attack.

39. Future Works Find more features Apply dynamic analysis techniques
Semantic analysis

40. Reference / suggested readings
X. Wang, C. Pan, P. Liu, and S. Zhu. Sigfree: A signature free buffer overflow attack blocker. In USENIX Security, July 2006.
Kolter, J. Z., and Maloof, M. A. Learning to detect malicious executables in the wild Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining Seattle, WA, USA Pages: 470 – 478, 2004.

41. Email Worm Detection (behavioural approach)
Outgoing s
The Model
Feature extraction
Test data
Training data
Machine Learning
Classifier
Clean or Infected ?

42. Feature Extraction Per email features Per window features
Binary valued Features
Presence of HTML; script tags/attributes; embedded images; hyperlinks;
Presence of binary, text attachments; MIME types of file attachments
Continuous-valued Features
Number of attachments; Number of words/characters in the subject and body
Per window features
Number of s sent; Number of unique recipients; Number of unique sender addresses; Average number of words/characters per subject, body; average word length:; Variance in number of words/characters per subject, body; Variance in word length
Ratio of s with attachments

43. Feature Reduction & Selection
Principal Component Analysis
Reduce higher dimensional data into lower dimension
Helps reducing noise, overfitting
Decesion Tree
Used to Select Best features

44. Experiments Data Set Training, Testing:
Contains instances for both normal and viral s.
Six worm types:
bagle.f, bubbleboy, mydoom.m, mydoom.u, netsky.d, sobig.f
Collected from UC Berkeley
Training, Testing:
Decision Tree: C4.5 algorithm (J48) on Weka Systems
Support Vector Machine (SVM) and Naïve Bayes (NB).

45. Results

46. Conclusion & Future Work
Three approaches has been tested
Apply classifier directly
Apply dimension reduction (PCA) and then classify
Apply feature selection (decision tree) and then classify
Decision tree has the best performance
Future Plans
Combine content based with behavioral approaches
Offensive Operations
Honeypots, Information operations