Using Engine Signature to Detect Metamorphic Malware Mohamed R. Chouchane and Arun Lakhotia Software Research Laboratory The University of Louisiana at Lafayette Fourth Workshop on Rapid Malcode (WORM) November 3rd, 2006 George Mason University, Fairfax, VA, USA

SCAM'062 9/28/2006 Metamorphic Malware Virus Form - C M M Virus Form - A Form - B Metamorphic malware changes as it propagates It creates multiple variants of itself

SCAM'063 9/28/2006 Metamorphic Malware Challenge Signature Virus Form - C M M Virus Form - A Form - B Too many signatures challenge the AV Scanner Using different signatures for most variants cannot scale. Antivirus scanners using extracted byte sequences, or “signatures” to identify known malware.

WORM'064 11/03/2006 Engine Signature: Track Variants to their Engine One Engine  Source of Variation Engine-friendly code is “Code written for the engine” Idea: Engine Signature vs. Virus Signature  Lightens burden of one signature per variant  Analogous to determining likelihood of engine authorship E-friendly malware release feedback Engine variant

WORM'065 11/03/2006 Engine-Friendliness 10% friendly20% friendly90% friendly100% friendly Low E-friendliness Input Variants Output Variants Metamorphic Engine Instruction Substitution Garbage Insertion

WORM'066 11/03/2006 Code Substitution: Evol mov [esi+4], 9  mov esi+4], 6 add esi+4], 3 mov [ebp+8], ecx  push eax mov eax, ecx mov [ebp+8], eax pop eax push 4  mov eax, 4 push eax push eax  push eax mov eax, 2Bh Clues

WORM'067 11/03/2006 Scoring Function S E (V)=  c  s w c e cs / |V| S E (V) measures how dense a code segment V is with clues from some code-substituting engine E. Clues are weighted according to their length. Can explore other weight assignments Code Segment Clue Count per Site push 7 mov 2 sub 0 mov 0 pop 0 mov 2 add 0 mov 2 add 0 push 8 mov 2 add 0 mov 2 add 0 pop 0 S E = 25/15 =1.667

WORM'068 11/03/2006 Evaluation: Non-Evol Segments Frequency distributions of the scores of 2nd to 7th generation with initial E-friendliness 5%(figure at left) and 50% (figure at right) The E-friendlier the Eve, the higher the score Later variants tend to score higher “Convergence” behavior

WORM'069 11/03/2006 Evaluation: Simulated Evol Segments Frequency distributions of the scores of 2nd to 4th generations (left to right) of simulated Evol variants Certain range of values Gaussian Like 2 nd, 3 rd, and 4 th gen variants scored 1.62, 1.95, and 2.13, respectively

WORM' /03/2006 Discussion Limitations  Small clues  Less transformation options  Low friendliness  Malware open to traditional signature scanning  More analysis may be needed Improvement and Further work  Investigate other weight assignments  Investigate engines which expand and shrink code  Functional relationship among parameters  Use engine signature to determine toolkit authorship

WORM' /03/2006 Software Research Lab Center for Advanced Computer Studies University of Louisiana at Lafayette Arun Lakhotia Director Andrew Walenstein Research Scientist Michael Venable Software Engineer and Alumnus Ph.D. Students Mohamed R. Chouchane Md Enamul Karim M.S. Students Matthew Hayes Christopher Thompson Alumni Nitin Jyoti, Avertlabs Aditya Kapoor, McAfee Erik Uday Kumar, Authentium Rachit Mathur, McAfee Moinuddin Mohammed, Microsoft Prashant Pathak, Symantec Prabhat Singh, Symantec Funded by: Louisiana Governor’s IT Initiative

WORM' /03/2006 more at “Using Engine Signature to Detect Metamorphic Malware” Mohamed R. Chouchane and Arun Lakhotia Software Research Laboratory The University of Louisiana at Lafayette