Normalizing Metamorphic Malware Using Term Rewriting A. Walenstein, R. Mathur, M. R. Chouchane, and A. Lakhotia Software Research Laboratory The University of Louisiana at Lafayette Sixth IEEE International Workshop on Source Code Analysis and Manipulation 27th-29th September 2006 Philadelphia, PA, USA

SCAM'062 9/28/2006 About this Work The core of the paper's work formed the Master's thesis of Rachit Mathur. He has since graduated and is now working at McAfee.

SCAM'063 9/28/2006 Malware Identification Anti-Virus Signature Virus Form - A Malware are malicious programs such as viruses, worms, and Trojans. Antivirus scanners use extracted patterns, or “signatures” to identify known malware. Signature

SCAM'064 9/28/2006 Metamorphic Malware Virus Form - C M M Virus Form - A Form - B Metamorphic malware change as they propagate They create multiple variants of themselves

SCAM'065 9/28/2006 Metamorphic Malware Challenge Anti-Virus Signature Virus Form - C M M Virus Form - A Form - B Too many signatures challenge the AV Scanner Using different signatures for most variants cannot scale.

SCAM'066 9/28/2006 Proposed approach: normalizer Anti-Virus Signature NormalForm N N N Virus Form - A Virus Form - B Virus Form - C M M Normalizer Construction Problem: Reduce the number of signatures needed to detect all variants. Virus

SCAM'067 9/28/2006 mov [ebp - 3], eax push ecx mov ecx,ebp add ecx,33 push esi mov esi,ecx sub esi,34 mov [esi-2],eax pop esi pop ecx push ecx mov ecx, ebp push eax mov eax, 33 add ecx, eax pop eax push esi mov esi, ecx push edx mov edx, 34 sub esi, edx pop edx mov [esi - 2], eax pop esi pop ecx push ecx mov ecx, [ebp + 10] mov ecx, ebp push eax add eax, 2342 mov eax, 33 add ecx, eax pop eax mov eax, esi push eax mov esi, ecx push edx xor edx, 778f mov edx, 34 sub esi, edx pop edx mov [esi-2], eax pop esi pop ecx push ecx mov ecx,ebp add ecx,33 mov [ecx-36],eax pop ecx Inspiration: “undo” transformations

SCAM'068 9/28/2006 Problem 1: “naïve” undo is naïve mov eax, 0x04 push eax mov eax, 0x04 push eax push 0x04 mov eax, 0x04 push 0x04 mov edi, 0x04 3. mov eax, 0x04 push eax 2. push eax mov eax, 0x04 push 0x04 1. push ecx mov ecx, 0x04 mov edi, ecx pop ecx push eax

SCAM'069 9/28/2006 Problem 2: conditional transformations mov edi, 0x04 mov eax, 0x04 push eax mov eax, 0x04 push 0x04 push ecx mov ecx, 0x04 mov edi, ecx pop ecx push eax eax not live unconditional Q: how to reorient rules while guaranteeing termination? eax not live

SCAM'0610 9/28/2006 Term rewriting approach Adopted term-rewriting framework  Model the metamorphic engine as TRS  Modify it to create normalizing rule set and engine apply completion procedure, which reorients rules  Can guarantee needed properties (termination, confluence)

SCAM'0611 9/28/2006 mov eax, 0x04 push eax mov eax, 0x04 Completion procedure sketch mov eax, 0x04 push eax push 0x04 mov eax, 0x04 push 0x04 Critical Pairs

SCAM'0612 9/28/2006 mov eax, 0x04 push eax mov eax, 0x04 Completion procedure sketch mov eax, 0x04 push eax push 0x04 mov eax, 0x04 push 0x04 Reorient New Rule

SCAM'0613 9/28/2006 What to do when completion procedure fails? Successful completion guarantees a unique normal form for all variants:  The “perfect” normalizer but  Completion procedure may not terminate!  Number of rules in the normalizer may be too high to be practical  Does not take into account conditions  Need alternative scheme

SCAM'0614 9/28/2006 Priority Scheme Partition N into N U and N C Still Reducible? Input Program Normalize w.r.t N U HALT no Y N U – Unconditional rules N C – Conditional rules If possible, Apply a rule from N C yes 1.Simple 2.No Need for costly/imprecise condition evaluation 3.Improved through Ad-hoc completion

SCAM'0615 9/28/2006 Question: condition checking required? Conditional rules require checking of conditions  Can be expensive, or impossible  What is the practical penalty of incorrectly checking conditions? e.g., ignoring conditions completely?

SCAM'0616 9/28/2006 Case Study W32.Evol Virus can generate huge number of variants Tested the normalization schemes on 26 variants over 6 generations Manually Extracted rules used by W32.Evol  55 rules  84 overlaps TXL implementations:  Ordinary and priority-based evaluation

Results NormalizerGenerationEve23456 Avg. size of original Convergen t Avg. size of normal form 2173 Priority AC Avg. size of normal form 2166 Priority WC Avg. size of normal form Lines not in common % in common

SCAM'0618 9/28/2006 Contributions Applications for assisting malware scanners  Initial exploration of possibility of “perfect” normalization  Indications of usefulness of heuristic alternatives (priority scheme and ignoring conditions)

SCAM'0619 9/28/2006 Future Work Expanded scope and empirical study  Extensions for semantics-non-preserving metamorphic engines?  Localized normalization using term rewriting M. Chouchane and A. Lakhotia “Using Engine Signature to Detect Metamorphic Malware”, Workshop on Rapid Malcode, Fairfax, VA, Nov (to appear) More at

SCAM'0620 9/28/2006 Software Research Lab Center for Advanced Computer Studies University of Louisiana at Lafayette Arun Lakhotia Director Andrew Walenstein Research Scientist Michael Venable Software Engineer and Alumnus Ph.D. Students Mohamed R. Chouchane Md Enamul Karim M.S. Students Christopher Thompson Matthew Hayes Alumni Nitin Jyoti, Avertlabs Aditya Kapoor, McAfee Erik Uday Kumar, Authentium Rachit Mathur, McAfee Moinuddin Mohammed, Microsoft Prashant Pathak, Symantec Prabhat Singh, Symantec Funded by: Louisiana Governor’s IT Initiative