Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani TU Vienna Christopher UCSB Engin Institute Eurecom Xiaoyong Zhou, XiaoFeng Indiana Univ. at Bloominton 1 USENIX Security Symposium ‘09

Outline Motivation System Overview System Details Evaluation Limitation Conclution 2

MOTIVATION Effectiveness & Efficiency 3

Motivation Efficiency – Binary signature based detection – Network-based detection Effectiveness – Behavior-based detection Detection based on malware's behavior Behavior is hard to obfuscate Behavior is hard to randomize Behavior is often stable across various malware version 4

Motivation This Paper proposes… – A behavior-based solution with Efficiency – For end hosts 5

SYSTEM OVERVIEW Modeling Behaviors and Making detection efficient 6

System Overview Malware behaviors – Manifest on system (i.e., survive reboot) (Over-) write system executables, dlls, files Create registry entries Register as Windows (startup) service – Conceal from being detected Restart under some stealthy name (e.g., svchost.exe) Inject into legitimate processes – Replicate Send s Copy to Samba shares, USB drives, etc. Scan and exploit services on LAN or WAN 7

System Overview Detection based on execution characteristics – Execute malware in full system emulator (Anubis) – Monitor interaction with the operating system – Perform detailed taint analysis – Generate detection graphs Describe sequence of required system calls leading to security relevant system activity Include dependencies to related, previous calls (using taint dependencies) Detect described behavior on end host – Log system call activity of unknown executable – Match against behavior graph 8

System Overview Example: Agent (trojan) As part of its system manifestation, it – Reads content from binary image – Decrypts binary content Proprietary decryption routine Simple, XOR based algorithm – Stores binary in system file (C:\Windows\system32\drivers\ip6fw.sys) – Later, restarts IPv6 firewall Turns itself into a system service 9

System Overview 10

SYSTEM DETAILS Generate Behavior Graphs, Match Behavior Graphs 11

System Details Behavior graphs – Directed acyclic graph – Node: system calls – Edges: dependencies Dependencies – Handle dependencies Direct value propagation System provided identifiers Must be constant 12

System Details Data dependencies – Arbitrary data (& control) dependency between system calls – Might modify values between system calls 13

System Details Generate behavior graphs – Analyze executable in Anubis sandbox Obtain instruction level log Obtain program flow log Obtain memory access log Generate precise taint propagation trees – Data/control dependencies – Instructions that access/generate tainted data – Link system calls consuming data with all taint generating calls (sources) 14

System Details Generate behavior graphs (cont.) – Scan logs for security relevant behavior Provided with a list of interesting system calls Extract propagation formulas 15

System Details Match behavior graphs – Active(inactive) node – Simple(complex) function – Security-relevant system calls or the Buttom – Confirmed(deactivate all) 16

System Details 17

System Details 18

EVALUATION Effectiveness, Efficiency 19

Evaluation Effectiveness 20

Evaluation 21

Evaluation False Positive – IE, Firefox, Thunderbird, putty, notepad – 0 22

Evalution Efficiency 23

LIMITATION & CONCLUSION 24

Limitation Evading signature generation – Detect the virtual environment – Delays, time-triggered behavior Modifying the algorithm behavior 25

Conclusion Behavior can be detected Behavior detection is fast enough for end hosts – Approach intrinsically robust against polymorphism and metamorphism – To some extent, behavior graphs are usable across malware variants 26