1. Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo Milani Comparetti @ TU Vienna Christopher Kruegel @ UCSB Engin Kirda @ Institute Eurecom Xiaoyong Zhou, XiaoFeng Wang @ Indiana Univ. at Bloominton 1 USENIX Security Symposium ‘09

2. Outline Motivation System Overview System Details Evaluation Limitation Conclution 2

3. MOTIVATION Effectiveness & Efficiency 3

4. Motivation Efficiency – Binary signature based detection – Network-based detection Effectiveness – Behavior-based detection Detection based on malware's behavior Behavior is hard to obfuscate Behavior is hard to randomize Behavior is often stable across various malware version 4

5. Motivation This Paper proposes… – A behavior-based solution with Efficiency – For end hosts 5

6. SYSTEM OVERVIEW Modeling Behaviors and Making detection efficient 6

7. System Overview Malware behaviors – Manifest on system (i.e., survive reboot) (Over-) write system executables, dlls, files Create registry entries Register as Windows (startup) service – Conceal from being detected Restart under some stealthy name (e.g., svchost.exe) Inject into legitimate processes – Replicate Send emails Copy to Samba shares, USB drives, etc. Scan and exploit services on LAN or WAN 7

8. System Overview Detection based on execution characteristics – Execute malware in full system emulator (Anubis) – Monitor interaction with the operating system – Perform detailed taint analysis – Generate detection graphs Describe sequence of required system calls leading to security relevant system activity Include dependencies to related, previous calls (using taint dependencies) Detect described behavior on end host – Log system call activity of unknown executable – Match against behavior graph 8

9. System Overview Example: Agent (trojan) As part of its system manifestation, it – Reads content from binary image – Decrypts binary content Proprietary decryption routine Simple, XOR based algorithm – Stores binary in system file (C:\Windows\system32\drivers\ip6fw.sys) – Later, restarts IPv6 firewall Turns itself into a system service 9

10. System Overview 10

11. SYSTEM DETAILS Generate Behavior Graphs, Match Behavior Graphs 11

12. System Details Behavior graphs – Directed acyclic graph – Node: system calls – Edges: dependencies Dependencies – Handle dependencies Direct value propagation System provided identifiers Must be constant 12

13. System Details Data dependencies – Arbitrary data (& control) dependency between system calls – Might modify values between system calls 13

14. System Details Generate behavior graphs – Analyze executable in Anubis sandbox Obtain instruction level log Obtain program flow log Obtain memory access log Generate precise taint propagation trees – Data/control dependencies – Instructions that access/generate tainted data – Link system calls consuming data with all taint generating calls (sources) 14

15. System Details Generate behavior graphs (cont.) – Scan logs for security relevant behavior Provided with a list of interesting system calls Extract propagation formulas 15

16. System Details Match behavior graphs – Active(inactive) node – Simple(complex) function – Security-relevant system calls or the Buttom – Confirmed(deactivate all) 16

17. System Details 17

18. System Details 18

19. EVALUATION Effectiveness, Efficiency 19

20. Evaluation Effectiveness 20

21. Evaluation 21

22. Evaluation False Positive – IE, Firefox, Thunderbird, putty, notepad – 0 22

23. Evalution Efficiency 23

24. LIMITATION & CONCLUSION 24

25. Limitation Evading signature generation – Detect the virtual environment – Delays, time-triggered behavior Modifying the algorithm behavior 25

26. Conclusion Behavior can be detected Behavior detection is fast enough for end hosts – Approach intrinsically robust against polymorphism and metamorphism – To some extent, behavior graphs are usable across malware variants 26