Executive Director and Endowed Chair CS 5323 Malware Detection Prof. Ravi Sandhu Executive Director and Endowed Chair Lecture 12 ravi.utsa@gmail.com www.profsandhu.com © Ravi Sandhu World-Leading Research with Real-World Impact!

Highlights Virus detection is undecidable Cohen dissertation (1985), paper (1987) Anti-virus (more generally anti-malware) is a great business model Need regular updates Infinite supply of new malware Malware can be stealthy Malware can be really stealthy © Ravi Sandhu World-Leading Research with Real-World Impact! 2

Malware Detection Techniques Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007. © Ravi Sandhu World-Leading Research with Real-World Impact! 3

Malware Detection Techniques Misuse Detection Behavior-Based Detection Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007. © Ravi Sandhu World-Leading Research with Real-World Impact! 4

Signature Limitations S needs regular updates Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007. © Ravi Sandhu World-Leading Research with Real-World Impact! 5

Anomaly Based Training Phase Detection Phase Infer patterns Infer specifications © Ravi Sandhu World-Leading Research with Real-World Impact! 6

Anomaly Based Limitations Blue area is false positives If white area extends outside blue area we have false negatives Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007. © Ravi Sandhu World-Leading Research with Real-World Impact! 7

Stealthy Malware Defeat signature-based detection Encrypted malware Polymorphic malware Metamorphic malware Rootkit can misrepresent the existence or content of executable files You, I., and Yim, K. Malware obfuscation techniques: A brief survey. IEEE International Conference on Broadband, Wireless Computing, Communication and Applications, Nov 2010, pp. 297-300. © Ravi Sandhu World-Leading Research with Real-World Impact! 8

Encrypted Malware execute malware Encrypted Main Body Key Decryptor Cleartext Main Body propagate malware Encrypted Main Body Key’ Decryptor © Ravi Sandhu World-Leading Research with Real-World Impact! 9

Encrypted Malware execute malware Encrypted Main Body Key Decryptor Cleartext Main Body propagate malware reveals signature Encrypted Main Body Key’ Decryptor © Ravi Sandhu World-Leading Research with Real-World Impact! 10

Polymorphic Malware execute malware Encrypted Main Body Key Decryptor Cleartext Main Body propagate malware Encrypted Main Body Key’ Obfuscated Decryptor © Ravi Sandhu World-Leading Research with Real-World Impact! 11

Polymorphic Malware execute malware Encrypted Main Body Key Decryptor Cleartext Main Body propagate malware no signature Encrypted Main Body Key’ Obfuscated Decryptor © Ravi Sandhu World-Leading Research with Real-World Impact! 12

Execute in a sandbox and detect the signature after decryption Polymorphic Malware execute malware Encrypted Main Body Key Decryptor Cleartext Main Body propagate malware no signature Encrypted Main Body Key’ Obfuscated Decryptor Execute in a sandbox and detect the signature after decryption © Ravi Sandhu World-Leading Research with Real-World Impact! 13

Polymorphic Malware execute malware Encrypted Main Body Key Decryptor Cleartext Main Body propagate malware no signature Encrypted Main Body Key’ Obfuscated Decryptor Execute in a sandbox and detect the signature after decryption Mutation Engines automate this construction © Ravi Sandhu World-Leading Research with Real-World Impact! 14

Metamorphic Malware execute malware Original Main Body propagate malware execute malware Obfuscated Main Body Obfuscated Main Body propagate malware execute malware Obfuscated Main Body Obfuscated Main Body no signature © Ravi Sandhu World-Leading Research with Real-World Impact! 15

Obfuscation Techniques Dead-Code Insertion Register Reassignment Subroutine Reordering Instruction substitution Code transposition Code Integration © Ravi Sandhu World-Leading Research with Real-World Impact! 16

Really Stealthy Malware Not visible in source code Reappears in binary code due to malware infected compiler In theory could reappear in binary code due to other components in binary execution workflow Loader Linker OS BIOS Ken Thompson. Reflections on trusting trust. Commun. ACM 27, 8 (August 1984), 761-763. © Ravi Sandhu World-Leading Research with Real-World Impact! 17

Malicious Compiler Inserts a Backdoor OS Login module Malicious Compiler Binary Infected Login Binary © Ravi Sandhu World-Leading Research with Real-World Impact! 18

Malicious Compiler Inserts a Backdoor OS Login module Assumption: Malicious behavior cannot be detected in binary, but may be detectable in compiler source Malicious Compiler Binary Infected Login Binary © Ravi Sandhu World-Leading Research with Real-World Impact! 19

Self-Compiler Compiler source for language L Compiler binary for language L Compiler binary for language L © Ravi Sandhu World-Leading Research with Real-World Impact! 20

Malicious Self-Compiler in Binary and Source Malicious Compiler source for language L Compiler binary for language L Malicious Compiler binary for language L © Ravi Sandhu World-Leading Research with Real-World Impact! 21

Malicious Self-Compiler in Binary and Source Source code analysis will reveal malicious behavior Malicious Compiler source for language L Compiler binary for language L Malicious Compiler binary for language L © Ravi Sandhu World-Leading Research with Real-World Impact! 22

Doubly Malicious Self-Compiler in Binary and Source Source code analysis will reveal doubly malicious behavior Doubly Malicious Compiler source for language L Compiler binary for language L Doubly Malicious Compiler binary for language L © Ravi Sandhu World-Leading Research with Real-World Impact! 23

Doubly Malicious Complier Binary Behavior Compiler source for language L OS Login module Doubly Malicious Compiler binary for language L Doubly Malicious Compiler binary for language L Infected Login Binary Doubly Malicious Compiler binary for language L © Ravi Sandhu World-Leading Research with Real-World Impact! 24

Doubly Malicious Complier Binary Behavior Compiler source for language L OS Login module Doubly Malicious Compiler binary for language L Doubly Malicious Compiler binary for language L Infected Login Binary Doubly Malicious Compiler binary for language L No trace of malicious behavior in source code © Ravi Sandhu World-Leading Research with Real-World Impact! 25

Malicious Self-Compiler in Binary but not in Source Compiler source for language L Malicious Compiler binary for language L Malicious Compiler binary for language L partial countermeasure Wheeler, D.A., Countering trusting trust through diverse double-compiling, 21st Annual Computer Security Applications Conference, pp.13-48, 5-9 Dec. 2005. © Ravi Sandhu World-Leading Research with Real-World Impact! 26