Copyright © 2011, A Behavior-based Methodology for Malware Detection Student: Hsun-Yi Tsai Advisor: Dr. Kuo-Chen Wang 2012/04/30

Copyright © 2011, Outline Introduction Problem Statement Sandboxes Behavior Rules Prototype Malicious Degree Evaluation Conclusion and Future Works References

Copyright © 2011, Introduction Signature-based detection may fail sometimes –Malware developers may make some changes to evade detection Malware and their variations still share the same behaviors in high level –Malicious behaviors are similar most of the time Behavior-based detection –To detect unknown malware or the variations of known malware by analyzing their behaviors

Copyright © 2011, Problem Statement Given –Several sandboxes –l known malware M i = {M 1,M 2, …, M l } for training –m known malware S j = {S 1, S 2, …, S m } for testing Objective –n behaviors B k = {B 1,B 2, …, B n } –n weights W k = {W 1,W 2, …, W n } –MD (Malicious degree)

Copyright © 2011, Sandboxes Online (Web-based) –GFI Sandbox –Norman Sandbox –Anubis Sandbox Offline (PC-based) –Avast Sandbox –Buster Sandbox Analyzer

Copyright © 2011, Behavior Rules Malware Host Behaviors –Creates Mutex –Creates Hidden File –Starts EXE in System –Checks for Debugger –Starts EXE in Documents –Windows/Run Registry Key Set –Hooks Keyboard –Modifies Files in System –Deletes Original Sample –More than 5 Processes –Opens Physical Memory –Deletes Files in System –Auto Start Malware Network Behaviors –Makes Network Connections DNS Query HTTP Connection File Download

Copyright © 2011, Behavior Rules (Cont.) Ulrich Bayer et al. [13]

Copyright © 2011, Prototype

Copyright © 2011, Malicious Degree

Copyright © 2011, Weight Training Module - ANN Using Artificial Neural Network (ANN) to train weights

Copyright © 2011, Weight Training Module - ANN (Cont.) Neuron for ANN hidden layer

Copyright © 2011, Weight Training Module - ANN (Cont.) Neuron for ANN output layer

Copyright © 2011, Weight Training Module - ANN (Cont.) Delta learning process d: expected target value Mean square error: Weight set :, : learning factor; x: input value

Copyright © 2011, Evaluation – Initial Weights BehaviorWeight Creates Mutex Creates Hidden File Starts EXE in System Checks for Debugger Starts EXE in Documents Windows/Run Registry Key Set Hooks Keyboard Modifies File in System Deletes Original Sample More than 5 Processes Opens Physical Memory Delete File in System Autorun0.24

Copyright © 2011, Evaluation (Cont.) Try to find the optimal MD value makes PF and PN approximate to 0.

Copyright © 2011, Evaluation (Cont.) Training data and testing data Threshold of MD value. MaliciousBenignTotal Training Testing353166

Copyright © 2011, Evaluation (Cont.) With Creates Hidden File, Windows/Run Registry Key Set, More than 5 Processes, and Delete File in System: WeightsAccuracy Frequency98% 192.4% 0.591%

Copyright © 2011, Evaluation (Cont.) Without Creates Hidden File, Windows/Run Registry Key Set, More than 5 Processes, and Delete File in System: WeightsAccuracy Frequency97% 194% %

Copyright © 2011, Conclusion and Future Work Conclusion –Collect several common behaviors of malwares –Construct Malicious Degree (MD) formula Future work –Add more malware network behaviors –Classify malwares according to their typical behaviors –Detect unknown malwares

Copyright © 2011, References [1] GFI Sandbox. [2] Norman Sandbox. [3] Anubis Sandbox. [4] Avast Sandbox. [5] Buster Sandbox Analyxer (BSA). [6] Blast's Security. [7] VX heaven. [8] “A malware tool chain : active collection, detection, and analysis,” NBL, National Chiao Tung University. [9] U. Bayer, I. Habibi, D. Balzarotti, E. Krida, and C. Kruege, “A view on current malware behaviors,” Proceedings of the 2 nd USENIX Workshop on Large-Scale Exploits and Emergent Threats : botnets, spyware, worms, and more, pp , Apr , [10] U. Bayer, C. Kruegel, and E. Kirda, “TTAnalyze: a tool for analyzing malware,” Proceedings of 15 th European Institute for Computer Antivirus Research, Apr [11] P. M. Comparetti, G, Salvaneschi, E. Kirda, C. Kolbitsch, C. Kruegel, and S. Zanero, ”Identifying dormant functionality in malware programs,” Proceedings of Security and Privacy (SP), 2010 IEEE Symposium, pp , May 16-19, [12] M. Egele, C. Kruegel, E. Kirda, H. Yin, and D. Song, “Dynamic spyware analysis,” Proceedings of USENIX Annual Technical Conference, pp , Jun [13] J. Kinder, S. Katzenbeisser, C. Schallhart, and H. Veith, “Detecting malicious code by model checking,” Proceedings of the 2 nd International Conference on Intrusion and Malware Detection and Vulnerability Assessment (DIMVA’05), pp , 2005.

Copyright © 2011, References (Cont.) [14] W. Liu, P. Ren, K. Liu, and H. X. Duan, “Behavior-based malware analysis and detection,” Proceedings of Complexity and Data Mining (IWCDM), pp , Sep , [15] C. Mihai and J. Somesh, “Static analysis of executables to detect malicious patterns,” Proceedings of the 12 th conference on USENIX Security Symposium, Vol. 12, pp , Dec , [16] A. Moser, C. Kruegel, and E. Kirda, “Exploring multiple execution paths for malware analysis,” Proceedings of 2007 IEEE Symposium on Security and Privacy, pp , May 20-23, [17] J. Rabek, R. Khazan, S. Lewandowskia, and R. Cunningham, “Detection of injected, dynamically generated, and ob-fuscated malicious code,” Proceedings of the 2003 ACM workshop on Rapid malcode, pp , Oct , [18] A. Sabjornsen, J. Willcock, T. Panas, D. Quinlan, and Z. Su, “Detecting code clones in binary executables,” Proceedings of the 18 th international symposium on Software testing and analysis, pp , [19] M. Shankarapani, K. Kancherla, S. Ramammoorthy, R. Movva, and S. Mukkamala, “Kernel machines for malware classification and similarity analysis,” Proceedings of Neural Networks (IJCNN), The 2010 International Joint Conference, pp.1 - 6, Jul , [20] C. Wang, J. Pang, R. Zhao, W. Fu, and X. Liu, “Malware detection based on suspicious behavior identification,” Proceedings of Education Technology and Computer Science, Vol. 2, pp , Mar. 7-8, [21] C. Willems, T. Holz, and F. Freiling. “Toward automated dynamic malware analysis using CWSandbox,” IEEE Security and Privacy, Vol. 5, No. 2, pp , May , 2007.