1. Copyright © 2012, MBL@CS.NCTU Malware Detection Based on Malicious Behaviors Using Artificial Neural Network Student: Hsun-Yi Tsai Advisor: Dr. Kuo-Chen Wang 2012/05/28

2. Copyright © 2012, MBL@CS.NCTU Outline Introduction Problem Statement Related Work Design Approach –Sandboxes –Behaviors –Proposed Algorithm –Weight Training –Malicious Degree Evaluation Conclusion and Future Works References 2

3. Copyright © 2012, MBL@CS.NCTU Introduction 3 In recent years, malware has been severe threats to the cyber security –Virus, Worms, Trojan horse, Botnet … Traditional signature-based malware detection algorithms [15] [17] Drawbacks of signature-based malware detection algorithms –Need human and time to approve –Need to update the malicious digest frequently –Easily bypassed by obfuscation methods –Can not detect zero day malware –Increase false negative rate

4. Copyright © 2012, MBL@CS.NCTU Introduction (Cont.) To conquer the shortcomings of the signature- based malware detection algorithms, behavior- based malware detection algorithms were proposed Behavior-based malware detection algorithms [14] [19] –Detect the unknown malware or the variations of known malware –Decrease false negative rate (FNR) –Increase false positive rate (FPR) To decrease the FPR, we proposed a behavioral neural network-based malware detection algorithm 4

5. Copyright © 2012, MBL@CS.NCTU Problem Statement Given –Several sandboxes –l known malware M i = {M 1,M 2, …, M l } for training –m known malware S j = {S 1, S 2, …, S m } for testing Objective –n behaviors B k = {B 1,B 2, …, B n } –n weights W k = {W 1,W 2, …, W n } –MD (Malicious degree) 5

6. Copyright © 2012, MBL@CS.NCTU Related Work MBF [14] –File, process, network, and registry actions –16 malicious behavior feature (MBF) –Three malicious degree: high, warning, and low RADUX [19] –Reverse Analysis for Detecting Unsafe eXecution (RADUX) –Collected 9 common malicious behaviors –Bayes’ theorem 6

7. Copyright © 2012, MBL@CS.NCTU Related Work (Cont.) Approach MBF [14]RADUX [19]Our Scheme Main idea Analyze behavior features Analyze API callsAnalyze malicious behaviors Number of malicious behaviors 16913 Calculating of malicious degree Non-weighted algorithmWeighted algorithm Adjusting of weights NoneBayes’ theoremArtificial neural network (ANN) False positive rateLowHighLow False negative rateNot AvailableHighLow Accuracy rateHighLowHigh 7

8. Copyright © 2012, MBL@CS.NCTU Background - Sandboxes Dynamic analysis system Isolated environment Interact with malware Record runtime behaviors 8

9. Copyright © 2012, MBL@CS.NCTU Background - Sandboxes (Cont.) Web-based sandboxes –GFI Sandbox [1] –Norman Sandbox [2] –Anubis Sandbox [3] PC-based sandboxes –Avast Sandbox [4] –Buster Sandbox Analyzer [5] 9

10. Copyright © 2012, MBL@CS.NCTU Design Approach-Behaviors Malware Host Behaviors –Creates Mutex –Creates Hidden File –Starts EXE in System –Checks for Debugger –Starts EXE in Documents –Windows/Run Registry Key Set –Hooks Keyboard –Modifies Files in System –Deletes Original Sample –More than 5 Processes –Opens Physical Memory –Deletes Files in System –Auto Start Malware Network Behaviors –Makes Network Connections DNS Query HTTP Connection File Download 10

11. Copyright © 2012, MBL@CS.NCTU Design Approach-Behaviors (Cont.) GFI [1]Norman [2]Anubis [3]Avast [4]BSA [5] Creates MutexVVV Creates Hidden FileVVVVV Starts EXE in SystemVVV Checks for DebuggerV Starts EXE in DocumentsV Windows/Run Registry Key SetVVVVV Hooks KeyboardV V Modifies File in SystemVVVVV Deletes Original SampleV V More than 5 ProcessesVVVVV Opens Physical MemoryV V Delete File in SystemVVVVV Auto Start V DNS QueryVVV HTTP ConnectionVVVV File DownloadVVV 11

12. Copyright © 2012, MBL@CS.NCTU Design Approach-Behaviors (Cont.) Ulrich Bayer et al. [10] 12

13. Copyright © 2012, MBL@CS.NCTU Design Approach-Proposed Algorithm 13

14. Copyright © 2012, MBL@CS.NCTU Design Approach – Weight Training Using Artificial Neural Network (ANN) to train weights 14

15. Copyright © 2012, MBL@CS.NCTU Design Approach – Weight Training (Cont.) Neuron for ANN hidden layer 15

16. Copyright © 2012, MBL@CS.NCTU Design Approach – Weight Training (Cont.) Neuron for ANN output layer 16

17. Copyright © 2012, MBL@CS.NCTU Design Approach – Weight Training (Cont.) Delta learning process d: expected target value Mean square error: Weight set :, : learning factor; x: input value 17

18. Copyright © 2012, MBL@CS.NCTU Design Approach-Malicious Degree 18

19. Copyright © 2012, MBL@CS.NCTU Evaluation Try to find the optimal MD value to make FPR and FNR approximate to 0. Benign Malicious MD Threshold Ambiguous 19

20. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) Matlab 7.11.0 Initial weights and bias: random by function initnw Transfer function: tangent-sigmoid function Architecture of ANN (Matlab 7.11.0): 20

21. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) Malicious sample source: Blast’s Security [6] and VX Heaven [7] websites Benign sample source: Portable execution files under windows XP SP2 Training data and testing data 21 MaliciousBenignTotal Training500 1000 Testing500 1000

22. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) Mean square error: 0.19 Execution time: 2 seconds MD threshold (according to training data) 22 Range of threshold

23. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) Choose threshold 23

24. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) Experiment results 24 TPTNFPFNFPRFNRAccuracy 4834946171.2%96.6%97.7%

25. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) 25 Approach TP / (TP + FN)FN / (TP + FN) FP / (FP + TN)TN / (FP + TN) Our Scheme 96.6%3.4% 1.2%98.8% MBF [14] Not Available 2.13%97.87% RADUX [19] 95.6%4.4% 9.8%90.2%

26. Copyright © 2012, MBL@CS.NCTU Evaluation (Cont.) Weights Accuracy Rate Weights in Hidden Layer Weights in Output Layer Random 98.8% FrequencyRandom98% 1192.42% 0.5 91% Without ANNNone91.36% 26

27. Copyright © 2012, MBL@CS.NCTU Conclusion and Future Work Conclusion –Collect several common behaviors of malwares –Compose Malicious Degree (MD) formula –The false positive rate and false negative rate is approximated to 0 –Detect unknown malware Future work –Automate the system –Implement PC-based sandboxes –Add more malware network behaviors –Classify malwares according to their typical behaviors 27

28. Copyright © 2012, MBL@CS.NCTU References [1] GFI Sandbox. http://www.gfi.com/malware-analysis-tool [2] Norman Sandbox. http://www.norman.com/security_center/security_tools [3] Anubis Sandbox. http://anubis.iseclab.org/ [4] Avast Sandbox. http://www.avast.com/zh-cn/index [5] Buster Sandbox Analyzer (BSA). http://bsa.isoftware.nl/ [6] Blast's Security. http://www.sacour.cn [7] VX heaven. http://vx.netlux.org/vl.php [8] Neural Network Toolbox. http://dali.feld.cvut.cz/ucebna/matlab/toolbox/nnet/initnw.html [9] “A malware tool chain: active collection, detection, and analysis,” NBL, National Chiao Tung University. [10] U. Bayer, I. Habibi, D. Balzarotti, E. Krida, and C. Kruege, “A view on current malware behaviors,” Proceedings of the 2nd USENIX Workshop on Large-Scale Exploits and Emergent Threats : botnets, spyware, worms, and more, pp. 1 - 11, Apr. 22-24, 2009. [11] U. Bayer, C. Kruegel, and E. Kirda, “TTAnalyze: a tool for analyzing malware,” Proceedings of 15th European Institute for Computer Antivirus Research, Apr. 2006. [12] M. Egele, C. Kruegel, E. Kirda, H. Yin, and D. Song, “Dynamic spyware analysis,” Proceedings of USENIX Annual Technical Conference, pp. 233 - 246, Jun. 2007. [13] H. J. Li, C. W. Tien, C. W. Tien, C. H. Lin, H. M. Lee, and A. B. Jeng, "AOS: An optimized sandbox method used in behavior-based malware detection," Proceedings of Machine Learning and Cybernetics (ICMLC), Vol. 1, pp. 404-409, Jul. 10-13, 2011. 28

29. Copyright © 2012, MBL@CS.NCTU References (Cont.) [14] W. Liu, P. Ren, K. Liu, and H. X. Duan, “Behavior-based malware analysis and detection,” Proceedings of Complexity and Data Mining (IWCDM), pp. 39 - 42, Sep. 24-28, 2011. [15] C. Mihai and J. Somesh, “Static analysis of executables to detect malicious patterns,” Proceedings of the 12th conference on USENIX Security Symposium, Vol. 12, pp. 169 - 186, Dec. 10-12, 2006. [16] A. Moser, C. Kruegel, and E. Kirda, “Exploring multiple execution paths for malware analysis,” Proceedings of 2007 IEEE Symposium on Security and Privacy, pp. 231 - 245, May 20-23, 2007. [17] J. Rabek, R. Khazan, S. Lewandowskia, and R. Cunningham, “Detection of injected, dynamically generated, and ob-fuscated malicious code,” Proceedings of the 2003 ACM workshop on Rapid malcode, pp. 76 - 82, Oct. 27-30, 2003. [18] K. Rieck, T. Holz, C. Willems, P. Dussel, and P. Laskov, “Learning and Classification of Malware Behavior,” in Detection of Intrusions and Malware, and Vulnerability Assessment, Vol. 5137, pp. 108-125, Oct. 9, 2008. [19] C. Wang, J. Pang, R. Zhao, W. Fu, and X. Liu, “Malware detection based on suspicious behavior identification,” Proceedings of Education Technology and Computer Science, Vol. 2, pp. 198 - 202, Mar. 7-8, 2009. [20] C. Willems, T. Holz, and F. Freiling. “Toward automated dynamic malware analysis using CWSandbox,” IEEE Security and Privacy, Vol. 5, No. 2, pp. 32 - 39, May. 20-23, 2007. [21] Y. Zhang, J. Pang, R. Zhao, and Z. Guo,"Artificial neural network for decision of software maliciousness," Proceedings of Intelligent Computing and Intelligent Systems (ICIS), Vol. 2, pp. 622 - 625, Oct. 29-31, 2010. 29