nd Joint Workshop between Security Research Labs in JAPAN and KOREA Polymorphic Worm Detection by Instruction Distribution Kihun Lee HPC Lab., Postech 2 nd Joint Workshop between Security Research Labs in Korea and Japan

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 2/13 Contents  Introduction  Background Polymorphic Worm  Related Works Polygraph Using a Control Flow Graph  Problem Definition  Proposal Idea  Conclusions and Future Works

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 3/13 Introduction  Toward defending against Internet worms, NIDSs have been proposed by the security community. IDS searches inbound traffic for known patterns, or “signature”.  Unfortunately, the worms became more sophisticated! Substantially changes its payload.

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 4/13 Polymorphic Worm (1/2)  IDSs search for similar byte sequence Author of worm have to prevent this:  ciphering techniques  obfuscating the decryption routine  Can’t find a sufficiently specific sequence. Background Typical polymorphic worm structure

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 5/13 Polymorphic Worm (2/2) Background Polymorphic worm cycle

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 6/13 Polygraph  The system that proposed to defense the polymorphic worms Idea : use the combination of “short invariant contents” Assumption : combination of many general contents is sufficiently specific. Problems :  Even though combine all of them, an outcome can be remain too general.  Decision time is too late.  Token of the signature can be located after a long garbage sequence. Related Works

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 7/13 Using a Control Flow Graph  A complementary approach to reach the same goal with Polygraph Idea : using structural information of executables Assumption : at least some parts of a worm contain executable machine code.  Decryptor part of polymorphic worm Problems :  Because of huge performance overhead, it cannot operate on-line.  generating a graph, coloring the graph  Manufacturing the control flow is not difficult technique. Related Works

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 8/13 Problem Definition  Scope of problem: The worm of which propagation mechanism is using a vulnerability of a server application.  Assumption : At least some parts of a worm contain executable machine code. Linear disassemble has a little overhead so that can operate on- line.  Problem definition Make a decision whether the inbound packet has an executable code or not. Make a decision whether the executable code is a polymorphic exploit code or a legitimate code.

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 9/13 Motivated Experiment  If disassemble the packet, Case 1 : executable code tend to  Kinds of instruction :   Number of each instruction :  Case 2 : non-executable code tent to  Kinds of instruction :   Number of each instruction :   Decoding error (invalid instruction) :  number of each instruction ( sorted by decreasing order)

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 10/13 Find Executable Code  Let K = “kinds of instruction”, T = “total number of instructions”, E = “the number of decoding error”.  Calculate the expression : Non-executable code : tend to very small value. Executable code : tend to relatively large value. Threshold Distinguish between executable code and non-executable code Proposal Idea

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 11/13 Distinguish Legitimate Code (1/2)  Use the “verifying instruction” For example, “call”, “ret”, “int”, etc.  Typically, normal executable code has a lot of “call” instructions.  One “call” instruction per 10~15 instructions.  NOP sled cannot include any “call” instruction.  Decryptor is a very simple routine so that it rarely has a “call” instruction.  Moreover, decryptor can’t know the address of the function of dynamic linking library. Proposal Idea

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 12/13 Distinguish Legitimate Code (2/2)  Let V = “the number of verifying instruction”  Calculate the expression : Polymorphic exploit code : is relatively small value. Legitimate code : is relatively large value. Threshold Distinguish between exploit code and legitimate code Proposal Idea

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 13/13 Conclusions and Future Works  Conclusions Proposed idea can identify and isolate the polymorphic worm. It is based on static analysis; so it can runs in real- time. It can discover the worm traffic by not flow level but packet level examination.  Future Works Refine the idea. Investigate more samples to get a generality. How to extract a signature?

nd Joint Workshop between Security Research Labs in JAPAN and KOREA 14/13 References  J. Newsome, B. Karp, and D. Song. Polygraph: Automatically Generating Signatures for Polymorphic Worms. In IEEE Symposium on Security and Privacy,  C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna. Polymorphic Worm Detection Using Structural Information of Executables. In RAID  O. Kolesnikov, and W. Lee. Advanced Polymorphic Worms: Evading IDS by Blending in with Normal Traffic. In 12th ACM conference on Computer and communications security.  P. Akritidis, E.P. Markatos, M. Polychronakis, and K. Anagnostakis. STRIDE: Polymorphic Sled Detection Through Instruction Sequence Analysis. 12th ACM conference on Computer and communications security.  T. DeTristan, T. Ulenspiegel, Y. Malcom, and M. von Underduk. Polymorphic Shellcode Engine Using Spectrum Analysis.  Etc.