Presentation is loading. Please wait.

Presentation is loading. Please wait.

Automatic Protocol Format Reverse Engineering through Context-Aware Monitored Execution Zhiqiang Lin 1 Xuxian Jiang 2, Dongyan Xu 1, Xiangyu Zhang 1 1.

Published byJulian Armstrong Modified over 9 years ago

Similar presentations

Presentation on theme: "Automatic Protocol Format Reverse Engineering through Context-Aware Monitored Execution Zhiqiang Lin 1 Xuxian Jiang 2, Dongyan Xu 1, Xiangyu Zhang 1 1."— Presentation transcript:

1 Automatic Protocol Format Reverse Engineering through Context-Aware Monitored Execution Zhiqiang Lin 1 Xuxian Jiang 2, Dongyan Xu 1, Xiangyu Zhang 1 1 Purdue University 2 George Mason University February 12 th, 2008 The 15 th Annual Network and Distributed System Security Symposium

2 Motivation  Protocol reverse engineering  A process to recover protocol specifications  E.g., fields and their relationships  Applications:  Network-based Intrusion detection  Network management  Penetration test  …

3 Challenges 0x0040: cd46 4745 5420 2f6e 6577 732e 6874 6d6c 0x0050: 2048 5454 502f 312e 300d 0a55 7365 722d 0x0060: 4167 656e 743a 2057 6765 742f 312e 3130 0x0070: 2e32 2028 5265 6420 4861 7420 6d6f 6469 0x0080: 6669 6564 290d 0a41 6363 6570 743a 202a 0x0090: 2f2a 0d0a 486f 7374 3a20 3132 392e 3137 0x00a0: 342e 3838 2e37 310d 0a43 6f6e 6e65 6374 0x00b0: 696f 6e3a 204b 6565 702d 416c 6976 650d. 0x00c0: 0a0d 0a  Multiple fields in a single message  Non-static size of fields  Complex relationships among protocol fields Sequential Parallel Hierarchical

4 Challenges HTTP-Request = Request-Line (( general-header | request-header | entity-header ) CRLF)* CRLF [ message-body ] Request-Line = Method SP Request-URI SP HTTP-Version CRLF Parallel Sequential Hierarchical A BNF Specification of HTTP Request (RFC2616)

5 Related Work  Network Trace  Protocol Informatics  Discoverer [W. Cui et. al. Security’07]  Binary Analysis  Polyglot [J. Caballero et. al. CCS’07]  Automatic Network Protocol Analysis [G. Wondracek et. al. NDSS’08]

6 Observation 119 int read_header(int sid) {... 137 if (sscanf(line, "%[^ ] %[^ ] %[^ ]", conn[sid].dat->in_RequestMethod, conn[sid].dat->in_RequestURI, conn[sid].dat->in_Protocol)!=3)... 147 while (strlen(line)>0) {... 154 if (strncasecmp(line, "Cookie: ", 8)==0) 155 strncpy(conn[sid].dat->in_Cookie, (char *)&line+8, sizeof(conn[sid].dat->in_Cookie)-1); 156 if (strncasecmp(line, "Host: ", 6)==0) 157 strncpy(conn[sid].dat->in_Host, (char *)&line+6, sizeof(conn[sid].dat->in_Host)-1); … 160 if (strncasecmp(line, "User-Agent: ", 12)==0) 161 strncpy(conn[sid].dat->in_UserAgent, (char *)&line+12, sizeof(conn[sid].dat->in_UserAgent)-1); 162 }... 187 } Code snippet in http.c (null-httpd-0.5.0)

7 AutoFormat -- Basic Idea Execution Context Protocol Fields G E T / n e w s … Context One Field Another Field

8 System Overview Context-aware Execution Monitor GET /news.html 0 'G' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr 1 'E' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr 2 'T' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr … 24 ‘\n’ main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr … 0 'G' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x1F7F3 ->0xF5A8->ap_read_request->ap_getword_white 0 'G' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr 1 'E' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr 2 'T' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr … 24 ‘\n’ main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr … 0 'G' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x1F7F3 ->0xF5A8->ap_read_request->ap_getword_white Log call stackEIP input

9 Protocol Field Identifier  Analyze log file  Step 1: build protocol field tree from the logged data.  Step 2: refine the tree using three heuristics  Step 3: output the result

10 Example: Apache log data 0 'G' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr 1 'E' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr 2 'T' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr … 24 ‘\n’ main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x4BA56A2 ->0xF5A8->ap_read_request->ap_rgetline_core->ap_get_brigade->0x2D2CE->ap_get_brigade->0x2D667 ->apr_brigade_split_line->memchr … 24 '\n' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x26187 ->0xF5A8->ap_read_request->ap_rgetline_core 23 '\r‘ main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x26322 ->0xF5A8->ap_read_request->ap_rgetline_core 0 'G' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x1F7F3 ->0xF5A8->ap_read_request->ap_getword_white 1 'E' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x1F7F3 ->0xF5A8->ap_read_request->ap_getword_white 2 'T' main->ap_mpm_run->0x15C57->0x15B38->0x15941->ap_process_connection->ap_run_process_connection 0x1F7F3 ->0xF5A8->ap_read_request->ap_getword_white … GET /news.html HTTP/1.0\r\n\r\n GET

11 Step 1 -- Building Protocol Field Tree root GET /news.html HTTP/1.0\r\n User−Agent: Wget/1.10.2 (Red Hat modified)\r\nAccept: */*\r\n…. GET /news.htmlGET HTTP/1.0

12 Step 1: Building Protocol Field Tree GET /news.html HTTP/1.0\r\n H news.html GET GET /news.html GET /news.html HTTP/1.0\r\n HTTP/1.0\r\n TTP/1.0 / / / news.html H H TTP/1.0

13 Step 2: Refinement (Tokenization) GET /news.html HTTP/1.0\r\n /news.html GET GET /news.html GET /news.html HTTP/1.0\r\n HTTP/1.0\r\n /news.html HTTP/1.0 GET /news.html HTTP/1.0\r\n H news.html GET GET /news.html GET /news.html HTTP/1.0\r\n HTTP/1.0\r\n TTP/1.0 / / / news.html H H TTP/1.0

14 Step 2: Refinement (Redundant Node Deletion) GET /news.html HTTP/1.0\r\n /news.html GET GET /news.html GET /news.html HTTP/1.0\r\n HTTP/1.0\r\n /news.html HTTP/1.0 GET /news.html HTTP/1.0\r\n /news.html GET GET /news.htmlHTTP/1.0\r\n

15 Step 2: Refinement (Node Insertion) GET /news.html HTTP/1.0\r\n /news.html GET GET /news.htmlHTTP/1.0\r\n GET /news.html HTTP/1.0\r\n /news.html GET GET /news.htmlHTTP/1.0\r\n

16 Step 3: Output the Result Parallel & Sequential Hierarchical GET /news.html HTTP/1.0\r\n /news.html GET GET /news.html HTTP/1.0\r\n /news.html GET HTTP/1.0\r\n 1 2 3 4

17 Evaluation  Implemented on top of Valgrind-3.2.3  Also applies to QEMU, PIN  Benchmark  30 messages with six known protocols and one unknown protocol.  Evaluation Metric  Re: Ratio of exact match |(A ∩ W)| / |W|  A: set of fields identified by AutoFormat  W: set of fields identified by Wireshark

18 Overall Result Re(F) = 88.5% Re(H) = 98.0% Re(P) = 100.0% Re=93.4% Re(F): Re for finest-grained fields Re(H): Re for hierarchical fields Re(P): Re for parallel fields

19 Experimental Result – Slapper Worm Nested data structure declaration Compiler inserted gap 1 2 3 1 2

20 Discussion  Dynamic Trace Dependency  Byte granularity  Protocol State Machine  Obfuscated binaries

21 Conclusion  AutoFormat  A tool for automatic protocol format extraction.  Key insight  A protocol implementation is programmed to recognize the protocol format and usually contains protocol field-specific execution context, and we can actually leverage such context to infer the hierarchical structure of protocol fields, and even get their BNF structures.

22 Thank you For more information: {zlin, dxu, xyzhang}@cs.purdue.edu xjiang@gmu.edu Q & A

Download ppt "Automatic Protocol Format Reverse Engineering through Context-Aware Monitored Execution Zhiqiang Lin 1 Xuxian Jiang 2, Dongyan Xu 1, Xiangyu Zhang 1 1."

Similar presentations

Parallel Virtual Machine Rama Vykunta. Introduction n PVM provides a unified frame work for developing parallel programs with the existing infrastructure.

Parallel Virtual Machine Rama Vykunta. Introduction n PVM provides a unified frame work for developing parallel programs with the existing infrastructure.
INTROPERF: TRANSPARENT CONTEXT- SENSITIVE MULTI-LAYER PERFORMANCE INFERENCE USING SYSTEM STACK TRACES Chung Hwan Kim*, Junghwan Rhee, Hui Zhang, Nipun.

INTROPERF: TRANSPARENT CONTEXT- SENSITIVE MULTI-LAYER PERFORMANCE INFERENCE USING SYSTEM STACK TRACES Chung Hwan Kim*, Junghwan Rhee, Hui Zhang, Nipun.
TaintScope: A Checksum-Aware Directed Fuzzing Tool for Automatic Software Vulnerability Detection Tielei Wang 1, Tao Wei 1, Guofei Gu 2, Wei Zou 1 1 Peking.

TaintScope: A Checksum-Aware Directed Fuzzing Tool for Automatic Software Vulnerability Detection Tielei Wang 1, Tao Wei 1, Guofei Gu 2, Wei Zou 1 1 Peking.
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University Extraction of.

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University Extraction of.
Linear Obfuscation to Combat Symbolic Execution Zhi Wang 1, Jiang Ming 2, Chunfu Jia 1 and Debin Gao 3 1 Nankai University 2 Pennsylvania State University.

Linear Obfuscation to Combat Symbolic Execution Zhi Wang 1, Jiang Ming 2, Chunfu Jia 1 and Debin Gao 3 1 Nankai University 2 Pennsylvania State University.
1 Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation David Brumley, Juan.

1 Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation David Brumley, Juan.
Distributed Process Scheduling Summery Distributed Process Scheduling Summery BY:-Yonatan Negash.

Distributed Process Scheduling Summery Distributed Process Scheduling Summery BY:-Yonatan Negash.
1 Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation David Brumley, Juan.

1 Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation David Brumley, Juan.
Report on Intrusion Detection and Data Fusion By Ganesh Godavari.

Report on Intrusion Detection and Data Fusion By Ganesh Godavari.
4/27/2006 ELEC7250: White 1 ELEC7250 VLSI Testing: Final Project Andrew White.

4/27/2006 ELEC7250: White 1 ELEC7250 VLSI Testing: Final Project Andrew White.
The abs_path in a URI If the abs_path is not present in the URL, it must be given as / in a Request-URI for a resource. Thus, if a user points a browser.

The abs_path in a URI If the abs_path is not present in the URL, it must be given as "/" in a Request-URI for a resource. Thus, if a user points a browser.
Data and Computer Communications Eighth Edition by William Stallings Lecture slides by Lawrie Brown Chapter 23 – Internet Applications Internet Directory.

Data and Computer Communications Eighth Edition by William Stallings Lecture slides by Lawrie Brown Chapter 23 – Internet Applications Internet Directory.
Process Coloring: an Information Flow-Preserving Approach to Malware Investigation Eugene Spafford, Dongyan Xu (Presenter) Department of Computer Science.

Process Coloring: an Information Flow-Preserving Approach to Malware Investigation Eugene Spafford, Dongyan Xu (Presenter) Department of Computer Science.
School of Computer Science and Information Systems

School of Computer Science and Information Systems
Winter Retreat - 2003 Connecting the Dots: Using Runtime Paths for Macro Analysis Mike Chen, Emre Kıcıman, Anthony Accardi, Armando Fox, Eric Brewer

Winter Retreat Connecting the Dots: Using Runtime Paths for Macro Analysis Mike Chen, Emre Kıcıman, Anthony Accardi, Armando Fox, Eric Brewer
General approach to exploit detection and signature generation White-box  Need the source code Gray-box  More accurate. But need to monitor a program's.

General approach to exploit detection and signature generation White-box  Need the source code Gray-box  More accurate. But need to monitor a program's.
The abs_path in a URI If the abs_path is not present in the URL, it must be given as / in a Request-URI for a resource. Thus, if a user points a browser.

The abs_path in a URI If the abs_path is not present in the URL, it must be given as "/" in a Request-URI for a resource. Thus, if a user points a browser.
Strategies to relate the program and problem domains using code instrumentation Mario Marcelo Berón University of Minho Pedro Rangel Henriques University.

Strategies to relate the program and problem domains using code instrumentation Mario Marcelo Berón University of Minho Pedro Rangel Henriques University.
2  Problem Definition  Project Purpose – Building Obfuscator  Obfuscation Quality  Obfuscation Using Opaque Predicates  Future Planning.

2  Problem Definition  Project Purpose – Building Obfuscator  Obfuscation Quality  Obfuscation Using Opaque Predicates  Future Planning.
PJSISSTA '001 Black-Box Test Reduction Using Input-Output Analysis ISSTA ‘00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois.

PJSISSTA '001 Black-Box Test Reduction Using Input-Output Analysis ISSTA ‘00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois.

Similar presentations

Ads by Google