A New Fuzzing Technique for Software Vulnerability Testing IEEE CONSEG 2009 Zhiyong Wu 1 J. William Atwood 2 Xueyong Zhu 3 1,3 Network Information Center University of Science and Technology of China Hefei, Anhui, China 2 Department of Computer Science and Software Engineering Concordia University Montreal, Quebec, Canada

2009/12/19Conseg 09 Fuzzing for Software Vulnerability2 Contents 2 1.Introduction and Motivation 2.FTSG Model 3.Related Techniques Static analysis Dynamic binary instrument and dynamic trace I/O analysis 4.GAMutator 5.Prototype System: DXFuzzing 6.Validation 7.Experiments 8.Conclusion

2009/12/19Conseg 09 Fuzzing for Software Vulnerability3 1 Introduction and Motivation 1 Introduction and Motivation C code of a vulnerable procedure 3 int process_chunck(char* head_str, char* data_str, char* program checksum){ char buf[60]; char buf1[32]; char buf2[32]; memset(buf, 0, 60); if ( true == strong_check(head_str,data_str,program checksum)){ if （ strlen(head_str) > 32 || strlen(data_str) >32) return -1; strcpy(buf1, head_str); strcpy(buf2, data_str); strcat(buf, head_str); strcat(buf, data_str);//error return 1; } else return -1; } knowledge-based fuzzing could pass it easily one-dimension m&g strategy can’t overflow if length(head_str) = 16 and length(data_str) = 20

2009/12/19Conseg 09 Fuzzing for Software Vulnerability4 2 FTSG Model 4 FTSG: Fuzzing Test Suites Generation FTSG ＝ ( s, L, N, C, F,OP,Result), OP ＝ {M, Slv}, Result ＝ {sampletree, mediumtree, newtree, testcase, testsuite}.

2009/12/19Conseg 09 Fuzzing for Software Vulnerability5 2 FTSG: Procedure for generating test cases 2 FTSG: Procedure for generating test cases by Mutation Operators and Slv 5 M = {m 1, …, m i, …, m k, GAMutator} F = {f 1,f 2, …, f e, …,f v } for (each m i in M except GAMutator) { while (!(mediumtree = m i (sampletree)) ) { newtree=Slv(mediumtree, C) } for (each f e in F) { while (!(mediumtree = GAMutator (sampletree, f e )) ) { newtree=Slv(mediumtree, C) }

2009/12/19Conseg 09 Fuzzing for Software Vulnerability6 2 FTSG: Total number of test cases 6

2009/12/19Conseg 09 Fuzzing for Software Vulnerability7 3 Related Techniques: 3 Related Techniques: Static analysis ， dynamic binary instrument and dynamic trace 7 TechniqueUsageTool Static analysis identify insecure functions IDA PRO Dynamic binary instrument get insecure functions’ dynamic input arguments values to calculate fitness value Pin Dynamic trace monitor buffer coverage Pydbg

2009/12/19Conseg 09 Fuzzing for Software Vulnerability8 3 Related Techniques: 3 Related Techniques: I/O analysis 8 MethodInstrument Target Characteristic static analysissource codefalse alarm execution- oriented analysis binary codesimple and precise

2009/12/19Conseg 09 Fuzzing for Software Vulnerability9 3 Related Techniques: 3 Related Techniques: I/O analysis: execution-oriented analysis 9 INPUTOUTPUTVALUE of O k t 1 = (a 1,a 2,…,a s,…,a n )O = {o 1,o 2, …, o k, … o n }V1V1 t 2 = (a 1,a 2,…,a s,…,a n )O = {o 1,o 2, …, o k, … o n }V2V2 t 3 = (a 1,a 2,…,a s’,…,a n )O = {o 1,o 2, …, o k, … o n }V3V3 x s influences output o k if and only if V 1 =V 2 ≠V 3 where a i ∈ D(x i ), a s’ ∈ D(x i ), a s ≠a s’

GAMutator  GAMutator mutates relative l or n in sampletree to trigger suspend vulnerability in f e.  l or n are the inputs that influence some arguments of f e. 2009/12/19Conseg 09 Fuzzing for Software Vulnerability10

Cont.  Special Characteristics of GAMutator:  A multi-dimension mutation operator.  A demand-oriented operator.  The number of test cases that GAMutator generates is not fixed.  Communicates with outside system.  The genetic algorithm here is used to generate test cases to trigger vulnerability in unsafe functions  The number of test cases generated by GAMutator is O(h). 2009/12/19Conseg 09 Fuzzing for Software Vulnerability11

2009/12/19Conseg 09 Fuzzing for Software Vulnerability12 4 GAMutator: 4 GAMutator: Heuristics and fitness function 12 Heuristics are used to generate test cases more likely to trigger vulnerability in f e in F. TWO EXAMPLES: 1 strcpy( dst, src) 2 malloc(a)

5 Prototype System: DXFuzzing 1) Locate insecure functions positions in target binary code by Program Analyzer. Record their information into database; 2) Analyze corresponding network protocols or file format in target application according to related knowledge, choose a sample file s and write a primitive xml test script manually which contains a sampletree; 3) Scheduling Engine calls XFuzzing to fuzz target application with m i and records runtime information with Program Analyzer when it is necessary. 2009/12/19Conseg 09 Fuzzing for Software Vulnerability13

2009/12/19Conseg 09 Fuzzing for Software Vulnerability14 Cont. 4) Data Mapper constructs relationships between X and F based on collected runtime information. 5) Scheduling Engine calls XFuzzing to fuzz target application with GAMutator. 14

2009/12/19Conseg 09 Fuzzing for Software Vulnerability15 6 Validation 1) Based on application-specific knowledge, DXFuzzing could generate test cases which easily pass strong program checks and validations in the program. 2) The problem of finding new combinations to trigger possible vulnerability in f e in F is especially suitable for genetic algorithm to solve. 15

Cont. 3) GAMutator does not only care about the relationships between l i and f e, but also cares about n j and f e. Because some f e in F is influenced by the n j, however, the n j is neglected in general. 4) Different from combinatorial test in black-box testing, the combination of l i or n j in DXFuzzing is decided by the I/O analysis; the values of l i or n j in some combination are refined by every generation. 2009/12/19Conseg 09 Fuzzing for Software Vulnerability16

Cont.  Execution-oriented I/O analysis in DXFuzzing is preferred here. 2009/12/19Conseg 09 Fuzzing for Software Vulnerability17

2009/12/19Conseg 09 Fuzzing for Software Vulnerability18 7 Experiments 18 LibPng library as the target application Some data are as follows: Function nameusePng.exeLibPng.dll v1.0.6 strcpy16 memcpy077 sprintf016 malloc18113 Table I insecure functions in target application IDINPUT ELEMENTS 101PngFile..IHDA_CHUNK_DATA.BitDepth 102PngFile..IHDA_CHUNK_DATA.ColorType 109PngFile..IHDA_CHUNK_DATA.Height 111PngFile..IHDA_CHUNK_DATA.Width Table II Input nodes

2009/12/19Conseg 09 Fuzzing for Software Vulnerability19 Cont. ID INSECURE FUNCTIONS 72pngrutil.c(2939):png_ptr- >row_buf=(png_bytep)png_malloc(png_ptr,row_bytes) 73pngrutil.c(2945):png_ptr- >prev_row=(png_bytep)png_malloc(png_ptr, png_uint_32)( png_ptr->rowbytes + 1)) 89pngread.c(1301):info_ptr- >row_pointers=(png_bytepp)png_malloc(png_ptr,info_ptr- >height * sizeof(png_bytep)) Table III Insecure functions influenced by input nodes

2009/12/19Conseg 09 Fuzzing for Software Vulnerability20 Cont. Figure 4. Relationships between inputs and insecure functions by static analysis Figure 5. Relationships between inputs and outputs by dynamic execution simple and precise

2009/12/19Conseg 09 Fuzzing for Software Vulnerability21 Cont. wwidth111 dBitDepth101 zArgument value of png_malloc73 Initial Values: w = 0x20, d = 0x01 w ∈ [0,0xfffffff] ， d ∈ [0,0xff].

Cont.  Further analyzing, we got d ∈ {1,2,4}.  w and d will generate 3×0x ＝ combination test cases.  However, there are only of them that could trigger this vulnerability if we set B=  For this case png_malloc could successfully allocate memory.  So the possibility is / = /12/19Conseg 09 Fuzzing for Software Vulnerability22

2009/12/19Conseg 09 Fuzzing for Software Vulnerability23 Cont. Width, BitDepth distribution when they trigger this vulnerability

2009/12/19Conseg 09 Fuzzing for Software Vulnerability24 Cont. ToolsNumber of vulnerability checkedNumber of test cases Smart Fuzzer GAFuzzing Peach DXFuzzing Table IV Vulnerabilities Found by Different Fuzzing Tools

2009/12/19Conseg 09 Fuzzing for Software Vulnerability25 Conclusion  Whitebox fuzzing is complex, time costly and there are still some problems such as path explosion, and is hard to pass strong program checks fully automatically.  Peach is an outstanding knowledge-based fuzzing tool. 25

Conclusion  DXFuzzing enriches current mutation methodology with multi-dimension input nodes mutation strategy without combinatorial explosion. So DXFuzzing could find some vulnerabilities that never will been found by one- dimension mutation fuzzing. 2009/12/19Conseg 09 Fuzzing for Software Vulnerability26

2009/12/19Conseg 09 Fuzzing for Software Vulnerability27 9 For More Information 27 For More Questions and Comments: