An Empirical Study of the Reliability in UNIX Utilities Barton Miller Lars Fredriksen Brysn So Presented by Liping Cai

Overview Introduction Tools Tests Results and Analysis Conclusions

Introduction Why the study is important Contributes to the testing community a large list of real bugs; Finds the bugs that might indicate future security holes; Finds that crashes might be caused by carelessly typing; Would like some meaningful and predictable response when inadvertently feeding programs noisy input; Major utilities should not crash because of noisy phone lines; Would like to see the interactions between random testing and more traditional industrial software testing;

Introduction The project proceeded in four steps: Construct programs to generate random characters and to help test interactive utilities; Use these programs to test a large number of utilities; Identify the strings that crash the programs; Identify the cause of the program crashes and categorize the common mistakes;

The Tools Fuzz: generating random Input Strings fuzz –o outfile | deqn -- capable of producing both printable and control characters, only printable characters, or either of these groups along with the NULL character. -- can specify a delay between each character. -- allows you to specify the seed for the random number generators, to provide for repeatable test. -- can record its output stream in a file, in addition to printing to its standard output. -- there are options to randomly insert newline characters in the output stream, and to limit the length of the output stream.

The Tools Ptyjig: Testing Interactive Utilities -- allocates a pseudo-terminal files -- starts the specified utility program. -- ptyjig passes characters that are sent to its input through the pseudo- terminal to be read by the utility -- example: fuzz –o outfile | ptyjig vi

The Tools The scripts: automating the tests Executes all the utilities for a given set of input characteristics Checks for the existence of a “core” file after each utility terminates; indicating the crash of that utility The core file and offending input data file are saved for later analysis

The tests 7 versions of UNIX 88 utility programs such as the mail program, screen editors, compilers and document formatting packages Each utility program was subjected to several different types of input streams. The major variations in test data were including control characters, NULL, and maximum length of the input stream. The input streams for interactive utilities have slightly different characteristics. To avoid overflowing the input buffers on the terminal device, the input was split into random length lines with a mean length of 128 characters.

The results and analysis Were able to crash or hang a significant number of utility programs on each system(from 24-33%); Detected 2 types of error results, crashing and hanging. Considered crashed if the program terminated producing a core file Considered hung if the program continued executing producing no output while having available input or continued to produce output after its input had stopped. A utility might crash on one system but not on another; Grouped the causes of the crashes into different categories;

The results and analysis (Crash causes) Pointer/Arrays Not checking for exceeding the range of the array. References through a null pointer. Not Checking Return code Sign of careless programming Input Functions gets(), scanf() – no means of bounds checking are possible. Sub-Process Access to sub-process should be carefully controlled or you should insure that the program input to the sub-process is first checked. Alternatively, the utility should be programmed to tolerate with the failure of a sub-process

The results and analysis (Crash causes) Interaction Effects !o%8f causes the VAX version of csh to crash and !o% f hung most versions of csh Bad Error Handler The error recovery routine only increments a count of the number of errors detected, did not exit the code. Signed Characters Can be considered sub-case of pointer/array error Race Conditions Undetermined Errors

Conclusions (Comments on the Results) All Arrays reference should be checked for valid bounds All input fields should be bounded Check all system call return values Pointer values should often be checked before being used. Judiciously extend your trust to others If you redefine something to look too much like something else, you may eventually forget about the redefinition Error handlers should handle errors Goto statements are generally a bad idea

Conclusions (Comments on Lurking Bugs) Bugs often be ignored It is often difficult to isolate exactly what activity caused the program to crash. It quicker to try a slightly different method to get the current job done than it is to find and report a bug. The number of bugs in UNIX might also be explained by its evolution. UNIX should not be singled out as a buggy operating system

FUZZ-Revisited First, the continued prevalence of bugs in the basic UNIX utilities seems a bit disturbing. Second, the reliability of network services and X- Window servers is good news. Third, X-Window applications are no less prone to failure (and seem to be more so) than the basic utilities. Fourth, the reliability of the freely-distributed GNU and Linux software was surprisingly good, and noticeably better than the commercially produced software.

References ftp://grilled.cs.wisc.edu/fuzz ftp://ftp.cs.wisc.edu/paradyn/technical_ papers/fuzz-revisited.pdf ftp://ftp.cs.wisc.edu/paradyn/technical_ papers/fuzz-revisited.pdf ftp://ftp.cs.wisc.edu/paradyn/technical_ papers/fuzz-nt.pdf ftp://ftp.cs.wisc.edu/paradyn/technical_ papers/fuzz-nt.pdf