Applying Dynamic Analysis to Test Corner Cases First Penka Vassileva Markova Madanlal Musuvathi

Motivation Many Systems can be modeled as state machines that interact with each other e.g. Networking Protocols Subtle bugs in such systems involve sequences of rare corner cases: TCP Impl in FIN-WAIT state, on receiving a retransmitted FIN packet, sends an ACK with a corrupted checksum. AODV on receiving a RERR packet out of order, just after a link failure, leads to a routing loop. Conventional testing misses these errors Such errors are difficult to detect using Static Analyses

Explicit State Enumeration Explicit State Enumeration is a promising approach to test such systems. It systematically explores the entire state space of the system. Better than “brute force” testing as it visits each system state only once Usually runs out of resources before it terminates The trick is to guide the search to interesting parts of state space Our Proposal: Use dynamic analyses to detect corner cases Search corner cases first

CMC – A C Model Checker CMC is an explicit state enumerator Runs C Code directly State consists of variables in the program Current Implementation restricts state to Global and Heap variables. State transitions are performed by transferring control to different points in the C code. Current Implementation requires each transition to be performed by a handler function in the code By executing code, CMC is able to handle implementation errors as well as protocol errors. It is Sound: No false positives

An Example SynSentListen SYN Client Connect SynSentListen empty SynSentSynRcv SYN-ACK Pkt Loss Pkt Recv ClosedListen empty ClientServer Network Retransmit Pkt Recv Pkt Loss State Transitions are performed by the Actual C Code

Fundamental Problem Real systems have infinite or large state space Memory (not time) is the key bottleneck Need to maintain a queue of unexplored states Each state can be as huge as 10K Partial solutions are available However, prioritizing state search essential Visit interesting states before running out of resources Can flush out low priority states to the disk

Prioritizing the State Search Idea Gather information from the runs that have been executed Use this information to rank how interesting the state generated from the current run are Techniques: Track values and simple invariants Look for complex “invariants” Path profiling

Values and simple invariants Tracking values and simple invariants: Inspiration: DIDUCE On-line detection Look at simple invariants, relax after a counterexample One heuristic: at exit point track the variables that are in the state Observation so far: get saturated very fast Use knowledge of the code or algorithm with heuristics to choose other expressions of interest

Complex Invariants More complex relationships between variables a>b may be more important for the output than the exact values of a or b. Inspiration: Daikon Overhead management analyze the collected information in intervals Observation: the number of bugs found goes down with time so, after the first stages it is worth spending the time to analyze such information Only look at some of the possible invariants at a time Use static analysis or heuristics to suggest more interesting invariants to gather information on After invalidating most of the current invariants, can add a new set to track and give it a lower confidence degree Side effect: can suggest real invariants But should reflect intervals on confidence level

Profiling Path/Edge Profiling Observation: code coverage matters Keep track about which branches/paths of the program has been taken more often then others Use existing profiling techniques

Challenges and Future Work Challenges Memory management Scaling Picking the right heuristics Evaluating which heuristics are more useful Future work Look into more complicated protocols Can we guide the generation of the test set? Intuition: executing the code at least once is probably good Unless it was the assert(false) case Try to find a counterexample and test case such that to get to this code?

Related Work Diduce Daikon JavaPathfinder Uses thread-interleaving heuristic to model-check A lot of work on Profiling A lot of Model Checking work