Automatically Inferring Temporal Properties for Program Evolution Jinlin Yang and David Evans 15 th IEEE International Symposium on Software Reliability Engineering 5 November 2004 Saint-Malo, France University of Virginia Computer Science

2 Temporal Properties Constrain ordering of events –Essential for program correctness (file open before read) Reveal important differences between programs: S  P  not a chef But, hard for humans to document correctly [Holzmann’s FSE 2002 keynote] Can we infer useful temporal properties automatically? This talk argues: Yes!

3 Dynamically Inferring Properties Inherently unsound: guessing properties of all executions by looking at a small number Value-based invariants –Daikon [Ernst, TSE, Feb 01] Temporal properties –Mining specification [Ammons, POPL 02] –Extracting component interfaces [Whaley, ISSTA 02] We focus on relationship between 2 or 3 events: automation, scalability

4 System Overview Inferred Properties 1 Different Properties Difference Analyzer Inferred Properties 2 Shared Properties Program Version 1 Program Version 2 Dynamic Inference … …

5 Property Inference Inferred Properties 1 Different Properties Difference Analyzer Inferred Properties 2 Shared Properties Program Version 1 Program Version 2 Dynamic Inference … … Program Instrumented Program Instrumentation Test Suite Execution Traces Testing Inferred Properties Candidate Property Patterns Inference

6 Candidate Property Patterns Response pattern [Dwyer, ICSE ‘99] [-P]* (P [-S]* S [-P]*)* –Filtered version (all non P/S events removed): S*(P + S + )* SPPSPS  SPSP  Too weak for our purposes

7 Partial Order of Patterns MultiEffect (PS + )* MultiCause (P + S)* EffectFirst S*(PS)* Alternating (PS)* OneCause S*(PS + )* OneEffect S*(P + S)* CauseFirst (P + S + )* Response S*(P + S + )* Stricter ΛΛ Λ Λ 3 Primitive Patterns 4 Derived Patterns

8 Inferring Properties For all pairs of events, P and S: –Check which (if any) of OneCause, CauseFirst and OneEffect are satisfied –Infer the strictest property satisfied if more than one primitive is MultiEffect (PS + )* MultiCause (P + S)* EffectFirst S*(PS)* Alternating (PS)* OneCause S*(PS + )* OneEffect S*(P + S)* CauseFirst (P + S + )* ΛΛ Λ Λ

9 Traces Example Trace 1: PSPSTrace 2: PPSAll Traces CauseFirst (P+S+)* OneCause S*(PS+)* OneEffect S*(P+S)* For any two events, determine the strictest pattern satisfied by all traces

10 Traces Example Trace 1: PSPSTrace 2: PPSAll Traces CauseFirst (P+S+)* OneCause S*(PS+)*  OneEffect S*(P+S)* For any two events, determine the strictest pattern satisfied by all traces

11 Traces Example Trace 1: PSPSTrace 2: PPSAll Traces CauseFirst (P+S+)* OneCause S*(PS+)*  OneEffect S*(P+S)* For any two events, determine the strictest pattern satisfied by all traces CauseFirst  OneEffect  MultiCause

12 Implementation Program Instrumented Program Instrumentation Test Suite Execution Traces Testing Inferred Properties Candidate Property Patterns Inference Automated for Java using JRat Method entry/exit events Manually for C code or output (today’s examples) Prototype (described in paper) 900 lines of Perl New implementation 8K lines of Java Improved performance Analyses, ranking heuristics

13 Experiments Hypotheses –We can automatically extract interesting temporal properties –Differences in inferred temporal properties among multiple versions can reveal interesting things Target programs –Tour bus simulator (8 student submissions) –OpenSSL (0.9.6, d)

14 Tour Bus Simulator Bus and each passenger are a separate thread Assignment in Fall 2003 graduate-level course (before we started this project) 8 submissions from the instructor (all believed to be correct)

15 Testing > cville_bus –N 2 –C 1 –T 2 Bus waiting for trip 1 Passenger 0 boards Bus drives around Charlottesville Passenger 0 exits Bus waiting for trip 2 Passenger 1 boards Bus drives around Charlottesville Passenger 1 exits Bus stops for the day N, the number of people C, the capacity of the bus T, the number of trips Executed each submission with 100 randomly generated inputs, where 20 < C ≤ 40 C+1 ≤ N ≤ 2C 1 ≤ T ≤ 10 Event traces extracted from program output (no need to instrument program)

16 Differences Reveal Problems Pattern7 Correct Versions 1 Faulty Version Alternating wait  drives MultiEffect drives  boards wait  exits wait  boards wait  drives wait  boards MultiCause boards  drives CauseFirst boards  exits drives  exits wait  exits –wait  drives not Alternating Bus drives around before all passengers exit Bug in locking code (misplaced synchronization) –drives  exits, wait  exits not MultiEffect Bus drives around twice before letting passengers exit Missing synchronization

17 OpenSSL Widely used implementation of the Secure Socket Layer protocol 6 versions [0.9.6, 0.9.7, 0.9.7a-d] between Sept 2000 and March 2004 We focus on the handshake protocol –38 different event types

18 SR_CLNT_HELLO SW_SRVR_HELLO SW_CERT SW_KEY_EXCH SW_CERT_REQ SW_SRVR_DONE SR_CERT SR_KEY_EXCH SR_CERT_VRFY SR_FINISHED SW_CHANGE SW_FINISHED BEFORE+ACCEPT SW_FLUSH OK SW_FLUSH Client Server

19 Testing Manually instrumented server to record handshake events Executed each version of server with 1000 randomly generated clients –Client modified advance to a randomly selected state with 5% probability

20 Inferred Alternating Patterns a0.9.7b0.9.7c0.9.7d SR_KEY_EXCH  SR_CERT_VRFY SW_CERT  SW_KEY_EXCH SW_SRVR_DONE  SR_CERT Documented change: ignore unrequested client certificates Fixed bug causing server crashes Race condition (present in all versions) 7 alternating patterns same for all versions

21 Partitioning Traces All Traces Correct Clients (never jump to random state) Faulty Clients No Server Error Server Error Follows SSL specification Missing Alternating patterns: allows handshake cycle Segmentation fault (<0.9.7d)

22 Performance Prototype implementation (described in paper) –Up to 3-4 minutes for examples (Pentium 4, 3GHz, 1G RAM) –Too slow (several days) for larger programs –Scales approximately as trace length  events 2 New implementation –Scales approximately as trace length  events –Analyze these experiments in <.5s Bus: 102 events, 100 traces, 222 events/trace: 0.4s OpenSSL: 35 events, 1000 traces, 18 events/trace: 0.3s –Feasible to analyze large systems Thousands of different events Traces with millions of events Preliminary results from experiments with JBoss

23 Summary of Experiments Useful in program evolution –Reveal interesting changes in OpenSSL –Identify unexpected differences Revealed bugs –Tour bus: identified faulty implementation Multiple implementations are rare, but multiple representations are common (i.e. design, model, code) –OpenSSL: Differences between versions revealed Flaws found by partitioning traces

24 Future Work/Research Questions Can we make this feasible in practice? –Automatically identify interesting events –Heuristics to identify important patterns –Scalability and automation Can we make it more useful? –More expressive patterns: more events, combine with data-flow –Understand impact of different testing strategies

25 Conclusion Automatically inferring temporal properties is feasible Even very simple property patterns reveal interesting program properties

26 Questions? This work is funded in part by the National Science Foundation. Thanks: Marty Humphrey for providing the student submissions, Chengdu Huang for help with OpenSSL, Joel Winstead, the anonymous reviewers, and anonymous CS650 students for writing buggy code.