Jinlin Yang and David Evans [jinlin, Department of Computer Science University of Virginia PASTE 2004 June 7 th Dynamically Inferring Temporal Properties

PASTE 2004Jinlin Yang2 Temporal Properties Example: fopen/fclose Essential for program’s correctness Hard for humans to document correctly [Holzmann, FSE ’02 keynote] Can we infer them automatically? Yes!

PASTE 2004Jinlin Yang3 Related Work Value-based invariants –Daikon [Ernst, TSE, Feb. ‘01] –We focus on invariants in sequence of events Temporal properties –Recovering thread models [Cook, JSS ‘04] –Mining specification [Ammons, POPL ‘02] –Extracting component interfaces [Whaley, ISSTA ‘02] –We focus on relationship between 2 or 3 events: greater automation, scalability

PASTE 2004Jinlin Yang4 Our Approach Program Instrumented Program Instrumentation Test Suite Execution Traces Testing Inferred Properties Candidate Property Patterns Inference

PASTE 2004Jinlin Yang5 Program Verification Inferred Properties Model Report Derived Properties Satisfaction or Counter- examples Report Generation Validation Program Dynamic Inference

PASTE 2004Jinlin Yang6 Program Evolution Inferred Properties 1 Different Properties Difference Analyzer Inferred Properties 2 Shared Properties Program Version 1 Program Version 2 Dynamic Inference

PASTE 2004Jinlin Yang7 Property Patterns Response pattern [Dwyer, ICSE ‘99] [-P]* (P [-S]* S [-P]*)* SPPSPS  SPSP  Too weak 3 primitive patterns 4 derived patterns

PASTE 2004Jinlin Yang8 Partial Order of Patterns MultiEffect PSS MultiCause PPS EffectFirst SPS Alternating PSPS OneCause SPSS OneEffect SPPS CauseFirst PPSS Response SPPSPS Stricter

PASTE 2004Jinlin Yang9 Logical Relation of Patterns MultiEffect PSS MultiCause PPS EffectFirst SPS Alternating PSPS OneCause SPSS OneEffect SPPS CauseFirst PPSS For each combination of two events Decide if they satisfy CauseFirst, OneCause, or OneEffect Find the strictest pattern Response SPPSPS Stricter ΛΛΛ Λ

PASTE 2004Jinlin Yang10 Find Strictest Pattern Trace 1: PSPSTrace 2: PPSAll Traces CauseFirst  OneCause  OneEffect  For any two events, determine the strictest pattern satisfied by all traces

PASTE 2004Jinlin Yang11 Find Strictest Pattern Trace 1: PSPSTrace 2: PPSAll Traces CauseFirst  OneCause  OneEffect  For any two events, determine the strictest pattern satisfied by all traces

PASTE 2004Jinlin Yang12 Find Strictest Pattern Trace 1: PSPSTrace 2: PPSAll Traces CauseFirst  OneCause  OneEffect  CauseFirstΛOneEffect  MultiCause For any two events, determine the strictest pattern satisfied by all traces

PASTE 2004Jinlin Yang13 Implementation Automated testing and inference Instrumentation and properties diff by hand 900 lines of Perl code 93 different events, 100 traces, each has 222 events on average: less than an hour Program Instrumented Program Instrumentation Test Suite Execution Traces Testing Inferred Properties Candidate Property Patterns Inference

PASTE 2004Jinlin Yang14 Experiments Hypotheses –We can automatically extract interesting temporal properties. –Differences of temporal properties among multiple versions can reveal interesting things. Target programs –Tour bus simulator (8 student submissions) –OpenSSL (0.9.6, d) –Simplified producer-consumer (in the paper)

PASTE 2004Jinlin Yang15 Tour Bus Simulator Bus and each passenger is a separate thread Assignment in Fall 2003 graduate-level course (before we started this project) 8 submissions from the instructor

PASTE 2004Jinlin Yang16 Testing >cville_bus –N 2 –C 1 –T 2 Bus waiting for trip 1 Passenger 0 gets in Bus drives around Charlottesville Passenger 0 gets off Bus waiting for trip 2 Passenger 1 gets in Bus drives around Charlottesville Passenger 1 gets off Bus stops for the day N, the number of people C, the capacity of the bus T, the number of trips Executed each submission with 100 randomly generated inputs, where 20 < C ≤ 40 C+1 ≤ N ≤ 2C 1 ≤ T ≤ 10

PASTE 2004Jinlin Yang17 Results: Tour Bus Simulator Pattern7 Correct Versions 1 Faulty Version Alternating wait  drives MultiEffect drives  gets off wait  gets off wait  gets in wait  drives wait  gets in MultiCause gets in  drives CauseFirst gets in  gets off drives  gets off wait  gets off Differences reveal problems –Bus drives around before all passengers get off –Other problems

PASTE 2004Jinlin Yang18 OpenSSL Widely used implementation of the Secure Socket Layer protocol 6 versions [0.9.6, 0.9.7, 0.9.7a-d] We focus on the handshake protocol

PASTE 2004Jinlin Yang19

PASTE 2004Jinlin Yang20 Testing Manually instrumented server Modified client: advance to a different state with 5% probability Executed each version of server with 1000 randomly generated clients

PASTE 2004Jinlin Yang21 Inferred Alternating Patterns a0.9.7b0.9.7c0.9.7d SR_KEY_EXCH  SR_CERT_VRFY SW_CERT  SW_KEY_EXCH SW_SRVR_DONE  SR_CERT Documented improvement Fixed bug Race condition 7 alternating patterns same for all versions

PASTE 2004Jinlin Yang22 Partitioning Traces Correct clients –Properties are exactly same as specified –All versions agree Faulty clients without server errors generated –Found one possible vulnerability of DOS attack Segmentation faults –A documented bug fixed since 0.9.7d

PASTE 2004Jinlin Yang23 Summary of Experiments Useful in program evolution –Reveal interesting changes in OpenSSL –Identify unexpected differences Revealed bugs –Tour bus: identified faulty implementation Multiple implementations are rare, but multiple representations are common (i.e. design, model, code) –OpenSSL: partitioning traces

PASTE 2004Jinlin Yang24 Future Work/Research Questions Automating instrumentation/validation –Can we automatically identify interesting events? Identifying more useful patterns –Combine data-flow with events –More expressive, more events –Ed will talk more about this Prioritize results –How to identify interesting differences? Testing –How do the inferred properties vary with the test suite?

PASTE 2004Jinlin Yang25 Conclusion Automatically inferring temporal properties is feasible Even very simple property patterns reveal interesting program properties

PASTE 2004Jinlin Yang26 Thank You! This work is funded in part by the National Science Foundation. We thank Marty Humphrey for providing the student submissions, Chengdu Huang for helping OpenSSL, Joel Winstead for discussion and comments on earlier version of this paper, anonymous PASTE reviewers for insightful comments, anonymous CS650 students who wrote the buggy code, and our research group members and CS graduate students at the UVa for helping improve the talk.