Property-Based Test Generation Li Tan, Oleg Sokolsky, and Insup Lee University of Pennsylvania

Temporal Property Translator Test Harness LTL formulae  1, …,  n Criteria not being covered Simulation-based randomized test generator Quasi linear (Lasso- shape) proof structure Test trace generator Traces {  1,…,  n } Traces Testing result Interface Definition Model Checker Specification Model (CHARON) Informal System Specification Environ. ModelingSystem Modeling Coverage Criteria Behavior specification LTL formulae  1, …,  n Hardware Specification/limitation Implementation Test result Feature conflict detection The Overview of Our Approach

Goal: Using model-checking technique to make test generation more efficient, flexible, and centered on the system-specific properties (features). Step I. Preparing specifications Properties (feature specification) as linear temporal logic formula (optional) System specification system as CHARON (for hybrid systems) and EFSM (for discrete systems) Step II. Test generation using model checkers. (For hybrid systems) Simulation-based test generation with the assistance of predicate abstraction reachability analysis. (For Discrete system) (Option A) Using the proof structures of evidence-ready model checkers. (Option B) Reducing the test generation for LTL formula to safety check (For temporal specification only) Functional test. Generating non-trivial test traces for temporal specification (feature specification) Detecting conflicting in temporal specification. Step III: Realizing test harness. Temporal (feature) Spec. + Model (optional) Test suite (Finite set of finite traces) Model-checking based Test generator

From Property and Model to Test suite: Property-based test generation I. From infinite length to finite: Synthesizing test suites for 9 LTL property

A infinite Lasso-shaped test suite can be checked adequately by finite steps if the implementation is bounded. Turn=1, : c1, : c2 Turn=1, c1, : c2 Turn=2, c1, : c2 Turn=2, c1, c2 Turn=1, c1, c2 A quasi-linear proof skeleton + Estimating the number of relevant implementation states using slicing A finite test suite

Test Generation using Model Checkers Option A: Modifying model checkers and retaining proofs. Option B: Using the idea of reducing LTL model checking to reachability analysis [A. Biere etc], but enhancing the observer to retain proof SMV model Linear Temporal Logic Specification + SMV model + Extended Observer Model Repetition information Extracted Proof Generated test trace

II. From infinite numbers of traces to finite: selecting interesting traces System properties are required to be held on all the paths, we will select only nontrivial paths, whose characteristics are caught by ELTL formula systematically deriving from the properties. LTL  => ELTL formulae a2e(  )={  (  ’ ! ð (  ’))|  ’ Á  } F = G(req -> F(cancel Ç response)) F Æ ( : G(req ! false)) = F Æ F(req) Test the case that there is request F Æ ( : G(true ! F(cancel Ç response))) = F Æ FG( : (cancel Ç response)) Test the case that no cancel or reponses occurs after time t, (hence should not a request occur). F Æ ( : G(req ! F(false Ç response))) = F Æ F(req Æ G( : response)) Test the case that no response follows a request (hence a cancel must be placed) F Æ ( : G(req ! F(cancel Ç false))) = F Æ F(req Æ G( : cancel)) Test the case that no cancel follows a request (hence a reponse must be placed)

From only Property to Test suite: Functional test generation So, what if only behavior (feature) specification is available …… LTL formulae  Nontrivial ELTL formulae Derived from   =a2e(  )  0 2  Æ 2   1 2  Æ 2   n 2  Æ 2  ……. Buchi automaton B 0 Buchi automaton B 1 Buchi automaton B n A trace satisfies  0 A trace satisfies  1 A trace satisfies  n Check nonemptiness

System Modeling CHARON (Model) Flatten hybrid model Implementation Test Suite Predicate set Bad set Reachability Checker Yes w/ Trace Simulation /refinement NO w/ more predicates YES No Testing Hybrid system: simulation-based test generator with predicate-abstract reachability analysis Concretize Coverage Criteria

An implementation of simulation-based test generator a. CHARON simulator with test generator b. Progress report of test generator c. Visual display of generated test traces.

Realizing Test Harness Charon model for CARA Charon model For patient Closed Charon Model for CARA Simulation-based test generator Coverage criteriae Test trace Variable back_EMF Value Time …… Test harness as I/O Interface CARA simulator /model-generated code Standalone executable program Test Result

Conclusion 1. Applying model-checking technique to traditional domain of test generation is appealing. 1. Test generation is centralized on system-specific properties 2. State-of-art model checkers may be adapted as general purpose test generator (and think properties as programs ). 3. Techniques in model checking may help find interesting test traces and provide new angle to view and think test generation. 2. Property-based test generation requires integrated efforts. 1. Test generation  witness generation. 1. Proof is necessary to generate partial test suite and perform optimization. 2. Proof is also needed to extend the notion of “testable” properties. 2. Model-based code generation may help build test harness.