1. System Design Research Laboratory Specification-based Testing with Linear Temporal Logic Li Tan Oleg Sokolsky Insup Lee University of Pennsylvania

2. System Design Research Laboratory Outline Motivation and the framework Property-coverage criterion Derive coverage criterion from system requirement in temporal logic. Test generation using model checkers. Path-truncating strategy: from the infinite traces to finite tests. Black-box testing. White-box testing. Conclusion

3. System Design Research Laboratory Motivations Formal methods play a major role in designing and verifying safety-critical reactive systems. Model-based design to accelerate the software design and facilitate formal verification. Temporal logics used to specify the system’s behavior requirements. Testing is still an important link in software development cycle. Some of its advantages, Testing can directly check the implementation. Testing will not suffer from the state-explosion problems faced by model checking. Question: how to produce high-quality test suite from temporal logic-based specification?

4. System Design Research Laboratory Preliminary: LTL and 9 LTL Linear Temporal Logic (LTL) define the property held by the executions of a system, Path formula: a property held by an execution. P ::=a | : a| P Æ P | P Ç P | X P | P U P | P R P a is an atomic proposition. X is the “next time” operator. U and R are the “until” and “release” operators. P 1 U P 2 : P 1 holds till P 2 holds P 1 R P 2 : P 2 holds unless an occurrence of P 1 releases it (shorthands) G P (Always hold) for false R P, and F P ( Eventually hold) for true U P A (for all the paths), and E (exists a path) are the path quantifiers. LTL: A P. P holds on all the executions. LTL is an expressive and popular logic. LTL is supported by many state-of-the-art model checkers, including SPIN and (Nu)SMV 9 LTL: E P. P holds on some execution.

5. System Design Research Laboratory LTL and 9 LTL by Example non-critical section 0 critical section 0 try 0 :=1 try 0: :=1 try 1 =1? turn=1? turn  1? try 0 :=0 turn=0? try 1  1? grant 0 :=1 try 0 :=0 grant 0 :=0 non-critical section 1 critical section 1 try 1 :=1 try 1: :=1 try 0 =1? turn=0? turn  0? try 1 :=0 turn=1? try 0  1? grant 1 :=1 try 1 :=0 Grant 1 :=0 System requirement: (LTL)  mux =AG((try 1 =1)->F(grant 1 =1)) Every request to access the critical section 1 is finally granted. Extended FSM model T of Dekkar’s algorithm for mutual exclusion problem P0P0 P1P1 turn:=1 turn:=0 111’ 2 3 4 5 6 8 2’ 3’ 4’ 5’ 6’ 8’ 77’

6. System Design Research Laboratory Challenges and Problems LTL specifiies the property that holds on all the paths. To prove  mux on T, all the executions need to be checked. Executions are infinite in length. Test has to be done in finite time. Test suite has to be finite in length and in size Challenge: which execution shall be covered in test and how many steps we shall test?

7. System Design Research Laboratory Intuition and Informal Approach Only those “non-trivial” executions need to be checked. A request is finally granted: If a request to access the critical section 1 has been made at least once, then we shall check if it is finally granted. 9 LTL formula: f 1 =E ( F (try 1 =1) Æ  mux ) Access is by invitation only. If P 1 never access the critical section 1 after a time, then we shall check that no request has ever been made since then. 9 LTL formula: f 2 =E ( FG (grant 1  1) Æ  mux ) Question: can we formally derive these 9 LTL characterizing test coverage directly from the LTL formula?

8. System Design Research Laboratory Property-coverage criterion: a formal approach Obtain the trapping formula by syntactically manipulating (mutating) LTL formula. Given : a LTL formula . Output: a set of 9 LTL formulae G(  ) G(  )={E(  Æ :  [a ! ð (a)] | a is an atomic proposition inside  } The polarity of a is positive if a is nested in the even number of negations in . ð (a)=false if and only if the polarity of a is positive. Property-coverage criterion for a test suite TS: each formula in G(  ) is satisfied by at least one test in TS

9. System Design Research Laboratory An example: deriving trapping formulae from  mux  mux =AG((try 1 =1) ! F(grant 1 =1)) f’ 1 !!! f’ 2 !!! AG((try 1 =1) ! F(grant 1 =1)) E(  mux Æ : G((try 1 =1) ! false)) E(  mux Æ F(try 1 =1)) E(  mux Æ: G(true ! F(grant 1 =1))) E(  mux Æ FG(grant 1 =1)) A(G((try 1 =1) ! F(grant 1 =1))

10. System Design Research Laboratory Property-coverage testing and its implications Nonvacuity in model checking: avoid some part of formula go unchecked even when the system satisfies a LTL property. Example: A(a->b) holds trivially on any system a doesn’t occur. A subformula  affects  on the system T iff T ²  but 9 f. T ²  [  -> f]. T satisfies  nonvacuously on T if every subformula of  affects  on T. [Theorem 1] T satisfies  nonvacuously iff T satisfies  and T passes a property-coverage test suite. “Tests alone can show the present of an error but hardly its absence.” Property-coverage testing ensures that every piece of requirements is checked.

11. System Design Research Laboratory Test Generation using Model Checking  is a counter-example for T and Af if  is an execution of T and  ² : f. Model checks like SMV and SPIN produces an counterexample if T doesn’t satisfies Af. Goal of generating tests for M and  : test generation: find test cases as the executions of the model M that satisfy G(  )={Ef 1, Ef 2, , Ef n } Model Checking : Ef i on M! No. The counterexample  is an execution satisfy Ef i Yes. M doesn’t satisfies  nonvacuously. Check M and .

12. System Design Research Laboratory Are we done yet? Model M  Model Checker : E f i Yes M doesn’t satisfy  nonvacuously Stop Check M and  Implementation Counterexample  No Test case t G(  )= {E f 0,  E f n } Property- Coverage Criteria What is the missing link Could be infinitely longHas to be finite

13. System Design Research Laboratory From Infinite Trace to Finite Test A sample counterexample for M ² : E ( F (try 1 =1) Æ  mux ). {1,1’} ({2,1’} {7,1’}{7,2’}{8,2’}{8,7’}{1,7’}{1,8’}{2,8’})  try 0 :=0 grant 0 :=0 try 0 :=1 grant 0 :=1 try 0 :=1 try 1 :=1 grant 1 =1turn=1 try 1 :=0 grant 1 :=0 turn:=0 Lasso-shaped Trace If M ² : , there is always a lasso-shaped counterexample for M and . Counterexamples produced by LTL model checkers are lasso-shaped. When shall we stop?

14. System Design Research Laboratory Path-Truncating Strategy Given: a lasso-shaped test case  (  )  Question: Can we test the “loop”  for the finite time to know if the implementation passes  (  ) . White-box testing Yes, test till the same state is encountered twice at the same place on  Black-box testing: the only known factor is the maxium number of states n Yes, test till we can infer that the same state is encountered twice at the same place on  [Theorem 2] For a black-box system T with at most n states and a lasso-shaped test t=  ¢  , n is the least number such that T passes t if and only T passes  ¢ (   ) (n)

15. System Design Research Laboratory Conclusion We proposed a new paradigm to test Linear Temporal Logic property on the implementation The property-coverage criterion is proposed to select test cases by the LTL property. Coverage criterion is derived from the LTL property automatically. Tests are generated using model checkers. Path-truncating strategy is proposed to obtain finite test cases from infinite traces. G(  )= {E f 0,  E f n } Model Checker Model M  : E f i Yes M doesn’t satisfy  nonvacuously Counterexample  Stop Check M and f No ImplementationTest case t Property- Coverage Criterion Path-Truncating Strategy Test settings