Test generation using model-checking Thierry Jéron Irisa/Inria Rennes, France Adapted from works with C. Jard, B. Jeannet,

Slides:



Advertisements
Similar presentations
Hybrid BDD and All-SAT Method for Model Checking Orna Grumberg Joint work with Assaf Schuster and Avi Yadgar Technion – Israel Institute of Technology.
Advertisements

Black Box Checking Book: Chapter 9 Model Checking Finite state description of a system B. LTL formula. Translate into an automaton P. Check whether L(B)
Auto-Generation of Test Cases for Infinite States Reactive Systems Based on Symbolic Execution and Formula Rewriting Donghuo Chen School of Computer Science.
Compilation 2011 Static Analysis Johnni Winther Michael I. Schwartzbach Aarhus University.
An Introduction to the Model Verifier verds Wenhui Zhang September 15 th, 2010.
Abstraction and Modular Reasoning for the Verification of Software Corina Pasareanu NASA Ames Research Center.
Friday, April 17, 2015 Requirements Specifications Interaction Analysis Bill Mitchell Software and Systems Research Labs Motorola UK Research Labs Bill.
SOFTWARE TESTING. INTRODUCTION  Software Testing is the process of executing a program or system with the intent of finding errors.  It involves any.
Timed Automata.
Testing Concurrent/Distributed Systems Review of Final CEN 5076 Class 14 – 12/05.
Rigorous Software Development CSCI-GA Instructor: Thomas Wies Spring 2012 Lecture 13.
Efficient Reachability Analysis for Verification of Asynchronous Systems Nishant Sinha.
1 Temporal Claims A temporal claim is defined in Promela by the syntax: never { … body … } never is a keyword, like proctype. The body is the same as for.
A survey of techniques for precise program slicing Komondoor V. Raghavan Indian Institute of Science, Bangalore.
VerTeCS Verification models and techniques applied to the Testing and Control of reactive Systems Thierry Jéron IRISA/INRIA Rennes, France
ISBN Chapter 3 Describing Syntax and Semantics.
An Integration of Program Analysis and Automated Theorem Proving Bill J. Ellis & Andrew Ireland School of Mathematical & Computer Sciences Heriot-Watt.
Automated Model-Based Testing of Hybrid Systems Michiel van Osch PROSE January 25,
Introduction to Computability Theory
Convertibility Verification and Converter Synthesis: Two Faces of the Same Coin Jie-Hong Jiang EE249 Discussion 11/21/2002 Passerone et al., ICCAD ’ 02.
1 Basic abstract interpretation theory. 2 The general idea §a semantics l any definition style, from a denotational definition to a detailed interpreter.
Model-based Testing of Hybrid Systems Michiel van Osch IPA Spring Days on Testing 19 April – 21 April 2006.
Model checking dynamic states in GROOVE Arend Rensink Formal Methods and Tools University of Twente.
Abstractions. Outline Informal intuition Why do we need abstraction? What is an abstraction and what is not an abstraction A framework for abstractions.
Synergy: A New Algorithm for Property Checking
1 Carnegie Mellon UniversitySPINFlavio Lerda SPIN An explicit state model checker.
1 Jan Tretmans Embedded Systems Institute Eindhoven Radboud University Nijmegen Model-Based Testing.
Towards Automatic Generation of Parameterized Test Cases from Abstractions Jens R. Calamé Natalia Ioustinova Jaco van de Pol Centrum voor Wiskunde en Informatica,
Copyright © 2006 The McGraw-Hill Companies, Inc. Programming Languages 2nd edition Tucker and Noonan Chapter 18 Program Correctness To treat programming.
Describing Syntax and Semantics
1 Formal Engineering of Reliable Software LASER 2004 school Tutorial, Lecture1 Natasha Sharygina Carnegie Mellon University.
Program Analysis Mooly Sagiv Tel Aviv University Sunday Scrieber 8 Monday Schrieber.
1 3rd of July 2009 CEA LIST Symbolic execution based model checking of open systems with unbounded variables Nicolas RAPIN CEA LIST.
System/Software Testing
Regular Model Checking Ahmed Bouajjani,Benget Jonsson, Marcus Nillson and Tayssir Touili Moran Ben Tulila
ECE 720T5 Winter 2014 Cyber-Physical Systems Rodolfo Pellizzoni.
Using Mathematica for modeling, simulation and property checking of hardware systems Ghiath AL SAMMANE VDS group : Verification & Modeling of Digital systems.
1 Automatic Refinement and Vacuity Detection for Symbolic Trajectory Evaluation Orna Grumberg Technion Haifa, Israel Joint work with Rachel Tzoref.
Introduction Algorithms and Conventions The design and analysis of algorithms is the core subject matter of Computer Science. Given a problem, we want.
CIS 842: Specification and Verification of Reactive Systems Lecture Specifications: Sequencing Properties Copyright , Matt Dwyer, John Hatcliff,
1 Bisimulations as a Technique for State Space Reductions.
Conformance Test Suites, Extensionally Arend Rensink University of Twente Dutch Workshop on Formal Testing Techniques University of Twente 13 September.
Institut Mines-Télécom Symbolic Passive Testing - Application to an industrial case study (Diamonds project) Pramila Mouttappa, Stephane Maag and Ana Cavalli.
Natallia Kokash (Accepted for PACO’2011) ACG, 31/05/ Input-output conformance testing for channel-based connectors 1.
1 Black-box conformance testing for real-time systems Stavros Tripakis VERIMAG Joint work with Moez Krichen.
1 CSEP590 – Model Checking and Automated Verification Lecture outline for August 6, 2003.
SDS Foil no 1 V&V&S Verification, Validation and Synthesis: doing away with defects Verification, Validation and Synthesis: doing away with defects.
CIS 540 Principles of Embedded Computation Spring Instructor: Rajeev Alur
CTL Model-checking for Systems with Unspecified Components Summer-1384 Hajar Niamehr Neda Noroozi.
Lecture 5 1 CSP tools for verification of Sec Prot Overview of the lecture The Casper interface Refinement checking and FDR Model checking Theorem proving.
Towards Interoperability Test Generation of Time Dependent Protocols: a Case Study Zhiliang Wang, Jianping Wu, Xia Yin Department of Computer Science Tsinghua.
CIS 540 Principles of Embedded Computation Spring Instructor: Rajeev Alur
Software Quality Assurance and Testing Fazal Rehman Shamil.
Model Checking Lecture 1. Model checking, narrowly interpreted: Decision procedures for checking if a given Kripke structure is a model for a given formula.
CMPSC 16 Problem Solving with Computers I Spring 2014 Instructor: Tevfik Bultan Lecture 4: Introduction to C: Control Flow.
An Automated Test Generation Process from UML Models to TTCN-3 Jens R. Calamé.
Model Checking Lecture 1: Specification Tom Henzinger.
Symbolic Model Checking of Software Nishant Sinha with Edmund Clarke, Flavio Lerda, Michael Theobald Carnegie Mellon University.
CIS 540 Principles of Embedded Computation Spring Instructor: Rajeev Alur
An Automated Test Generation Process from UML Models to TTCN-3
Testing Tutorial 7.
Hybrid BDD and All-SAT Method for Model Checking
Software Testing.
Program Synthesis is a Game
Programming Languages 2nd edition Tucker and Noonan
An explicit state model checker
Translating Linear Temporal Logic into Büchi Automata
Test Generation with Abstraction and Concretization
Programming Languages 2nd edition Tucker and Noonan
Presentation transcript:

Test generation using model-checking Thierry Jéron Irisa/Inria Rennes, France Adapted from works with C. Jard, B. Jeannet, J. C. Fernandez, H. Marchand, P. Morel, V. Rusu, V. Tschaen, E. Zinovieva et al.

2 Model-checking / Conformance Testing S: Specification P: Safety Property e.g. given by observer A : P S ² P iff S does not exhibit (finite) bad behaviors wrt P i.e. L(S) Å L( A : P, Violate ) = ; I: Black box (unknown except its interface) S: Specification I conf. / S iff I does not exhibit (finite) bad external behaviors wrt S Rmk: safety property on ext. behavior I conf. S, STr (I) Å STr(CT(S), Fail) = ; Violate Exaustive check on S £ A : P Finite partially controllable experiment on Exec(I || CT(S)) Similar problems, different means Fail

3 Some links between Model-checking for Conformance Testing Test selection using model-checking : –S deter., controllable, P reachability: TC ' counter-exple of S ² : P [Engels et al. 97, Gargantini et al.99] –Extension to coverage [Hong et al.02, Blom et al.04] –Non-controllable case is a bit more complex (this talk) Checking properties on the implementation –Black-box checking [Peled]: learn I by experiment, model-check I ² P –Model-checking and testing safety properties: build tests from S and P that check whether I ioco S, I ² P and S ² P.

4 Outline Conformance Testing with ioco Test selection using model-checking Selection by a test purpose for (finite) ioLTS Selection by a test purpose for (infinite) ioSTS Selection by a (negative) safety observer Conclusion

5 Conformance Testing Problem: check if a black-box implementation conforms to a given specification by experimentations using test cases –the source code of the implementation is unknown –only its interface can be controlled/observed by the environment Implementation Conforms to ? Tester Specification ? !

6 ioco Testing theory [Tretmans 96] Specification: known ioLTS S Implementation: unknown ioLTS I Conformance –Visible behaviors : suspension traces STr(.) – ioco: partial inclusion of STr(I) in STr(S) Test cases : ioLTS TC to build –Execution I || TC (synchronisation on common actions) ! verdict. –Properties: TS= {TC} is Sound: verdict(I||TC) = fail ) : (I ioco S) Exhaustive: : (I ioco S) ) 9 TC 2 TS, verdict(I||TC)= fail Test case selection problem: Renounce to exhaustiveness in practice, select a finite TS likely to discover non-conformances

7 The ioLTS model ?A ?B ! Z ! Y  ! X  M = (Q,q 0, , ! )  =  ? [  ! [   det(  (S))=(2 Q,  *(q 0 ),  vis, ! vis )  vis =  ? [  !  Deterministic automaton ?A ?B !Z !Y !X ?B   (M) = (Q,q 0,  , !  )    =  ? [  !  [    !     ! [ !  !  ! [ {q © !  | q 2 quiescent (M)} Suspension automaton ?A ?B ! Z ! Y  ! X    det Quiescence is observable STr(S), Tr(  (S)) Visible behaviors

8 Conformance relation I2 ioco S Unspecified Input Allowed : I1 ioco S Unspecified Output/quiescence Not allowed I ioco S, 8  2 STr(S), Out(  (I) after  ) µ Out(  S  after  ) ?A ?B !Z !Y !X ?B  det(  (S)) ?A ?B !Z !Y !X ?B  ?A ?B !Z !Y !X ?B  !X ?A

9 Non-conformance Observer ?A ?B !Z !Y !X ?B  det(  (S)) I ioco S, STr(I) Å Tr(A : ioco S,Fail) = ; Complete det(  (S)) by unspecified outputs in  !  ! non-conformance observer A « : ioco S » Tr(A : ioco S,Fail) = STr(S).  !   \ STr(S) Fail !otw The canonical tester of S for ioco is CT(S), mirror ( A : I ioco S ) I ioco S, I || CT(S) 9 Fail !A !B ?Z ?Y ?X !B  ?otw Fail mirror Rmk: If I was known, could be model-checked

10 Non-deterministic test generation [Tretmans 96] ?A ?B !Z !Y !X ?B  !otw Fail A : I ioco S A test case TC is (the mirror of) any finite « controllable » unfolding of CT(S) ?A !Z !Y !X ?B  !otw Fail ?A !Z !Y !X  !otw Fail Pass TS = {TC that can be built this way} is sound exhaustive TS infinite ) selection needed

11 Test selection with (positive) Test Purpose Test purpose = description of some interesting visible behaviours of S to be tested Test purpose model : deterministic and complete ioLTS + Accepting states TP = (Q tp, q 0tp,  vis, ! tp, Accept) Tr(TP) =  vis * ( ( complete) Tr (TP, Accept) µ  vis * Test selection goal : build a sound test cases that is both: –A positive observer of STr(S) Å Tr(TP,Accept) ! Pass verdict –A negative observer of STr(S).  !   \ STr(S) ! Fail verdict !z * Accept !x * TP *

12 Principles of test selection TP CTG S  (S) TC det(  (S)) SP = A : ioco S £ TP Tr(SP) = Tr( A : ioco S ) Å Tr(TP) Tr(SP,Fail) = Tr( A : ioco S, Fail) Å Tr(TP) Tr(SP,Accept, : Fail) = STr(S) Å Tr(TP,Accept) STr(S) Tr(TP) =  vis * Tr(TP,Accept) Tr(CTG, Pass) = Tr(SP,Accept, : Fail) Tr(CTG, Fail) µ Tr(SP,Fail) Controllable Test Case A : ioco S Tr( A : ioco S ) =pref(STr(S).  !  ) Tr( A : ioco S, Fail) = STr(S).  !   \ STr(S) Focus on Accept i..e restrict to states coreachable to Accept

13 Synchronous product !z * Accept !x * TP  ?A ?B !Z !Y !X ?B  A : ioco S ?C !Y ?C !Y SP = A : ioco S x TP * !oth Fail 3 (Accept, Fail) ?B ?A ?B !Y !X !Z  !Z  ?B ?C !Y ?C !Y  Fail !oth  Accept = : Fail £ Accept TP Tr(SP) = pref(STr(S).  !  ) Tr(SP,Fail SP ) = STr(S).  !   \ STr(S) Tr(SP,Accept SP ) = STr(S) Å Tr(TP,Accept) ! Pass verdict ! Fail verdict

14 Selection of the Complete Test Graph CTG Goal: remove traces that cannot lead to Accept 1. Computation of coreach(Accept) i.e. states on traces to Accept pref(Tr(SP,Accept)) = Tr(SP, coreach(Accept)) Accept ?B ?A ?B !Y !X !Z  coreach(Accept) Init !Z  ?B ?C !Y ?C !Y  Fail !oth

15 Building the Complete Test Graph (2) 2. Verdict assignment Pass verdict ! Accepted traces: Tr(SP,Accept) = STr(S) Å Tr(TP,Accept) Inconc verdict ! Refused traces correct traces after which Accept is unreachable Optimality: Cut controlable ones and detect early : [pref(Tr(SP,Accept)).  !   \ Tr(SP, Fail)] \ pref(Tr(SP,Accept))] Fail verdict ! Non-conformant traces pref(Tr(SP,Accept)).  !  Å Tr(SP,Fail) ) preserves soundness Pass reach(Init) Inconc ?B ?A ?B !Y !X ! Z    coreach(Accept) Init Fail !oth

16 Building the Complete Test Graph (3) Pass Inconc !B !A !B ?Y ?X ?Z  coreach(Accept) Init 3. Mirror image ! Tester point of view CTG(S,TP) Properties soundness: only non-conformance is detected (limit) exhaustiveness: every non-conformance may be detected : (I ioco S) ) 9 TP, verdict(CTG(S,TP) || I) = fail NB: controllable test cases : no choice between output and ouput/input Fail ? oth

17 Controllable Test cases Pass reach(Init) Inconc !A !B ?Y ?X ?Z  4. Solving controllability conflicts e.g. DFS from Pass states coreach(Accept) Init Fail ?oth

18 Implementation in TGV Test Purpose bcg, IF Simulation API of S Umlaut Object Géode Caesar open IF open Specification UMLSDL Lotos IF TGV Test case (bcg, aut, TTCN, ) Internal actions in TP ) improves selection Lazzy construction of intermediate IOLTS ) limits state space explosion TP CTG S  (S) TC det(  (S)) A : ioco S £ TP A : ioco S TP CTG S  (S £ TP) det(  (S £ TP)) S £ TP TC Integrated in several tool chains (CADP, Objectgéode, Agedis, IF) Experimented on numerous case studies

19 Test selection for ioSTS Motivations: –Use higher level specification model with control and data –Avoid state space explosion due to enumeration of data values Specification model S: ioSTS = i/o automata extended with data ioSTS semantics [[ S ]] = infinite states ioLTS Implementation model: unknown (infinite state) ioLTS Conformance testing theory: ioco Test selection from S and Test Purpose TP : ioSTS observer –Syntactical constructions of A : ioco S, A : ioco S £ TP, CTG(S,TP) guided by coreachability analysis –coreachability undecidable ) over-approximated analysis –Produces uninstanciated test programs p ¸ x ? a(p) y := y + p

20 ioSTS model p=y-x ∧ p > 2 !nok(p) Init Ry Rx Cmp End ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x ∧ p ¸ 2 !ok(p) S = (V S,  S, , T S ) [a(p) : G(v S,p); v S :=A(v S,p)] 2 T S TP = (V S [ V TP,  TP, , T TP ) Observer of actions and variables of S true * Wait Acc * ⌝ (p=2 ∧ x≥3) !ok(p) Sink * p=2 ∧ x≥3 !ok(p) Accept:=true true !nok(p) [a(p) : G(v S,v TP,p); v TP := A(v S, v TP,p)] 2 T TP NB: Locations = values of an additional variable l (but useful in figures)

21 Simplifying asumptions on ioSTS (for this talk) 1. ioSTS are supposed to be deterministic Determinisation of ioSTS into ioSTS is not always possible ) restrictions needed No internal loop Finite lookahead ! Determinisation procedure for this sub-class : postpone assignments 2. Quiescence is not treated here augment model with universal quantification in guards G1A1G1A1 G 2 a(p) A 2 G 1 Æ G 2 ± A 1 a(p) A 2 ± A 1 «  closure » G 1 a(p a ) A 1 G b b(p b ) A b G 2 a(p a ) A 2 G c c(p c ) A c G 1 [ G 2 a(p a ) v a := p a G 1 Æ G b ± A 1 b(p b ) A b ± A 1 G 2 Æ G c ± A 2 c(p c ) A c ± A 2 « Subset construction » G 1 \ G 2  ; Idea: after a finite delay, the effect of actions must be known: diagnosability

22 ioLTS semantics of ioSTS Runs(S): v 0 !   (p) v 1 !   (p) v 2 … 2 Q 0. (  vis,Q) * Traces(S) :   (p).   (p) … 2  vis * S = (V S,  S, , T S ) p=y-x ∧ p > 2 !nok(p) Init Ry Rx Cmp End ?start ?a(p) x:=p !end p=y-x ∧ p ¸ 2 !ok(p) true t = [a(p) : G(v S,p); v’ S :=A(v S,p)] is firable in state v iff 9 , G(v,  )=true v – t ! v’=A(v,  ) ?a(p) y:=p … (Init,v x,v y ) … … (Rx, v x,v y ) … (End,-,-) (Cmp,3,6) … (Ry,3,v y ) … ?start !end ?a(- 1 ) ?a(0) ?a(+ 1 ) ?a(3) ?a(6) !ok(3) (Rx, 3,4) (Cmp,3,0) !nok(-3) (Rx,3,0) [[ S ]] = S = (Q,Q 0, , ! ) Q = D (V S ) Q 0 = D (V S ) Å  S  =  £ D (P S ) ?a(0)

23 Principle of syntactical test selection TP: ioSTS CTG: ioSTS S: ioSTS SP = A : ioco S £ TP Syntactical Synchronous Product: runs( SP ) = runs( A ) Å runs( TP ) runs( SP,Accept) = runs( A ) Å runs( TP,Accept) runs( SP,Fail) = runs( A, Fail) STr( SP ) = STr( A ) STr( SP,Accept)=p  vis (runs( SP, Accept)) STr( SP,Fail)= STr( A, Fail) runs( S ) STr( S )=p  vis (runs( S )) runs( TP ) runs( TP,Accept) Sub-ioSTS of SP focussed on Accept Tr( CTG, Pass) = Tr( SP, Accept) ioSTS syntactical operations deterministic A : ioco S runs( A ), runs( A, Fail) STr( A )= pref( STr( S ).  ! ) STr( A, Fail) = STr( S ).  ! \ STr( S )

24 Non-conformance observer A : ioco S p=y-x ∧ p < 2 !nok(p) Init Ry Rx Cmp End ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x ∧ p ¸ 2 !ok(p) true Æ Verd=none Fail !oth STr( A, Fail) = STr( S ).  !  \STr( S ) ) soundness : ( Ç i G i (v,p)) !a(p) Verd:= Fail G 1 (v,p) !a(p) v:= A 1 (v,p) G n (v,p) !a(p) v:= A n (v,p) Fail 1. Add new variable Verd with initial value none i.e.  Æ Verd = none 2. 8 ouput !a, 8 t carrying !a :

25 Syntactical product Syntactical product A : ioco S £ TP A : ioco S = (V A,  A, , T A ) A : ioco S £ TP = (V S [ V TP,  S Å  TP, , T S £ TP ) G 1 (v S,p) a(p) v’ S := A 1 (v S,p) G 2 (v S,v TP,p) a(p) v’ TP := A 2 (v S, v TP,p) G 1 (v S,p) Æ G 2 (v S, v TP, p) a(p) := TP = (V S [ V TP,  TP, , T TP )

26 Syntactical product: example Verd=none Idle Ry Rx Cmp End ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x Æ p ¸ 2 !ok(p) p=y-x Æ p < 2 !nok(p) A : ioco S Fail ! oth Wait Acc * ⌝ (p=2 Æ x≥3) !ok(p) Sink * p=2 Æ x≥3 !ok(p) Accept:=true * TP !nok(p) true Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x Æ p ¸ 2 Æ p=2 Æ x≥3 !ok(p) Accept:= true Rx Acc Rx Sink p=y-x Æ p ¸ 2 Æ ⌝ (p=2 Æ x≥3) !ok(p) SP, A : ioco S £ TP p=y-x Æ p < 2 !nok(p) Fail ! oth Verd:=Fail ¶ (Accept, Fail) Verd=none

27 Syntactical Test Selection (1) Pass: Tr( SP,Accept) ! Observer of Tr(SP, Accept) G(v,p) a(p) v:= A(v,p) G(v,p) Æ Verd = none a(p) v:= A(v,p) Verd :=if A Accept then Pass else Verd A : ioco S £ TP CTG 1. Assignment of Pass verdicts

28 Syntactical Test Selection (2) 2. Selection and assignment of Inconc verdicts coreach(Accept) not computable ) compute over-approximations: coreach  ¶ coreach(Accept) 8 assignment A, pre  (A) (coreach  ) ¶ pre(A) ( coreach  ) Idea: pre  (A) (coreach  ) = Nec. Cond. to go in coreach  : pre  (A) (coreach  ) = Suf. Cond. to go outside coreach   µ outisde coreach(Accept) coreach coreach  G(v,p) Æ pre  (A)(coreach  ) a(p) v’:=A(v,p) G(v,p) Æ : pre  (A)(coreach  ) a(p) v’:=A(v,p) G(v,p) a(p) v’:=A(v,p)

29 Syntactical test selection (3): guard strengthening Rule for inputs of S: keep conditions leading to coreach , cut other ones (controllable): Rule for outputs of S keep all conditions (uncontrollable), those leading outside coreach  produce Inconc: Inconc coreach coreach  G(v,p) Æ pre  (A) ! a(p) v’:=A(v,p) G(v,p) Æ : pre  (A) ! a(p) v’:=A(v,p) Verd:=Inconc coreach coreach  G(v,p) Æ pre  (A) ? a(p) v’:=A(v,p) G(v,p) Æ : pre  (A) ? a(p) v’:=A(v,p)

30 Test selection: example 1 st over-approximation : control Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x ∧ p ¸ 2 Æ p=2 Æ x≥3 !ok(p) Accept := true p=y-x Æ p < 2 !nok(p) Rx Acc Rx Sink p=y-x ∧ p ¸ 2 Æ ⌝ (p=2 Æ x≥3) !ok(p) SP = A : ioco S £ TP Fail ! oth Verd:= Fail Verd= none Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start ?a(p) y:=p ?a(p) x:=p !end Verd:= Inconc p=y-x ∧ p ¸ 2 Æ p=2 Æ x≥3 !ok(p) Verd:=Pass p=y-x Æ p < 2 !nok(p) Verd:=Inconc Rx Acc p=y-x p ¸ 2 Æ ⌝ (p=2 Æ x≥3) !ok(p) Verd:= Inconc CTG 1 (S, TP) Fail ! oth Verd:=Fail Rx Sink Verd=none Abstraction on control: only the location is taken into account in coreach  ⊥ ⊥ ⊤⊤ ⊤⊤

31 Test selection: example 2 nd approximation computed by NBAC (polyhedra) Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x ∧ p ¸ 2 Æ p=2 Æ x≥3 !ok(p) Accept :=true p=y-x Æ p < 2 !nok(p) Rx Acc Rx Sink p=y-x Æ p ¸ 2 Æ ⌝ (p=2 Æ x≥3) !ok(p) SP = A : ioco S x TP x≥3 y-x=2 Æ x≥3 ⊤ ⊤ ⊤ ⊥ ⊥ Fail ! oth Verd:=Fail pre  (x:=p)(x ¸ 3) = p ¸ 3 Verd= none Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start p = x+2 ?a(p) y:=p p ¸ 3 ?a(p) x:=p !end Verd:= Inconc p=y-x Æ p ¸ 2 Æ p=2 Æ x≥3 !ok(p) Verd:=Pass p=y-x Æ p < 2 !nok(p) Verd:= Inconc Rx Acc Rx Sink p=y-x Æ p ¸ 2 Æ ⌝ (p=2 Æ x≥3) !ok(p) Verd:= Inconc CTG 2 (S, TP) Fail ! oth Verd:=Fail Verd=none coreach 

32 Simplification: over-approximation reach  of reach(  ) Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start p = x+2 ?a(p) y:=p p ¸ 3 ?a(p) x:=p !end Verd:= Inconc p=y-x ∧ p ¸ 2 Æ p=2 Æ x≥3 !ok(p) Verd:=Pass p=y-x Æ p > 2 !nok(p) Verd:= Inconc Rx Acc - Sink p=y-x Æ p ¸ 2 Æ ⌝ (p=2 Æ x≥3) !ok(p) Verd:= Inconc CTG 2 (S, TP) x≥3 y-x=2 Æ x≥3 ⊤ ⊤ y-x=2 Æ x≥3 ⊥ ⊤ Simplify guards according to reach  (false ) cut) NB: semantics is unchanged Fail ! oth Verd:=Fail Verd= none Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start p = x+2 ?a(p) y:=p p ¸ 3 ?a(p) x:=p !end Verd:= Inconc p=2 !ok(p) Verd:=Pass Rx Acc CTG 2 (S, TP) Fail ?oth Verd:=Fail false !nok(p) Verd:= Inconc - Sink false !ok(p) Verd:= Inconc Verd= none

33 Consequences of over-approximation on test cases reach(  ∩ coreach    Accept reach(  ) ?oth Inconc Fail reach(  ∩ coreach   ? ! ? ? For two abstractions  1 and  2 ( e.g.  1 : control vs  2 : polyhedra) pre  (A) (coreach  ) ¶ pre  (A) (coreach  ) ) Tr( CTG 1 ) ¶ Tr( CTG 2 ) In CTG 1 Inconc detected later More fail verdicts (all sound) Limit cases: exact analysis: best guiding to Accept no analysis: no guiding to Accept

34 Test execution Ouputs: [ ! a(p) : G(v,p); v:=A(v,p)] : v is known, choose  s.t. G(v,  ),by constraint solving send !a(  ), assign v:=A(v,  ) Inputs: [ ? a(p) : G(v,p); v:=A(v,p)]: v is known, receive ? a(  ) evaluate G(v,  ) if true, assign v:= A(v,  ) (input complete) !start. !a(4). !a(6). ?ok(2) : Pass !start. ?end : Inconc !start. !a(5). !a(7). ?ok(3) : Fail !start. !a(6). !a(8). ? nok(2) : Fail Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start p = x+2 ?a(p) y:=p p ¸ 3 ?a(p) x:=p !end Verd:= Inconc p=2 !ok(p) Verd:=Pass Rx Acc CTG 2 (S, TP) Fail ! oth Verd:=Fail Verd= none

35 The STG Tool Test case selection Symbolic operations Over-approximated analysis by abstract interpretation (NBAC) Test case execution Compilation C++/Java Constraint solving (Omega) Experimentations : CEPS IUT C++/Java C++/ Java Test case Test purposeSpecification Test case selection Compilation dotty Omega verdict NBAC Test case

36 Model-checking and Testing safety properties P properties S specification I implementation S ² P ? y/n/u I ² P ? I conf S ? n/u Development process

37 Model-checking a safety property Idle Ry Rx Cmp End ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x Æ p ¸ 2 !ok(p) p=y-x Æ p < 2 !nok(p) S Wait Violate * p = -10 !nok(p) Violate := true * A: PA: P Model-checking S ² P ? reduces to reachability in S £ A : P (undecidable) Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x Æ p ¸ 2 !ok(p) p=y-x Æ p < 2 Æ p  -10 !nok(p) p=y-x Æ p < 2 Æ p  -10 !nok(p) Rx Violate S ² : P : ?start.?a(10).?a(0).!nok(-10) ! (Rx, Violate) With any abstraction S  ² : P ? is Yes but S  ² : P ; S ² : P Result of S ² P ? could be Unknown

38 Test selection from a safety observer Wait Violate * p = -10 !nok(p) Violate := true * A: PA: P Idle Ry Rx Cmp End ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x Æ p ¸ 2 !ok(p) p=y-x Æ p < 2 !nok(p) A : ioco S !oth Fail ! oth Verd:=Fail S ² : PI ² : PS ² : PI ² : P : I ioco S I ² : P : I ioco S !start.!a(10). !a(0).?nok(-10) Fail A : ioco S £ A : P Idle Wait Ry Wait Rx Wait Cmp Wait End Wait ?start ?a(p) y:=p ?a(p) x:=p !end p=y-x Æ p ¸ 2 !ok(p) p=y-x Æ p < 2) Æ p  -10 !nok(p) p = -10 !nok(p) Violate :=true Verd:=Fail p=y-x Æ p < 2 Æ p= -10 !nok(p) Violate:=true Violate Fail Violate : (p=y-x Æ p < 2) Æ p =-10 !nok(p) Violate :=true Verd:=Fail ! oth Verd:=Fail

39 Conclusion Unified framework for Ioco-based Test selection –For (Positive) Test Purpose and (Negative) Safety Observers –For finite ioLTS and infinite ioSTS –Using verification: coreachability analysis, over-approximations Complementarity of testing and model-checking Research work needed for, e.g. Theories and algorithms for other models of reactive systems e.g. with time, data, stack, probabilities …and combinations Coverage : measures, selection Links with structural testing techniques ….

40 Bibliography J. Tretmans. Test Generation with Inputs, Outputs and Repetitive Quiescence. Software - Concepts and Tools, Vol. 17(3), pp Springer-Verlag, C. Jard, T. Jéron, TGV: theory, principles and algorithms, A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems, Software Tools for Technology Transfer (STTT), 6, October B. Jeannet, T. Jéron, V. Rusu, E. Zinovieva, Symbolic Test Selection based on Approximate Analysis, in TACAS'05, LNCS 3440, Edinburgh (Scottland), April Vlad Rusu, Hervé Marchand, Thierry Jéron, Automatic Verification and Conformance Testing for Validating Safety Properties of Reactive Systems, in Formal Methods 2005 (FM05), July Available at