1. How Model-Checking Can Help Model Exploration Marsha Chechik Dept of Computer Science University of Toronto Joint work with Arie Gurfinkel, Benet Devereux

2. Overview of Automated Verification Answer + Counterexample Answer + Counterexample SW/HW artifact SW/HW artifact Correctness properties Correctness properties Temporal logic Temporal logic Model of System Model of System Model Extraction Model Extraction Translation Checker Engine Checker Engine Correct?

3. Correctness properties: CTL  propositional temporal logic  branching-time logic, allowing explicit quantification over possible futures  Syntax:  True and False are CTL formulas  if p and q are CTL formulae, then so are:  p, p  q, p  q  EX p - p is true in some next states  EF p - along some path, p is true in some future state;  E[p U q] - along some path, p holds until q holds;  EG p - along some path, p holds in every state Universal quantification: AX p, AF p, A[p U q], AG p  ECTL: EX, EF, EU, EG, disjunction, negation atomic  ACTL: AX, AF, AU, AG, conjunction, negation atomic

4. Ü Conventional state machines  M =  S is a (finite) set of states  A is a (finite) set of propositional variables  s 0 is a unique initial state ( s 0  S )  I: S  2 A is a labelling function that maps each state to the set of propositional variables that hold in it  R  S  S is a (total) transition relation p qrp qr  p q r pqrpqr s0s0 s2s2 s1s1 Models:Kripke Structures

5. Counterexamples and Witnesses Ü Goal:  explain why the answer is as given  counterexample:why  (s 0 ) = F i.e., why the property fails to hold  witness: why  (s 0 ) = T i.e., why the property holds Ü Counterexamples and mathematical proofs:  to disprove that  holds on all elements of S, produce a single element s  S s.t.  holds on s.  counterexamples are restricted to universally-quantified formulas  counterexamples are paths (trees) from initial state illustrating the failure of property

6. Examples Ü Witnesses  EG p pqrpqr  p q r p qrp qr s0s0 s2s2 s1s1 pqrpqr s0s0 p qrp qr s2s2 Ü Counterexamples  AG p pqrpqr s0s0  p q r s1s1  EX p  EX q pqrpqr s0s0 p qrp qr s2s2  p q r s1s1  AX p  AX q pqrpqr s0s0 p qrp qr s2s2  p q r s1s1

7. Witnesses/Counterexamples Ü Counterexamples (SMV)  AG( Running  AF Idle ) is false  long path leading to state Running  followed by a counterexample for AF  EX p is false  nothing given even though it is false!  AG ( Running  EF Idle )  no feedback given when the result is either true or false Ü Witnesses (SMV)  similar -- just negate the properties!  A witness to  is a counterexample to 

8. So, what do we want? Ü Witnesses/counterexamples  Good  short -- contain only as much information as necessary  correspond to the model  Bad  not available in all cases  often hard to navigate to an “interesting part” Ü Proofs:  Good:  complete (available for all temporal properties) all information is here  Bad:  too verbose  not particularly intuitive Where is the model? Want proof-like counter-examples [TACAS’03]

9. Example: Cruise Control System (CCS) For keeping an automobile running at a certain speed  Driver accelerates to desired speed and then presses a button on steering wheel (Button = bCruise)  System maintains car speed until  The driver presses the brake pedal (Break)  The driver presses the accelerator (Accel)  The driver turns the cruise control off (Button=bOff)  The driver turns the ignition off (Ignition)  The car’s speed becomes uncontrollable (Toofast)  The system can be reactivated by pressing a “resume” button (Button=bResume) Ü The controlled variable is Throttle

10. Model-Checking : Example

11. Proof View: Example

12. Proof-Like Counter- Example (part 1) Property: AG((CC=Cruise) -> Brake)

13. Proof-Like Counter-Example (part 2) Property: AG((CC=Cruise) -> Brake)

14. Proof-Like Counter- Example (part 3) Property: AG((CC=Cruise) -> Brake)

15. Mixed Quantifier Formula: Example

16. Mixed Quantifier Formula: Proof View Property: EF (AX Brake)

17. Mixed Quantifier Property: Example Property: EF (AX Brake)

18. The Framework Model- checker engine Model- checker engine KEGVis SMV model (with fairness) Temporal logic property (CTL) solution + proof-like witness partial witness/ counterexample Navigation/ exploration strategies

19. Exploration Ü Witness/Counterexample exploration:  dealing with large witnesses Ü How?  how to fast-forward to “interesting” parts  Specify a starting condition, e.g., navigate by a formula Example:AG(Running  AF Idle) not interested in how to get to Running  Specify stopping condition  Pick direction (forward/backward)  how to limit information given to user so that interesting cases are easier to find

20. Navigation Ü Witness/Counterexample Navigation:  making the most interesting choice Ü Sources of choices:  explicit (disjunction)  which part of property to consider  Example: (EF p)  (EG q)  implicit (via EX)  which state to pick as a witness?  Example: EX p pp pp s0s0 s1s1 s3s3 s2s2

21. Bounds in Property-Based Navigation Ü Depth of exploration  Example: witness for AF   feasible: (AF p)(s 0 ) = (AF 1 p)(s 0 )  unfeasible: when bound  |S| (witness is as big as the model!) pqpq pqpq pqpq s0s0 s2s2 s1s1  Can get partial knowledge using depth of exploration  Example: (AF p)(s 0 ), set depth to 3 p p s0s0 s1s1 s4s4 s2s2 s3s3  what if depth is smaller than expected?

22. State-based navigation Ü Pick successor in which  (some propositional formula) holds  EX p pqpq p  q pqpq s0s0 s1s1 s3s3 s2s2 Ü Pick a state using number of successors  least -- linear  greatest -- branching ts Ü Attempt to maintain largest common prefix  Example: (EX p )  (EX q )  try to pick next state where p and q hold  greedy approximation

23.  shortest witness not necessarily the most interesting!  Example: (EF Good  EF Error)(s 0 ) = (E[T U Good ]) (s 0 )  (E[T U Error]) (s 0 ) = (E [T U 7 Good ]) (s 0 )  (E [T U 3 Error]) (s 0 )  picking the shortest counter-example  automatically  manually based on size or additional information Strategy: A combination of navigation and exploration to ensure that some user-specified goal is met Example: shortest counterexample  traditionally,  counterexample generators always do shortest (greedy) counterexample

24. Strategies Ü Choices:  anything based on paths, states, etc.  including depth of expansion, history, longest common prefix, shortest counter-example...  decision procedure always greedy (and thus non-optimal)  if a strategy was not followed, does not mean that it could not be followed! Ü Examples:  User has complete control.  whenever there is a choice, always ask user  Always attempt to go through Idle state.  Always choose state s over t ; otherwise, ask user

25. Part II Finding Properties

26. AG(p  q) AG(p  AX q) Model-Checking Ü Typically used for verification Answer + Counter-example Answer + Counter-example Checker Engine Checker Engine Correct? Ü Where do the properties come from? Ü What to do when they do not hold? So, goal is not just verification but discovery of properties that the system should have! Temporal logic property Temporal logic property Model of System (state-based) Model of System (state-based)

27. Query Checking [Chan, CAV’00] Ü Goal: speed-up design understanding  discover properties not known a priori p qrp qr  p q r pqrpqr s0s0 s2s2 s1s1 (p  q)  r is strongest solution to AG ? x, p  q is strongest solution to AG ? x {p,q} Ü Temporal logic query  temporal logic formula with placeholders (unknowns)  e.g., AG ? x, AG (p  ? x )  evaluates to strongest propositional formula that makes query true. Ü Some applications  provide partial explanation when property holds  e.g. instead of AG (a  b), ask AG ? x {a, b}  answer a  b is stronger!  provide diagnostic information when property fails  e.g. if AG (req  AF ack) fails - ask AG (req  AF ? x )

28. Types of queries Ü number of placeholders  e.g., AG ? x, AG (? x  EX ? y ) Ü positive vs negative  positive queries - placeholder under even # of negations  e.g. AG ? x  then look for strongest solutions  negative queries - placeholder under odd # of negations  e.g. AG (? x  p) = AG (  ? x  p)  then look for weakest solutions  mixed queries - neither positive nor negative Ü number of maximally strong solutions  “valid” queries - one strongest solution  arbitrary queries - several strongest solutions p qrp qr  p q r pqrpqr s0s0 s2s2 s1s1 two solutions to (EX ? x )(s 0 ): p  q  r and  p  q  r

29. Deciding TLQ Problem Ü Related work on TLQ  Original definition [Chan, CAV’00]  algorithm for “valid” (single strongest solution) subset of CTL  Extended by [Bruns & Godefroid, LICS’01]  arbitrary temporal logic formulae  via extended alternating automata Ü Our work  [Gurfinkel, Chechik, Devereux, FSE’02, TSE’03]  reduction to multi-valued model-checking  implementation that deals with arbitrary temporal logic formulae  … with arbitrary number of “unknowns”  answers include witnesses

30. Example: Cruise Control System (CCS) For keeping an automobile running at a certain speed  Driver accelerates to desired speed and then presses a button on steering wheel (Button = bCruise)  System maintains car speed until  The driver presses the brake pedal (Break)  The driver presses the accelerator (Accel)  The driver turns the cruise control off (Button=bOff)  The driver turns the ignition off (Ignition)  The car’s speed becomes uncontrollable (Toofast)  The system can be reactivated by pressing a “resume” button (Button=bResume) Ü The controlled variable is Throttle

31. Specification of CCS Ü Specified using SCR method  State changes – in response to events (changes in environment) @T(a) WHEN b abab a @T(a) WHEN b =  a  b  a’  Input: monitored variables  Output:controlled variables  System state: modeclasses  Sets of states (modes) that partition the state space  The system is in exactly one mode of each modeclass at any point

32. Mode TransitionTable for CCS Cruise@T(Button=bResume) WHEN Ignition AND Running AND NOT(Toofast) AND NOT(Brake) AND NOT(Accel) Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT(Break) AND NOT(Accel) Inactive@F(Running) WHEN Ignition Off@F(Ignition)Override @T(Brake) OR @T(Accel) OR @T(Button=bOff) Inactive@T(Toofast) OR @F(Running) Off@F(Ignition)Cruise @T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel) Off@F(Ignition)Inactive @T(Ignition)Off New ModeEventOld Mode Initial Mode: Off WHEN NOT(Ignition)

33. Event Table for Throttle Initial: Throttle=tOff tAccelThrottle’= @T(Inmode) when (Speed=slow) Cruise @T(Speed=slow) Cruise EventsModes @T(Speed=ok) @T(Inmode) when (Speed=ok) @T(Speed=fast) @T(Inmode) when (Speed=fast) @F(Inmode) tMaintaintDeceltOff

34. Applications of TLQ “find all possible values of Throttle EF ( (CC=Cruise)  ? x {Throttle}) when the system is in mode Cruise” “what modes can follow Off” EF(CC=Off  EX? x {CC}) “which pairs of modes can EF(? x {CC}  EX? y {CC}) follow each other” Ü Invariant Discovery “find the strongest invariant” AG ? x “find invariant of mode Inactive AG ((CC=Inactive)  ? x {Ignition,Running}) w.r.t. Ignition and Running” “find invariants of all modes w.r.t AG (? x {CC}  ? y {Ignition,Running}) Ignition and Running” Ü Reachability analysis “are all modes of modeclass CC reachable?” EF(CC=Cruise) EF ? x {CC}

35. Query-Checking Witnesses Ü A witness  a subtree explaining why ECTL formula holds  produced automatically by model-checker p qrp qr  p q r pqrpqr s0s0 s2s2 s1s1 Ü Example:  query: ( EX ? x { p })(s 0 )  solutions:? x = p ? x =  p s0s0 s2s2 s1s1 ? x = p ? x =  p  witness: Same strategies as for exploring model-checking witnesses available

36. TLQ Applications: Testing Ü Current approach (Gargantini, Heitmeyer [FSE’99])  branch coverage  for each mode in mode transition table, test each event at least once  for each mode, test every no-change at least once Off@F(Ignition)Cruise @T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel) Off@F(Ignition)Inactive @T(Ignition)Off New ModeEventOld Mode  e.g. for mode Off:  need @T(Ignition) and no-change  form CTL properties  EF((CC=Off)  EX(CC=Inactive))  EF((CC=Off)  EX(CC=Off))  witness produced by model-checker is the test case!

37. Query-Checking for Testing Ü our approach:  witness to single query  EF((CC=Off)  EX ? x {CC}) Off@F(Ignition)Cruise @T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel) Off@F(Ignition)Inactive @T(Ignition)Off New ModeEventOld Mode

38. Testing Transitions from Mode Off

39. Generated Witness Property: EF((CC=Off)  EX ? x {CC})

40. Query Checking for Testing Ü An even better approach:  coverage of the entire mode transition table  EF(? x {CC}  EX ? y {CC})

41. Testing All Transitions from Table

42. Witness

43. Implementation Ü Our framework is named XChek [CAV’02]  multi-valued model-checking engine  TLQSolver [CAV’03]  query checker built on top of it  KEGVis [TACAS’03,FME’03]  tool for visualization and exploration of count. ex. and witnesses Ü Input:  XML models (based on GXL derived language)  SMV-like input language (synchronous product of simple state machines)  + fairness condition (e.g., p is true infinitely often)  CTL property or query Ü For more info  To obtain a prototype version  send e-mail to xchek@cs.toronto.edu  http://www.cs.toronto.edu/~chechik/publications.html

44. Questions? Comments? Concerns? Suggestions? THANKS FOR YOUR ATTENTION!

45. Overview of Multi-Valued Model- Checking Answer + Counter-example Answer + Counter-example How Correct? SW/HW artifact SW/HW artifact Correctness properties Correctness properties MV-logic Checker Engine Checker Engine Temporal logic Temporal logic MV-Model of System MV-Model of System Model Extraction Model Extraction Translation

46. Multi-Valued Algebras Ü Use additional ‘truth values’ to represent levels of contradiction, uncertainty or anything else  e.g., True, False, Maybe, Likely, etc. Ü Can be defined on a (finite) distributive lattice of truth values  with True at the top and False at the bottom  using lattice meet as conjunction…  and lattice join as disjunction Ü Negation is defined to preserve involution, i.e.  A = A Ü Then get preservation of associativity, idempotency, distributivity, and De Morgan’s laws Ü These algebras are called quasi-boolean Ü (optional) may also add a refinement operator

47. T F T F M (Maybe) Classical logic Representing uncertainty Uses: reasoning about abstraction and partial systems Representing disagreement and uncertainty Multi-Valued Algebras: Examples TT TM TF MT MM FT FMMF FF

48. Multi-valued state machines: Xkripke structures Ü Extension of conventional state machines (Kripke structures)  variables take any value from the logic  transitions between states take any value from the logic  False transitions are not shown (by convention) Ü Example: p= TT q= FT r= TT p= FF q= TT r= TT p= TT q= TF r= TT s0s0 s2s2 s1s1 TT FT TF TT FF TFFT

49. Partial information Ü Algebra:  use three-valued algebra (Kleene)  intermediate value represents incomplete information or uncertainty T F M  compact representation for all possible refinements of this model  if a property is True/False on the partial model, it is True/False on a refined one  initial theory developed by Bruns & Godefroid, CAV’99 p= T q= F r= T p= M q= M r= F p= T q= M r= T s0s0 s2s2 s1s1 T T M M T

50. Reasoning about Abstraction  a way to overcome the state-space explosion problem in classical model-checking  collapses sets of concrete states into a single abstract state  … thus indicating that any differences between the concrete states within a single abstracts state are ignored  Goal: state-wise preservation  if a formula evaluates to True (False) in an abstract state, it evaluates to True (False) in the corresponding concrete state T F M p qrp qr  p q r pq rpq r s0s0 s2s2 s1s1 s 0,1 s2s2 p= M q= M r= T p= T q= T r= F M T M 

51. Complexity  Running time of the model checker is O(|S|  |  |  k), where  |S| - size of state space  |  | - the size of the XCTL formula  k - time to compute EX Same as classical!!!!!

52. Solving Query-Checking  arbitrary temporal logic formulae  not necessarily “valid”  positive, negative, mixed queries  any number of placeholders  output includes “reason” why the answer is as given  extend language of queries e.g. EF ? x {p, q, r}, where p and q are not true simultaneously

53. Some formalism  A - set of prop. variables  e.g., {p}  set of prop. formulas over A forms lattice ordered by  true pp p false {p,  p, true} {  p, true} {p, true} {true} {} {false, p,  p, true}  {}  {p,  p}  {  p}  {p}  {true}  {false}   B ={c |  b  B s.t. b  c}   {p,  p} = {p,  p, true}  X is an upset if  X=X  {p,  p} not an upset, {p,  p, true} is  result - upset lattice, ordered by set inclusion  each upset can be represented by a set of minimal elements   {false} can represent {p,  p, true, false} So, if X is a solution to a query, all elements of  X are solutions

54. Reasoning with Colors Ü Given a non-temporal formula with colors, how to evaluate in a state?  e.g.:  = (p  q  red)  (  p  q  green)  (  p   q  yellow)  (p   q  blue) in state where p  q holds s0s0 s1s1 pqpq pqpq s2s2  {}  {p,  p}  {  p}  {p}  {true}  {false}  answer: red Ü Given a temporal formula, how does it evaluate?  (EX  ) (s) =  t  succ(s )  (t)  e.g., evaluate ( EX  ) (s 0 ): (p  q  red)  (  p  q  green)  (  p   q  yellow)  (p   q  blue) = red  green = yellow

55. Encoding TLQ Ü Encoding non-temporal formula with ? x  if p  q holds in s 0, then ? x {p,q}(s 0 ) =  (p  q)  {}  {p,  p}  {  p}  {p}  {true}  {false} s0s0 s1s1 pqpq  p q s2s2  in general, ? x {p} = (p  p)  (  p   (  p)) = (p  green)  (  p  red)  each color is strongest possible solution  always get exactly one color per state! Ü Temporal formula  (EX ? x )(s) =  t  succ(s ) ? x (t)  e.g., ( EX ? x {p}) (s 0 ) = red  green = yellow =  {p,  p}

56. Queries with Multiple Placeholders Ü Solution  L i - lattice of propositional formulas over i th placeholder  solution - from upset lattice over L 1  …  L n Ü Example:  ? x  ( EX ? x  AX ? y )  solution in (B  L 2 )  ((C  L 2 )  (L 1  D)) = (B  L 2 )  (C  D)  i.e., {(x,y) | x  B  (x  C  y  D)} p qrp qr  p q r pqrpqr s0s0 s2s2 s1s1 Ü Another example:  ? x { p, q }  EX ? y { p, q } in state s 0  [[? x ]](s 0 ) =  {p   q}  [[EX? y ]](s 0 ) =  {p  q,  p  q}  [[? x ]](s 0 ) =  {p   q}   {false}  [[EX? y ]](s 0 ) =  {false}   {p  q,  p  q}  solution:   {p   q}   {p  q,  p  q}  answers: {(p   q, p  q), (p   q,  p  q)}

57. Negation Ü All occurrences of placeholder are either negative or positive  Example: AG  ? x  solve for AG ? x, choose  from it  AG  and thus AG  (   ) hold  so   is in solution-set for AG  ? x Ü A given placeholder (? x ) appears in both negative and positive forms  replace each positive occurrence with ? x+  replace each negative occurrence with ? x-  solve  the set of all solutions to ? x is intersection of solutions to ? x+ and ? x-

58. Multi-Valued Model-Checking Ü Multi-Valued CTL (XCTL)  introduce new constants that interact with true and false  e.g., true  red = red  allow to include these constants into CTL formulas Ü Multi-Valued Models  include colors in transition relations and/or in states Ü XChek: symbolic model-checker  receives  a lattice of colors, describing how to compose them  multi-valued model + XCTL property  fairness requirements (e.g., assume p holds infinitely often)  returns  appropriate color, counter-example/witness

59. Running time Ü Naïve query-checking algorithm O(|S|  |  |  2 2 n )  n - number of atomic propositions of interest  | S | - size of state space  |  | - the size of the CTL formula Ü Query-checking for valid queries O(|S|  |  |  2 n ) Ü Running time of our model checker O(|S|  |  |  dd)  dd - time to compute EX symbolically  depends on:  size of lattice (number of join-irreducible elements in it)  cost of performing union and intersections

60. Running time of Query Checker Ü Preliminaries  CT( n ) - complexity of performing an operation on terminal nodes of ADD  n -- number of propositions restricting the placeholder  complexity of performing any operation on entire ADD is linear in CT( n ) and exponential in | A | Theorem: Complexity of solving a query  with one placeholder is linear in |S|, |  |, CT( n ) and exponential in |A| Theorem: Complexity of solving a query  with multiple placeholders (? 1 (n 1 )…? k (n k )) is same as with single placeholder, where CT(  i=1 k n i ) but… how many terminal nodes are there?

61. Running time (Cont’d) CT( n ) is quadratic in the number of strongest solutions to  ! How many solutions are there in a query? Ü Queries about states  Example: AG (  q  AX ? x { p })  Number of solutions: O(2 n < | S |)  Query-checking - same complexity class as model-checking Ü Queries about paths  Example: EG (? x )  Number of solutions in worst case [hornus02]: O(2 2 n < 2 |S| )  Query-checking can be infeasible even for small problems!  … but not always!  Ex: EF EG ? x {CC} in a Cruise Control System (later in the talk)

62. Improvements to Running Time Ü Query-checking and model-checking  query built of some query-checking and some model- checking parts  AG(? x  AF p): CTL model-checking of AF p, result used for query- checking  Theorem: if placeholder occurs in scope of V temporal quantifiers is V  Q + (|  |  V )  M  Q - worst case complexity of query checking a formula with one temporal quantifier  M - worst case complexity of query checking a formula with one temporal quantifier

63. Improvements (Cont’d) Ü Heuristics:  constructing decision diagram for ? x is $$ …  but can be avoided in most queries  Transition relation is boolean so cost of performing conjunction and disjunction on terminal nodes is O(1) and does not depend on CT.