1. Orna Kupferman Hebrew University Formal Verification -- Deciding the Undecidable

2. Is the system correct? Does the system halt? CORRECT TM ={ : T satisfies  } HALT TM ={ : T halts} Undecidable!

3. Still…., is the system correct? Testing, Simulation- based Verification 1960+ Formal Verification 1980+

4. Is the system correct? Testing, Simulation- based Verification 1960+ Formal Verification 1980+ Proof-based methods State- exploration methods

5. begin z:=0 y=0 z:= z+x y:= y-1 z:= z+x y:= y-1 end yes no P :: A specification:  x=X, y=Y≥0  P  z=XY 

6. begin z:=0 y=0 z:= z+x y:= y-1 z:= z+x y:= y-1 end yes no l 0 : x=X ∧ y=Y≥0 l 1 : z=X (Y-y) l fin : z=XY z=X (Y-y) ∧ y  0 ∧ z’= z+X ∧ y’=y-1  z’=X(Y-y’) z=X (Y-y) ∧ y=0  z=XY x=X ∧ y=Y≥0 ∧ z=0  z=X(Y-y) 1. Find cut-points 2. Find invariants 3. Prove implications. A specification:  x=X, y=Y≥0  P  z=XY 

7. Proof-based methods: - Coming up with intermediate assertions is manual. - What if we do not succeed?

8. Proof-based methods State- exploration methods Is the system correct? Testing, Simulation- based Verification 1960+ Formal Verification 1980+ Model checking

9. Proof-based methods: - Coming up with intermediate assertions is manual. - What if we do not succeed? Model checking: - Fully automatic - A counterexample is given when the system does not satisfy the specification. On going behaviors of non-terminating systems

10. Traditional view: Turing machines, accept/reject an input. 1980s: Reactive systems: interact with an environment, generate on-going behaviors. Specifications: “ z=XY ”, “ the program sorts the numbers ” ;... Specifications: “ every request is eventually granted ” ; “ there is no deadlock ”, … Operating systems, communication protocols, elevator controllers,…

11. Model checking, the idea: System  A mathematical model M Desired behavior  A formal specification  The system has the desired behavior M satisfies 

12. Model checking: req grant A mathematical model of the system: A formal specification of the desired behavior: labeled state- transition graph “ every request is followed by a grant ” “ only finitely many grants ”... temporal logic, automata on infinite words

13. Temporal logic Atomic propositions: AP={p,q,…} Boolean operators: , , ,… G (always) F (eventually) X (next) U (until) GpGp p p p p p p p p FpFp p XpXp p pUqpUq p p p p p p p p q Temporal operators:  1 =G (req  F grant)  2 =GF grant  3 = req U (  req  grant) A: for all path E: exists a path

14. Model checking, the idea: System  A mathematical model M Desired behavior  A formal specification  The system has the desired behavior M satisfies 

15. What’s the big deal? - it actually works! - Specification formalisms - Efficient algorithms - Ways to cope with huge, possibly infinite state spaces - Further applications (synthesis, control) 2007 Edmund M. Clarke, E. Allen Emerson and Joseph Sifakis For [their roles] in developing Model-Checking into a highly effective verification technology, widely adopted in the hardware and software industries. 1996 Amir Pnueli For seminal work introducing temporal logic into computing science and for outstanding contributions to program and systems verification. - it involves beautiful theoretical challenges!

16. A specification: L  (2 AP )  A state of the system:   2 AP p,q A computation of the system:   (2 AP )  p,qpq specifications  languages Specification formalisms: On-going behaviors!

17. Specification formalisms: On-going behaviors! Temporal logic: ALWAYS (req  EVENTUALLY grant) Automata on infinite words:

18. Finite words: the run ends in an accepting state L(A)=(a+b)*b (all words ending with b) Büchi automata [1962, decidability of monadic S1S] a a b b q0q0 q1q1 Infinite words: the run visits an accepting state infinitely often L(A)=(a*b)  (all words with infinitely many b’s)

19. Finite words: the run ends in an accepting state L(A)=(a+b)*b (all words ending with b) Büchi automata [1962, decidability of monadic S1S] Infinite words: the run visits an accepting state infinitely often L(A)=(a*b)  (all words with infinitely many b’s) Complementation? a a b b q0q0 q1q1

20. Finite words: the run ends in an accepting state L(A)=(a+b)*b (all words ending with b) Büchi automata [1962, decidability of monadic S1S] a b a a q0q0 q1q1 Infinite words: the run visits an accepting state infinitely often L(A)=(a*b)  (all words with infinitely many b’s) Complementation? Determinization?? Containment??? Complementation?

21. Specification formalisms: On-going behaviors! Temporal logic: ALWAYS (req  EVENTUALLY grant) Automata on infinite words: Σ=2 AP {}, {grant}, {req, grant} {req} {req}, {} {grant}, {req, grant} 1977 [Pnueli] 1986 [Vardi, Wolper] 1962 [Büchi]

22. Specification formalisms: active research 1980s: CTL, LTL, CTL* (linear vs. branching)

23. Specification formalisms: linear vs. branching idle coin coffee tea idle coin coffee tea coin Linear approach: identical systems Branching approach: different systems EX(EX coffee ∧ EX tea)

24. Specification formalisms: active research 1980s: CTL, LTL, CTL* (linear vs. branching) 1990s: regular expressions, real time 2000s: PSL (industry), ATL (multi-agent systems, games) 2010s: quantitative properties

25. ALWAYS (request  EVENTUALLY grant) … … … … … … The Boolean setting does not distinguish between the different ways in which a specification is satisfied. √ √ √ √ √ Is satisfaction really Boolean?

26. ALWAYS (request  EVENTUALLY grant) … … … … … … 0 0.6 0.95 0.4 0.5 0.3

27. Specification formalisms: active research 1980s: CTL, LTL, CTL* (linear vs. branching) 1990s: regular expressions, real time 2000s: PSL (industry), ATL (multi-agents, games) 2010s: quantitative properties Even “more undecidable”

28. A weighted finite automaton (WFA) q 0,0 q1,q1, q 2,5 q 3,4 q 4,0 a,2 b,1 c,1 a,3 c,1 a,2 b,1 a,4 b,4 c,4 b,4 c,4 Every transition has a cost. c(q 1,a,q 3 )=2, c(q 4,c,q 4 )=4

29. A weighted finite automaton (WFA) q 0,0 q1,q1, q2,5q2,5 q3,4q3,4 q 4,0 a,2 b,1 c,1 a,3 c,1 a,2 b,1 a,4 b,4 c,4 b,4 c,4 Every transition has a cost. c(q 1,a,q 3 )=2, c(q 4,c,q 4 )=4 Every state has a final cost, charged if a run ends in it.  (q 3 )=4,  (q 2 )=5. Note: There are no accepting states. It is possible to have  ( q 1 )= .

30. The cost of a word (in DWFA) q 0,0 q 1,  q 2,5 q 3,4 q 4,2 a,2 b,1 c,1 a,3 c,1 a,2 b,1 a,4 b,4 c,4 b,4 c,4 In a deterministic WFA (DWFA), the cost of a word w is the cost of the single run of A on w.

31. The cost of a word (in a DWFA) q 0,0 q 1,  q 2,5 q 3,4 q4,2q4,2 a,2 b,1 c,1 a,3 c,1 a,2 b,1 a,4 b,4 c,4 b,4 c,4 In a deterministic WFA (DWFA), the cost of a word w is the cost of the single run of A on w. For w=babc, cost(A,w)=1+3+1+4+2=11

32. The Cost of a Word (WFA) In a non-deterministic WFA (NWFA), the cost of a word w is the cost of the cheapest run of A on w. q 0,0 q 1,0 q 2,5 q 3,4 q 4,6 a,2 b,4 b,1 c,1 a,3 c,1 a,2 b,1 a,4 b,4 c,4 b,2 c,3

33. The cost of a word (in a WFA) In a non-deterministic WFA (NWFA), the cost of a word w is the cost of the cheapest run of A on w. For w=bbc, cost(A,w)= min(9,12)=9 q 0,0 q1,0q1,0 q 2,5 q 3,4 q4,6q4,6 a,2 b,4 b,1 c,1 a,3 c,1 a,2 b,1 a,4 b,4 c,4 b,2 c,3 Upper path: 4+2+3+0=9 Lower path: 1+1+4+6=12 A weighted regular language: L:  * → R

34. The exciting world of weighted automata Given two WFAs A and A′, we say that A is cheaper than A’ if for every word w  Σ *, we have that cost(A,w) ≤ cost(A′,w). q 0,1 q 1,4 q 2,5 a,2 b,4 b,1 c,1 a,3 c,1 q 0,0 q 1,0 q 2,2 a,4 b,4 c,2 b,1 c,3 a,3 c,1 a,3 b,1a,3 b,6 ?≤?≤

35. How to check whether A ≤ A′ ? q 0,1 q 1,4 q 2,5 a,2 b,4 b,1 c,1 a,3 c,1 q 0,0 q 1,0 q 2,2 a,4 b,4 c,2 b,1 c,3 a,3 c,1 a,3 b,1a,3 b,6 ?≤?≤ Related problem: q0q0 q1q1 q2q2 a c a,c q0q0 q1q1 q2q2 a,b b,c a,c a,b   b Containment between NFAs

36. How to check whether A  A′ ? q0q0 q1q1 q2q2 a c a,c q0q0 q1q1 q2q2 a,b b,c a,c a,b   b A  A’ iff A  comp(A’) =  How to complement A′? Determinization!

37. q1q1 q0q0 q2q2 q3q3 a,1c,1 b,2 a,1 b,1 d,1 Weighted Automata A cost(abb) = cost(abc) = L(A) = {,, } min { 1+2+2, 1+1+1 } = 3 1+2+1 = 4

38. q1q1 q0q0 q2q2 q3q3 a,1c,1 b,2 a,1 b,1 d,1 Weighted Automata A -- There is no equivalent deterministic automaton.  (q 0,ab i )=  (q 0,ab j )=q cost(ab i )=i+1 cost(ab j )=j+1 cost(ab i c)=2i+2 cost(ab j c)=2j+2 What ’ s the cost of a c-transition from q?

39. q1q1 q0q0 q2q2 q3q3 a,1c,1 b,2 a,1 b,1 d,1 Weighted Automata A Open problem: Given a WFA, is there an equivalent DWFA? Undecidable? Undecidable #1: is there a word w such that L(w) ≥ 1 Undecidable #2: weighted containment (A ≤ A′)

40. Open problem: Given a WFA, is there an equivalent DWFA? Undecidable? Undecidable #1: is there a word w such that L(w) ≥ 1 Undecidable #2: weighted containment (A ≤ A′) Interesting connection #1: determinazability ↔ decidability Automata on infinite alphabets… Interesting connection #2: competitive ratio of online algorithms ↔ approximation required for determinization

41. Back to the Boolean setting… Model-checking algorithms Linear time: the automata-theoretic approach S satisfies  L(S)  L(  ) L(S)  comp(L(  )) =  Branching time: bottom up AGFEXXp AGFq Complementation … L(S)  L(  ) = 

42. The state-explosion problem Huge state spaces: - n Boolean variables --> 2 n configurations - parallelism - hierarchy, modularity - data in the model - software Coping with the state-explosion problem - symbolic methods - abstraction - compositionality The main challenge in bringing formal methods to practice!

43. Symbolic methods input x1,x2 output y1,y2 init(y1)=true init(y2)=false next(y1)=(y1  x1)  y2 next(y2)=(  y1  x2)  (y2  x1) Description of the hardware: O(number of variables). State space: exponential in the number of variables Symbolic methods: work with the description rather than with the state space. VERILOG -- hardware description language (HDL)

44. How to work with the description? 1.Use Binary Decision Diagrams in order to represent sets of states and transitions. BDD: a compact way to represent Boolean functions. f S : a BDD representing a set S of states x 1,x 2, …,x n fS:x1x2fS:x1x2 S={01_ _ _ _ _ _} x1x1 x2x2 T F 0 0 1 1 x1x2x1x2

45. How to work with the description? 1. Use Binary Decision Diagrams in order to represent sets of states and transitions. f S : a BDD representing a set of states (a formula over X) f R : a BDD representing the transition relation (a formula over X and X’) pre(S): the set of predecessors of S (a formula over X) The BDD f pre(S) can be obtained from f S and f R

46. BDD-based calculations of states satisfying EFp p p  pre(p) p  pre(p)  pre(pre((p)) fixed-point … All operations are done symbolically! (reachability to a state satisfying p)

47. An example to an interesting problem: Enumerative: linear. Symbolic: quadratic, O(n log n), linear … Input: a graph G given by a symbolic description of the edges, and a set T of states. Output: is there a path that visits T infinitely often? Bad cycle detection: T

48. How to work with the description? 1. Use Binary Decision Diagrams in order to represent sets of states and transitions. 2. Reduce model checking to the satisfiability problem and use SAT-solvers. The system has a bad behavior: the propositional formula that describes the values of the variables along a bad behavior is satisfiable.

49. Related research: - Symbolic algorithms. - BDDs: variable ordering, extensions, fixed-point based logics (  -calculus). - SAT-based methods: bounding the length of bad behaviors, SAT-solvers and their performance on formulas generated in bounded model checking, SMT (satisfiability modulo theories).

50. Abstraction: x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM x=0x=2x=4x=798x=800 x=353x=1000 ? BOOM

51. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 p1p2p3p4 08001000

52. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 x=0 800 < x < 1000 0 < x ≤ 800 x ≥ 1000 BOOM over- approximation

53. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 x=0 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 over- approximation No bad behavior in the abstraction No bad behavior in the system

54. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 x=0 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 over- approximation bad behavior in the abstraction bad behavior in the system

55. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 x=0 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 and refinement P5: x is even

56. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 P5: x is even x=0 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 and refinement 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 x is even x is odd

57. Predicate abstraction x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 P5: x is even x=0 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 and refinement 0 < x ≤ 800 x ≥ 1000 BOOM 800 < x < 1000 x is even x is odd

58. Predicate abstraction and refinement x := 0; while x < 1000 do x:= (x mod 800) +2 if x = 353 then x := 1000 BOOM p1: x=0 p2: 0 < x ≤ 800 p3: 800 < x < 1000 p4: x ≥ 1000 P5: x is even x=0 0 < x ≤ 800 x is even x ≥ 1000 BOOM 800 < x < 1000 x is odd 0 < x ≤ 800 x is odd BOOM is unreachable in the abstraction  unreachable in the system.

59. Related research: - which predicates? - how to refine? - over/under approximate

60. What’s the big deal? - it actually works! - Specification formalisms - Efficient algorithms - Ways to cope with huge, possibly infinite state spaces - Further applications (synthesis, control) 2007 Edmund M. Clarke, E. Allen Emerson and Joseph Sifakis For [their roles] in developing Model-Checking into a highly effective verification technology, widely adopted in the hardware and software industries. 1996 Amir Pnueli For seminal work introducing temporal logic into computing science and for outstanding contributions to program and systems verification. - it involves beautiful theoretical challenges!

61. Formal Verification + Beautiful landmark ideas - Ad-hoc implementation details

62. Formal Verification + Beautiful landmark ideas - Ad-hoc implementation details Fruitful collaboration with the industry!

63. Formal Verification + Beautiful landmark ideas - Ad-hoc implementation details Fruitful collaboration with the industry! Intractable, yet very practical.