1 E. Yahav School of Computer Science Tel-Aviv University Typestate Verification: Abstraction Techniques and Complexity Results J. Field, D. Goyal, G. Ramalingam IBM T.J. Watson Research Center

2 Motivation Component a library with cleanly encapsulated state Client a program that uses the library

3 Lightweight Specification  "correct usage" rules a client must follow  "call open() before read()" Certification does the client program satisfy the lightweight specification? Approach Component a library with cleanly encapsulated state Client a program that uses the library Tailor certification procedure to the specification being verified Canvas Project (Component Annotation, Verification and Stuff)

4 Typestate Checking [Strom and Yemini 1986] Statically verify that programs satisfy certain kinds of temporal safety properties  A file should be opened before it can be read  A file should not be read after it is closed  … Many recent approaches to verification based on typestate checking Interaction of aliasing and typestate checking not well understood

5 Goals How to increase precision of typestate checking for programs with aliasing Specialized abstractions  Exploit nature of property being verified  Exploit nature of programs being verified Complexity bounds  Relate difficulty of verification to the nature of the property/programs being verified

6 Specifications as Finite Automata Automaton defines set of valid operation sequences that can be performed on an object The set of valid sequences is prefix-closed  all transitions from error state are to itself Can also use regular expressions  implicit prefix-closure (in this talk)  read*;close not- closed close d erro r read closeread close read

7 Precise Typestate Checking and Aliasing Precise modulo common assumption that all paths in the program are executable No aliasing O(|S||V||P|) Two or more levels of pointers w/o recursive data structures PSPACE-hard Recursive data structures Undecidable Shallow aliasing (1-level pointers) ?

8 Typestate Checking + Shallow Aliasing x := new File(); y := new File(); z := y; if (?) { y.close(); z := x; } z.read();

9 Common Approach: Two-Phase Analysis Break certification into two phases  separate points-to analysis  apply typestate checking Separation of analyses can lead to imprecision Client Program Lightweight Specification not-closed close d error read closeread close read Points-to analysis Typestate checking

10 Loss of Precision in Two-phased Approach x := new File(); y := new File(); z := y; if (?) { y.close(); z := x; } z.read(); file 1 created file 2 closed file 2 may be closed read may be erroneous file 2 created z may point to file 1 or file 2 (independent attributes analysis)

11 Is Precise but Efficient Typestate Checking Possible? Not all finite state properties are equally hard to verify! read*;closeopen + ;read PolynomialPSPACE- complete

12 Plan Property Classes  Omission-Closed Properties  Repeatable-Enabling Sequence Properties Program Classes  In-degree Bounded Aliasing Summary

13 Omission-closed Properties Every subsequence of a valid sequence is also valid  read*;close  read;read;read;read;close There exists a finite set of forbidden subsequences s.t. a sequence is invalid iff it contains some forbidden subsequence as a subsequence  close;read  close;close Verifiable in polynomial time!

14 Abstracting Aliasing and Typestate information PredicateMeaning there exists an object that all variables in A point to, and this object’s state is in set S there exists an object in the error state

15 x := new File(); y := new File(); z := y; if (?) { y.close(); z := x; } z.read(); Example,…,, … Selected false predicates

16 Ensuring Precise Polynomial Analysis How do we perform verification in polynomial time without losing precision? Ensuring polynomial time  Independent attribute analysis  Polynomial number of predicates Ensuring no loss of precision  Require set of predicates to be disjunctively WP-closed

17 Selecting Predicates using Iterated Weakest Precondition Process applied to a specific property and not to a specific client program Start with a set of predicates of interest W = { } Repeat  for every predicate p in W and every statement form st,  compute WP(st,p)  simplify WP(st,p)  add the disjuncts of simplified formula to W Resulting set W is disjunctively WP-closed

18 Applying IWP To read*close not-closed closederror read close read close read x.read() x.close()  y.read() y.close() 

19 x := new File(); y := new File(); z := y; if (?) { y.close(); z := x; } z.read(); Example Revisited,…,, … Selected false predicates

20 Selecting Predicates for Omission-Closed Properties Construct the (part of) predecessor graph on sets of states, reachable from {error} read not-closed closed, error closed, error close read close Relevant predicate families

21 Eureka Steps in Simplifying Weakest-Preconditions WP (x.read(), ) could be simplified to   or   Choosing the first form is critical … the second form will not lead to a polynomial time solution!

22 What about properties that are not omission-closed? Can we always find similar polynomial-time “precise” verification algorithms? Unfortunately, that is unlikely …

23 Repeatable Enabling Sequence Properties RES properties  a sequence c is invalid, but b + c is valid  open + ;read  read  open;open;read Verification of RES properties  NP-complete for acyclic programs  PSPACE-complete for general programs no polynomial bound exists for the length of the shortest error path (unless PSPACE = NP)

24 The Intuition … p 1.open(); … p 2.open(); … … p k.open(); … q.read(); open + ;read example … p 1.close(); … p 2.close(); … … p k.close(); … q.read(); read*;close example a hard question k easy questions Is there a path along which q  p1 & … & q  pk ? Can q = p1? … Can q = pk?

25 Is that all? No…  some properties are neither omission-closed nor repeatable-enabling-sequence properties For example  open;read  not omission-closed - open;read valid but subsequence read not valid  open is an enabling sequence (enables read ), but not repeatable - open;open;read is not valid  more results in the paper …  (lock;unlock)*

26 Taking A Different Angle Recap: some property classes are hard to verify  e.g., open + ;read What to do? Exploit properties that programs usually have A single object usually not simultaneously pointed by a large number of program variables Is it possible to do finite-state verification efficiently at least for such programs?

27 Limited Aliasing Arbitrary finite state properties can be verified in time O(n k+1 )  programs of size n, maximum aliasing factor k Naive approaches take exponential time

28 Limited Aliasing – Key Ideas Use a new predicate vocabulary [A, s] denotes  all variables in A point to the same object O  O is in state s  no variable not in A points to object O Use a forward propagation analysis  Lazily introduce predicates  Complexity proportional to number of predicate instances that take non-false value

29 Beyond Shallow Programs: Heap Analysis Use instrumented heap analysis - heap analysis guided by properties of heap objects of interest  Parametric shape analysis via 3-valued logic (Sagiv, Reps, Wilhelm, TOPLAS 02)  Establishing Local Heap Safety Properties with Applications to memory-management (Shaham, Yahav, Kolodner, Sagiv, SAS 03) Our abstractions can be used to define heap object properties of interest  Improve precision of the analysis

30 Related Work Type-based  Deline and Fahndrich, PLDI 2001 (Vault)  Foster, Terauchi, and Aiken, PLDI 2002 Selective path-sensitivity  Das, Lerner, and Seigle, PLDI 2002 (ESP)  Engler, Chelf, Chou and Hallem, OSDI 2000 Complete abstract interpretations  Giacobazzi, Ranzato, and Scozzari JACM 2000 Other  Corbett et.al., ICSE 2000 (Bandera)  Ball and Rajamani, SPIN 2001 (SLAM)  Henzinger et al., POPL 2002 (BLAST)  Naumovich, Clarke, Osterwell, and Dwyer, FLAVERS (1997)  …

31 Omission- closed Repeatable enabling sequence Almost omission- closed Other E.g. read*;closeopen + ;readopen;read(lock;unlock)* Defn.  valid(  )  valid(  )  valid(    ).  valid(  )  k   k. valid(  )  valid(  ) Acyclic programs PNP-completeP? Cyclic programs PPSPACE- complete PEP => P General: ? ? Bounded aliasing P Summary

32

33 BACKUP SLIDES

34 Omission Closed WP Rules StmtWP(Stmt, ) x:=y x := new() if x  A false if x  A  A  {x} true if A={x}  init  S false if A={x}  init  S x.op() if pre(S,op) = S  At program entrytrue if |A|=1  init  S false if |A|  1  init  S

35 Bounded Aliasing Flow Rules Stmtflow(Stmt)([A,  ]) x:=y{[A  {x},  ]} if y  A {[A  {x},  ]} if y  A x := new(){[{x},init], [A  {x},  ]} if x  A {[A,  ]} if x  A x.op(){[A,succ( ,op)]} if x  A {[A,  ]} if x  A

36 DISCARDED SLIDES Slides beyond this point are discarded slides

37 Verification by counting Counting “how often” a predicate holds true can be done efficiently for acyclic programs and certain kinds of predicates Certain finite state properties can be verified efficiently using such a counting approach in the case of acyclic graphs Can be applied to programs with loops by “unrolling loops” …  effective technique for properties where bounds can be established on the length of shortest error paths

38 Example x := new File(); y := new File(); z := y; if (?) { y.close(); z := x; } z.read(); z points to unclosed file z points to closed file z points to unclosed file read is safe z points to unclosed file

39 Example f := new File(); While (?) { f.read(); … if (?) { f.close(); f := new File(); } read is safe

40 IWP: Omission-Closed Properties WP (x.op(), ) = ? pred(S,op) = set of states whose successor on operation op is in S case 1: if pred(S,op) = S, then WP (x.op(), ) = case 2: if pred(S,op)  S, then WP (x.op(), ) =  

41 IWP: Omission-Closed Properties case 3: if NOT (pred(S,op)  S) ?  then WP (x.op(), ) is more complicated … it cannot be expressed as a predicate of the form. However, any predicate that is relevant for verification of an omission-closed property satisfies pred(S,op)  S for every op …

42 Relevant Predicates for Omission-closed Properties Which predicates are relevant? Base is relevant Induction if is relevant after x.op() and if pred(S,op)  S, then is relevant

43 Applying IWP To read*close not-closed closederror read close read close read WP (x.read(), ) =  WP (x.close(), ) =   WP (y.read(), ) = WP (y.close(), ) =  

44 Abstract Transformers Computed using weakest precondition WP(x.read(), ) x.read() 

45 Iterated Weakest Precondition: A Methodology For Designing Analyses Start with a set B of predicates of interest Iteratively compute a superset T of B such that  for every predicate p of interest and every possible statement st, compute WP(st,p)  simplify WP(st,p) and identify the disjuncts of simplified formula as predicates of interest Resulting set is disjunctively WP-closed

46 Independent Attribute Predicate Abstraction A special class of abstract interpretations Defined by a set of predicates W  Predicate = Boolean function of the concrete state  “there exists an object in the error state”  “variables x and y point to the same object” Concrete state abstracted by value of the set of predicates  A function from W to {true, false} Independent attribute analysis  A set of concrete states abstracted by a function from W to {true, false, maybe}

47 Precise Predicate Abstraction Using Independent Attributes W = set of predicates for a program P For every predicate p i in W, and every statement St in P, there exist predicates p j 1, p j 2, …, p j m in W WP (St, p i ) = p j 1 - p j 2 -… - p j m Answering “exists a path making predicate w true at program point q” can be done in O(|W||P|) time

48 Predicates For read*close For a program P with a set of variables V The set of predicates W   for every variable x  for every pair of variables x and y O(|V| 2 ) predicates Precise verification can be done in time O(|V| 2 |P|)

49 open;read open;read can be expressed as two constraints  open;open, read;open, and read;read are forbidden subsequences  an omission-closed property that can be efficiently verified  read should be preceded by open  the repeatable-enabling-sequence open + ;read Is open + ;read easy to verify for programs guaranteed not to contain the forbidden subsequences open;open, read;open, and read;read ?

50 The Intuition … p 1.open(); … p 2.open(); … … p k.open(); … q.read(); open;read example Can pi = pj, for some i <> j? then, stop; Error! Is there a path along which q <> p 1 & q <> p 2 & … & q <> p k ? not the same  Error! How often is q.read() executed? How often is q = p i for some i? How often is q = p1? … How often is q = pk?

51 Verification by Counting X  PfPtX, , u exact ,u {x}read112- {x}open;read112- {y}read112- {y}open;read011-  read2241  open;read1233 PfPt x := new File(); y := new File(); x.open(); if (?) { y.open(); } x.read(); y.read(); u:

52 Almost Omission-Closed Properties There is an integer k such that every subsequence of a valid sequence of length greater than k is also valid  open;read Polynomial time verification for acyclic programs Polynomial time verification for programs that have a polynomial bound on the length of shortest error path