Shape Analysis via 3-Valued Logic Mooly Sagiv Thomas Reps Reinhard Wilhelm

... and also Tel-Aviv University –N. Dor –T. Lev-Ami –R. Manevich –E. Yahav Universität des Saarlandes –Jörg Bauer –Ronald Biber University of Wisconsin –F. DiMaio –D. Gopan –A. Loginov –A. Mulhern IBM Research –G. Ramalingam –J. Field –D. Goyal

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; 123 NULL x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL Materialization

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Example: In-Situ List Reversal List reverse (List x) { List y, t; y = NULL; while (x != NULL) { t = y; y = x; x = x  next; y  next = t; } return y; } typedef struct list_cell { int val; struct list_cell *next; } *List; x yt NULL

Original Problem: Shape Analysis (Jones and Muchnick 1981) Characterize dynamically allocated data –x points to an acyclic list, cyclic list, tree, dag, etc. –show that data-structure invariants hold Identify may-alias relationships Establish “disjointedness” properties –x and y point to structures that do not share cells

Properties of reverse(x) On entry, x points to an acyclic list On each iteration, x & y point to disjoint acyclic lists All the pointer dereferences are safe No memory leaks On exit, y points to an acyclic list On exit, x = = NULL On exit, all cells reachable from y were reachable from x on entry, and vice versa On exit, the order between neighbors in the y-list is opposite to the order they had in the x-list on entry

Why is Shape Analysis Difficult? Destructive updating through pointers –p  next = q –Produces complicated aliasing relationships Dynamic storage allocation –No bound on the size of run-time data structures

Why is Shape Analysis Difficult? Destructive updating through pointers –p  next = q –Produces complicated aliasing relationships Dynamic storage allocation –No bound on the size of run-time data structures Data-structure invariants typically only hold at the beginning and end of operations –Need to verify that data-structure invariants are re-established

A ‘Yacc’ for Shape Analysis: TVLA Parametric framework –Some instantiations  known analyses –Other instantiations  new analyses

A ‘Yacc’ for Shape Analysis: TVLA Parametric framework –Some instantiations  known analyses –Other instantiations  new analyses Applications beyond shape analysis –Partial correctness of sorting algorithms –Safety of mobile code –Deadlock detection in multi-threaded programs –Partial correctness of mark-and-sweep gc alg. –Correct usage of Java iterators

A ‘Yacc’ for Static Analysis: TVLA Parametric framework –Some instantiations  known analyses –Other instantiations  new analyses Applications beyond shape analysis –Partial correctness of sorting algorithms –Safety of mobile code –Deadlock detection in multi-threaded programs –Partial correctness of mark-and-sweep gc alg. –Correct usage of Java iterators

x yt NULL x y t x y t return y t = y y  next = t y = x x = x  next x != NULL x yt NULL x yt x y t x y t x y t x y t x y t x y t x y t x y t x y t x y t

Formalizing “... ” Informal: x Formal: x Summary node

A ‘Yacc’ for Static Analysis (Using Logic) Correctness proofs via inductive-assertion method Proof derivation via weakest-precondition calculus “Annotate your loops with invariants!” WP

Plan Operational semantics using first order logic Canonic Abstraction Abstract interpretation using canonic abstraction Discussion

Recorded Information (for reverse)

u1u1 u2u2 u3u3 u4u4 x y

Formulas: For Observing Properties Are x and y pointer aliases?  v: x(v)  y(v)

Formulas for Observing Properties Does x point to a non NULL element  v: x(v) Are x and y pointer aliases?  v: x(v)  y(v) Does x point to a cell with a self cycle?  v : x(v)  n(v,v)

x y u1u1 u2u2 u3u3 u4u4 Are x and y Pointer Aliases?  v: x(v)  y(v) x y u1u1  Yes

x ’ (v) = x(v) y ’ (v) = 0 t ’ (v) = t(v) n ’ (v1,v2) = n(v1,v2) Predicate-Update Formulas for ‘y = NULL’

x y u1u1 u2u2 u3u3 u4u4 y ’ (v) = 0 

Predicate-Update Formulas for ‘y = x’ x ’ (v) = x(v) y ’ (v) = x(v) t ’ (v) = t(v) n ’ (v 1,v 2 ) = n(v 1,v 2 )

x u1u1 u2u2 u3u3 u4u4 Predicate-Update Formulas for ‘y = x’ y ’ (v) = x(v) y

Predicate-Update Formulas for ‘x = x  n’ x ’ (v) =  v 1 : x(v 1 )  n(v 1,v) y ’ (v) = y(v) t ’ (v) = t(v) n ’ (v 1, v 2 ) = n(v 1, v 2 )

x u1u1 u2u2 u3u3 u4u4 Predicate-Update Formulas for ‘x = x  n’ y x ’ (v) =  v 1 : x(v 1 )  n(v 1,v)   x

Predicate-Update Formulas for ‘y  n = t’ x ’ (v) = x(v) y ’ (v) = y(v) t ’ (v) = t(v) n ’ (v 1,v 2 ) =  y(v 1 )  n(v 1,v 2 )  y(v 1 )  t(v 2 )

Why is Shape Analysis Difficult? Destructive updating through pointers –p  next = q –Produces complicated aliasing relationships Dynamic storage allocation –No bound on the size of run-time data structures Data-structure invariants typically only hold at the beginning and end of operations –Need to verify that data-structure invariants are re-established

Two- vs. Three-Valued Logic 01 Two-valued logic {0,1} {0}{1} Three-valued logic {0}  {0,1} {1}  {0,1}

Two- vs. Three-Valued Logic Two-valued logicThree-valued logic

Two- vs. Three-Valued Logic Three-valued logic 0 1 Two-valued logic {1} {0,1} {0} 1 ½ 0

Two- vs. Three-Valued Logic 01 Two-valued logic {0}{1} Three-valued logic {0,1}

Two- vs. Three-Valued Logic 01 Two-valued logic ½ 01 Three-valued logic 0  ½ 1  ½

1: True 0: False 1/2: Unknown A join semi-lattice: 0  1 = 1/2 Kleene Three-Valued Logic   1/2 Information order

Boolean Connectives [Kleene]

Canonical Abstraction u1u1 u2u2 u3u3 u4u4 x u1u1 x u 234        

Canonical Abstraction u1u1 u2u2 u3u3 u4u4 x u1u1 x u 234

Canonical Abstraction u1u1 u2u2 u3u3 u4u4 x u1u1 x u 234   

Canonical Abstraction Partition the individuals into equivalence classes based on the values of their unary predicates Collapse other predicates via  –p S (u’ 1,..., u’ k ) =  {p B (u 1,..., u k ) | f(u 1 )=u’ 1,..., f(u’ k )=u’ k ) } At most 2 n abstract individuals

Safety Properties of Concurrent Java Programs (Eran Yahav, POPL 01) Potentially unbounded heap Potentially unbounded number of threads Defects checked –RW interference –WW interference –Total deadlock –Nested monitors –Illegal thread interactions

l_0: while (true) { l_1:synchronized(sharedLock) { l_C:// critical actions l_2:} l_3: } Example - Mutual Exclusion

Concrete Configuration is_thread at[l_C] rval[this] held_by blocked is_thread at[l_1] rval[this] is_thread at[l_0] is_thread at[l_0] is_thread at[l_1] rval[this] blocked is_thread at[l_1]

Abstract Configuration is_thread at[l_C] rval[this] held_by blocked is_thread at[l_1] rval[this] is_thread at[l_0]

Embedding u1u1 u2u2 u3u3 u4u4 x u5u5 u6u6 u 12 u 34 u 56 x u 123 u 456 x

Canonical Abstraction: An Embedding Whose Result is of Bounded Size u1u1 u2u2 u3u3 u4u4 x u1u1 x u 234

Property-Extraction Principle Questions about a family of two-valued stores can be answered conservatively by evaluating a formula in a three-valued store Formula evaluates to 1  formula holds in every store in the family Formula evaluates to 0  formula does not hold in any store in the family Formula evaluates to 1/2  formula may hold in some; not hold in others  

The Embedding Theorem If a structure B can be embedded in a structure S by an onto function f, such that basic predicates are preserved, i.e., p B (u 1,.., u k )  p S (f(u 1 ),..., f(u k )) Then every formula  is preserved: –If    = 1 in S, then    = 1 in B –If    = 0 in S, then    = 0 in B –If    = 1/2 in S, then    could be 0 or 1 in B

The Embedding Theorem y x u1u1 u 34 u2u2 y x u1u1 u 234 y x u1u1 u3u3 u2u2 u4u4 x y u 1234  v: x(v)  y(v) Maybe No

Cyclic versus Acyclic Lists  x u1u1 u x u1u1 u x

The Instrumentation Principle Increase precision by storing the truth-value of some chosen formulas Nullary predicates –Observations on all the concrete stores Unary Predicates –Sets of individuals with designated properties Binary Predicates –Relationships between individuals

Cyclicity Predicate  x u1u1 u x u1u1 u x c[x]() =  v 1,v 2 : x(v 1 )  n * (v 1,v 2 )  n + (v 2, v 2 ) c[x]()=0

Cyclicity Predicate  x u1u1 u x u1u1 u x c[x]() =  v 1,v 2 : x(v 1 )  n * (v 1,v 2 )  n + (v 2, v 2 ) c[x]()=1

is = 0 Heap Sharing Predicate  x is(v) =  v 1,v 2 : n(v 1,v)  n(v 2,v)  v 1  v 2 u1u1 u x u1u1 u x is = 0

Heap Sharing Predicate  x is(v) =  v 1,v 2 : n(v 1,v)  n(v 2,v)  v 1  v 2 u1u1 u x u1u1 u x is = 0 is = 1

Sortedness Predicate inOrder(v) =  v1: n(v,v1)  dle(v, v1) u1u1 u x u1u1 u x inOrder = 1 n n  x inOrder = 1 n n n        

Sortedness Predicate  x inOrder(v) =  v1: n(v,v1)  dle(v, v1) u xx inOrder = 0inOrder = 1 n n n n n inOrder = 0          n  

Reachability Predicate r(v1, v2) = n * (v1,v2) u1u1 u x u1u1 u x n n x n n n r r r r r r r r

reachable-from-variable-x(v) acyclic-along-dimension-d(v) –à la ADDS c fb (v) =  v1: f(v, v1)  b(v1, v) tree(v) dag(v) AVL trees: –balanced(v), left-heavy(v), right-heavy(v) –... but not via height arithmetic Additional Instrumentation Predicates Need FO + TC

Shape Analysis via Abstract Interpretation Start with a set of 3-valured structures at the program entry Iteratively compute a set of 3-valued structures for every program point Apply canonic abstraction after every statement Check safety properties against 3-valued structures

Predicate-Update Formulas for “y = x” y ’ (v) = x(v) Old: u1u1 u x y New: u1u1 u x 

Predicate-Update Formulas for “x = x  n” x ’ (v) =  v 1 : x(v 1 )  n(v 1,v) y Old: u1u1 u x y New: u1u1 u x  

x yt NULL return y t = y y  next = t y = x x = x  next x != NULL x yt NULL x yt u1u1 u x y u1u1 u x y ’ (v) = x(v) 1010

x yt NULL x y t x y t return y t = y y  next = t y = x x = x  next x != NULL x yt NULL x yt x y t x y t x y t x y t x y t x y t x y t x y t x y t x y t

Naïve Transformer (x = x  n) x y Evaluate update formulas y x

How Are We Doing? Conservative Convenient But not very precise  –Advancing a pointer down a list loses precision –Cannot distinguish an acyclic list from a cyclic list

x ’ (v) = x(v) y ’ (v) = 0 t ’ (v) = t(v) n ’ (v 1,v 2 ) = n(v 1,v 2 ) is ’ (v) = is(v) Predicate-Update Formulae for ‘y = NULL’

Predicate-Update Formulae for ‘y = x’ x ’ (v) = x(v) y ’ (v) = x(v) t ’ (v) = t(v) n ’ (v 1,v 2 ) = n(v 1,v 2 ) is ’ (v) = is(v)

Predicate-Update Formulae for ‘x = x  n’ x ’ (v) =  v 1 : x(v 1 )  n(v 1,v) y ’ (v) = y(v) t ’ (v) = t(v) n ’ (v 1,v 2 ) = n(v 1, v 2 ) is ’ (v) = is(v)

x ’ (v) = x(v) y ’ (v) = y(v) t ’ (v) = t(v) n ’ (v 1,v 2 ) =  y(v 1 )  n(v 1,v 2 )  y(v 1 )  t(v 2 ) is ’ (v) = Predicate-Update Formulae for ‘y  n = t’  v 1,v 2 : (is(v)  n ’ (v 1,v)  n ’ (v 2,v)  v 1  v 2 )  (t(v)  n(v 1,v)   y(v 1 ))

x ’ (v) = x(v) y ’ (v) = y(v) t ’ (v) = t(v) n ’ (v 1,v 2 ) =  y(v 1 )  n(v 1,v 2 )  y(v 1 )  t(v 2 ) is ’ (v) = Predicate-Update Formulae for ‘y  n = t’ (  ((  v 1 : y(v 1 )  n(v 1,v 2 ))  t(v))  is(v))  ( ((  v 1 : y(v 1 )  n(v 1,v 2 ))  t(v))  (is(v)  t(v))   v 1,v 2 : n ’ (v 1,v)  n ’ (v 1,v)  v 1  v 2 )

Formalizing “... ” Informal: x y Formal: x y

Formalizing “... ” Informal: x y t2t2 t1t1 Formal: x y t2t2 t1t1

Formalizing “... ” Informal: x y Formal: x y reachable from variable x reachable from variable y r[x]r[x] r[y]r[y] r[x]r[x] r[y]r[y]

Formalizing “... ” Informal: x y t2t2 t1t1 Formal: t2t2 t1t1 r[x],r[t 1 ] r[y],r[t 2 ] r[x],r[t 1 ] r[y],r[t 2 ] x y r[y]r[y] r[x]r[x]r[x]r[x] r[y]r[y]

Updating Auxiliary Information t1t1 {r[t 1 ],r[x],r[y]}{p[t 1 ],r[x],r[y]} x {p[x],p[y]} {r[x],r[y]} y t1t1 {r[t 1 ],r[x]}{p[t 1 ],r[x]} x {p[x]} {r[x]} y = NULL

Materialization x = x  n Informal: x y y x x = x  n Formal: x y x y x y x = x  n y x

Materialization x = x  n [Chase, Wegman, & Zadeck 90] x y u1u1 u2u2 y x u1u1 u2u2 x = x  n [Sagiv, Reps, & Wilhelm 96, 98] x y u1u1 u2u2 y x u1u1 u3u3 u2u2

Naïve Transformer (x = x  n) x y Evaluate update formulas y x

Best Transformer (x = x  n) x y y x y x blur y x y x  Evaluate update formulas y x y x inverse embedding canonic abstraction

“Focus”-Based Transformer (x = x  n) x y y x y x Focus(x  n) “Partial  ” y x y x Evaluate update Formulas (Kleene) y x y x blur

The Focusing Principle “Bring the structure into better focus” –Selectively force 1/2 to 0 or 1 –Avoid indefiniteness Then apply the predicate-update formulae

(1) Focus on  v 1 : x(v 1 )  n(v 1,v) u1u1 x y u x y u1u1 u x y y u1u1 u.1 x  u1u1 u.0 u

x ’ (v) =  v 1 : x(v 1 )  n(v 1,v) (2) Evaluate Predicate-Update Formulae u1u1 u x y x y u1u1 u1u1 u y x y u1u1 x u.1 u.0 x y u1u1 u.1 u.0 y u1u1 u u

The Coercion Principle Increase precision by exploiting some structural properties possessed by all stores Structural properties captured by constraints Apply a constraint solver

(3) Apply Constraint Solver y u1u1 u1u1 u y x y u1u1 u1u1 u y x x u.1 u.0 y u1u1 x u.1 y u1u1 uu

n(v 1, v )  n(v 2, v)  v 1  v 2  is(v)  is(v)  n(v 1, v)  v 1  v 2   n(v 2, v) (3) Apply Constraint Solver x y u1u1 u.1 u  1  0  

(3) Apply Constraint Solver x y u1u1 u.1 u.0

(3) Apply Constraint Solver x y u1u1 u.1 u.0  is(v)  n(v 1, v)  v 1  v 2   n(v 2, v) 1 1  1  n(v 1, v )  n(v 2, v)  v 1  v 2  is(v)   0

(3) Apply Constraint Solver x y u1u1 u.0 x(v 1 )  x(v 2 )  v 1 = v 2 u.1 1  11 

(3) Apply Constraint Solver x y u1u1 u.0

Example: InsertSort Run Demo List InsertSort(List x) { List r, pr, rn, l, pl; r = x; pr = NULL; while (r != NULL) { l = x; rn = r  n; pl = NULL; while (l != r) { if (l  data > r  data) { pr  n = rn; r  n = l; if (pl = = NULL) x = r; else pl  n = r; r = pr; break; } pl = l; l = l  n; } pr = r; r = rn; } return x; } typedef struct list_cell { int data; struct list_cell *n; } *List;

Ongoing Work Deriving update-formulas (ESOP 03) Automatic Refinement (A. Loginov) Exploring encapsulation (J. Bauer & N. Rinezkey) Integrating arithmetic (F. Dimaio, N. Dor, D. Gopan) Improving scalability (R. Manevich) –Using BDDs (SAS 02) –Widening A generic Java front-end (R. Manevich) Compile-time GC (R. Shaham) Temporal properties (E. Yahav) Assume-Guarantee Reasoning (G. Yorsh) Providing concrete counter examples (G. Erez)

TVLA Experience Quite fast on small programs –But runs on medium programs too Not a panacea More instrumentation may lead to faster (and more precise) analysis Manually updating instrumentation predicates is difficult

Why is Shape Analysis Difficult? Destructive updating through pointers –p  next = q –Produces complicated aliasing relationships –Track aliasing on 3-valued structures Dynamic storage allocation –No bound on the size of run-time data structures –Canonical abstraction  finite-sized 3-valued structures Data-structure invariants typically only hold at the beginning and end of operations –Need to verify that data-structure invariants are re- established –Query the 3-valued structures that arise at the exit

Abstraction via logic – model-based –Logical + numeric abstraction –Finite differencing of logical formulasFinite differencing of logical formulas Abstraction via logic – proof-based –Shape analysis using theorem proversShape analysis using theorem provers –Symbolic implementation of the best transformer Applications –Verifying temporal properties of concurrent softwareVerifying temporal properties of concurrent software Projects in Progress

Logical + Numeric Abstractions x u1u1 u2u2 x u u’u’’u’’’ (1,2,3,4) u u’ u’’ u’’’ Position in list (1,2) {1} x [2,4] u1u1 u2u2 (1,4)

Logical + Numeric Abstractions y = x  next (1,2) {1} x [2,4] x u1u1 u2u2 u1u1 u2u2 (1,4) u1u1 u 2.1 u 2.0 x y (1,2,3) {1} x {2} x [3,4] u1u1 u 2.1 u 2.0 (1,2,4)

u1u1 u 2.1 u 2.0 Logical + Numeric Abstractions u1u1 u 2.1 u 2.0 x y

Example: Sortedness inOrder(v) =  v 1 : n(v, v 1 )  (  v    v 1  )  x inOrder = 1       xx   Yes sorted =  v: inOrder(v)

Example: Sortedness  x inOrder(v) =  v 1 : n(v, v 1 )  (  v    v 1  )       inOrder = 1inOrder = 0inOrder = 1 sorted =  v: inOrder(v) No      inOrder = 0inOrder = 1 xx

Example: InsertSort Run Demo List InsertSort(List x) { List r, pr, rn, l, pl; r = x; pr = NULL; while (r != NULL) { l = x; rn = r  n; pl = NULL; while (l != r) { if (l  data > r  data) { pr  n = rn; r  n = l; if (pl = = NULL) x = r; else pl  n = r; r = pr; break; } pl = l; l = l  n; } pr = r; r = rn; } return x; } typedef struct list_cell { int data; struct list_cell *n; } *List;

Abstraction via logic – model-based –Logical + numeric abstraction –Finite differencing of logical formulas [A. Loginov] Abstraction via logic – proof-based –Shape analysis using theorem provers [G. Yorsh]Shape analysis using theorem provers [G. Yorsh] –Symbolic implementation of the best transformer [T. Reps] Applications –Verifying temporal properties of concurrent software [E. Yahav]Verifying temporal properties of concurrent software [E. Yahav] Projects in Progress

Work in Progress: Automatic Generation of Update Formulas for Instrumentation Predicates Obligation: Must define predicate-update formulas to update the extra predicates

Work in Progress: Automatic Generation of Update Formulas for Instrumentation Predicates Currently, user provides –Update formulas for core predicates:  c,st (v) –Definitions of instrumentation predicates:  p (v) –Update formulas for instrumentation predicates: p,st (v) Consistently defined? Goal: Create p,st from  p and the  c,st dddd

Abstraction via logic – model-based –Logical + numeric abstraction –Finite differencing of logical formulas [A. Loginov] Abstraction via logic – proof-based –Shape analysis using theorem provers [G. Yorsh] –Symbolic implementation of the best transformer [T. Reps] Applications –Verifying temporal properties of concurrent software [E. Yahav]Verifying temporal properties of concurrent software [E. Yahav] Projects in Progress

Abstract Abstract Interpretation Concrete Memory configurations Descriptors of memory configurations   T#T# T  

Abstract Best Abstract Transformer Concrete  T  Memory configurations Descriptors of memory configurations 

Symbolic Operations: Three Value-Spaces Formulas Concrete Values Abstract Values   T T 

Symbolic Operations: Three Value-Spaces FormulasConcrete Values Abstract Values  T  T#T#

Symbolic Operations: Three Value-Spaces FormulasConcrete Values Abstract Values u1u1 x u x x...    

Abstraction via logic – model-based –Logical + numeric abstraction –Finite differencing of logical formulas [A. Loginov] Abstraction via logic – proof-based –Shape analysis using theorem provers [G. Yorsh] –Symbolic implementation of the best transformer [T. Reps] Applications –Verifying temporal properties of concurrent software [E. Yahav] Projects in Progress

Java Threads Are Heap-Allocated Objects  Thread Analysis  Shape Analysis A memory configuration: thread3 inCritical lock1 isAcquired thread1 atStart thread2 atStart thread4 atStart csLock heldBy

An abstract memory configuration: thread inCritical lock1 isAcquired thread ’ atStart csLock heldBy Java Threads Are Heap-Allocated Objects  Thread Analysis  Shape Analysis

Model checking means: Explore the space of possible transitions among abstract memory configurations Java Threads Are Heap-Allocated Objects  Thread Analysis  Shape Analysis

Spatial and Temporal Properties P   Q PQ  v.  u. x(u)  n + (u,v)    (  u. x(u)  n + (u,v)) n x n y n x n y

Spatial and Temporal Properties Relate memory locations across program configurations Specify properties relating to allocation and deallocation Example: concurrent garbage collection –Safety: “Only nodes reachable from the roots are collected” –Liveness: “All garbage nodes are eventually collected” Propositional temporal logic is not enough ETL: a more expressive specification language

Why is Shape Analysis Difficult? Destructive updating through pointers –p  next = q –Produces complicated aliasing relationships –Track aliasing on 3-valued structures Dynamic storage allocation –No bound on the size of run-time data structures –Canonical abstraction  finite-sized 3-valued structures Data-structure invariants typically only hold at the beginning and end of operations –Need to verify that data-structure invariants are re- established –Query the 3-valued structures that arise at the exit

A ‘Yacc’ for Static Analysis (Using Logic) First-order structures (= predicate tables) –Hold recorded information –Model-theoretic approach, not proof-theoretic Formulas –Means for observing and updating information Abstraction controlled by choice of instrumentation predicates –What distinctions do you need to observe? –“Predicate abstraction” is the special case of nullary predicates only

A ‘Yacc’ for Static Analysis Based on Logic: Who Cares? Broad scope of potential applicability –Not just shape properties: predicates are not restricted to be links! –Discrete systems in which a relational (+ numeric) structure evolves –Transition: evolution of relational + numeric state

Example: Mark and Sweep void Sweep() { unexplored = Universe collected =  while (unexplored   ) { x = SelectAndRemove(unexplored) if (x  marked) collected = collected  {x} } assert(collected = = Universe – Reachset(root) ) } void Mark(Node root) { if (root != NULL) { pending =  pending = pending  {root} marked =  while (pending   ) { x = SelectAndRemove(pending) marked = marked  {x} t = x  left if (t  NULL) if (t  marked) pending = pending  {t} t = x  right if (t  NULL) if (t  marked) pending = pending  {t} } assert(marked = = Reachset(root)) } Run Demo

What Stores Does a 3-Valued Structure Represent? Example 3-valued structure –individuals: {u 1 } –predicates: graphical presentation concrete stores represented x u1u1 33 x 88 x 37 x

Example 3-valued structure graphical presentation concrete stores What Stores Does a 3-Valued Structure Represent? u1u1 u x u1u1 u x  x

Example 3-valued structure graphical presentation concrete stores u1u1 u x u1u1 u x  x What Stores Does a 3-Valued Structure Represent?

ETL Examples Every object allocated is eventually freed –  (  v.  v    v) Every request is eventually assigned to a handler thread –   r:request.  r    t:thread. handles(t,r)

Evolution Temporal Logic (ETL) Defined over infinite traces Based on linear temporal logic, first-order logic, plus transitive closure –Formulas may include free variables Express properties of how structures evolve Special operators  v object v allocated  v object v deallocated Predicates represent properties of interest –E.g., blocked(t,l), held_by(l,t)

ETL Traces Sequence of configurations, augmented by –Evolution edges that relate individuals of successive configurations –Sets of allocations and deallocations … deallocationallocationthreadobject evolution edge

Fine, but… Traces are infinite –How do you represent an infinite program trace? The number of traces is possibly infinite –How do you represent the set of program traces?

ETL Traces via Logical Structures … deallocationallocationthreadlockworld evolution edge

ETL Traces via Logical Structures … succ deallocationallocationthreadlockworld existence edge evolution edge

ETL Traces via Logical Structures … deallocationallocationthreadlockworld evolution edge

Fine, but… Traces are infinite –How do you represent an infinite program trace? The number of traces is possibly infinite –How do you represent the set of program traces? Use canonical abstraction to abstract traces –A summary world can represent the infinite tails of an infinite number of unexplored trace –Canonical abstractions are finite

Equality Under Abstraction twe   evolution twe(u,u) = 1 twe(u,u) = ½

Abstract Interpretation (Over-)approximate possibly infinite set of infinite traces by a finite set of finite abstract traces Successive approximation –Start with an abstract trace that represents all traces –Repeatedly refine the state by exploring longer abstract traces Evaluate the property of interest over abstract traces

Current Implementation Manually –Convert ETL to FO+TC –Define instrumentation predicates for temporal subformulas Let TVLA do the rest Properties proven: –Termination of linked-list manipulation –Response (fair/unfair)