1. Extended Static Checking for Java Cormac Flanagan Slides courtesy of Rustan Leino

2. Motivation

3. Software development problem Software construction and maintenance are expensive Reliability is costly and difficult to achieve

4. Vision Increased programmer productivity and program reliability through increased rigor Record design decisions +Utilize automatic checking =Detect more errors and reduce costs

5. User's view Program with specifications Error messages public class Bag { private /*@non_null*/ int[] a; private int n; //@ invariant 0 <= n && n <= a.length; public Bag(/*@non_null*/ int[] initialElements) { n = initialElements.length; a = new int[n]; System.arraycopy(initialElements, 0, a, 0, n); } public void add(int x) { if (n == a.length) { int[] b = new int[2*(a.length+1)]; System.arraycopy(a, 0, b, 0, n); a = b; } a[n] = x; n++; } public int extractMin() { int m = Integer.MAX_VALUE; int mindex = 0; for (int i = 0; i < n; i++) { if (a[i] < m) { mindex = i; m = a[i]; } } if (0 < n) { n--; a[mindex] = a[n]; } return m; } // The program text continues down here, but if you’re // reading this, you probably aren’t paying attention to // the talk. Bag.java:18: Array index possibly too large

6. Extended Static Checker for Java (ESC/Java) Built at Systems Research Center Input: Java + user-supplied annotations Annotation language captures programmer design decisions Powered by – weakest precondition semantics – automatic theorem proving Performs modular checking

7. ESC/Java demo

8. Program checker design tradeoffs Missed errors Spurious warnings Annotation overhead Performance

9. ESC/Java architecture Translator Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Valid Resource exhausted

10. Tool architecture, detail Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator

11. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Tool architecture, detail Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator

12. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Simple – non_null Method annotations – requires E; – ensures E; – exsures (T x) E; – modifies w; Object invariants – invariant E;

13. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Simple – non_null Method annotations – requires E; – modifies w; – ensures E; – exsures (T x) E; Object invariants – invariant E;

14. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Specification expressions – side-effect free Java expressions no ++, no method calls – result ensures result >= 0; – old(E) ensures x == old(x)+1; – (forall T x; E), (exists T x; E), ==> (forall int j; 0 a[j] > 0);

15. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Specification expressions – side-effect free Java expressions no ++, no method calls – result ensures result >= 0; – old(E) ensures x == old(x)+1; – (forall T x; E), (exists T x; E), ==> (forall int j; 0 a[j] > 0);

16. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Miscellaneous – assert E; – assume E; assume x >= 0; // because x == y*y – nowarn x = a[j]; //@ nowarn – axiom E; axiom (forall int x; x ≫ 2 >= 0); Concurrency, ghost variables

17. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Sugared commands S,T::=assert E |assume E |x = E |raise |S ; T |S ! T |S [] T |loop {inv E} S  T end |call x = m(E) |…

18. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Sugared commands x = t.f.g; assert t ≠ null; tmp = select(f, t); assert tmp ≠ null; x = select(g, tmp) if (x = 0; */ (assume x < 0; x = -x []assume ￢(x < 0) ); assert x >= 0

19. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Sugared commands x = t.f.g; assert lblneg(“Null@58.9”, t ≠ null); tmp = select(f, t); assert lblneg(“Null@58.11”, tmp ≠ null); x = select(g, tmp) if (x = 0; */ (assume x < 0; assume lblpos(“Then^280:7”, true); x = -x []assume ￢ (x < 0); assume lblpos(“Else^280:7”, true) ); assert x >= 0

20. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Primitive commands S,T::=assert E |assume E |x = E |raise |S ; T |S ! T |S [] T |loop {inv E} S  T end |call x = m(E) |…

21. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Primitive commands //@ requires Pre; modifies w; ensures Post; void m(U u); call x = m(E) var u in u = E; assert Pre; var w 0 in w 0 = w; havoc w; assume Post; x = result end

22. |raise |S ; T |S ! T |S [] T Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Passive commands S,T::=assert E |assume E |x = E

23. private int scanPunctuation(int nextchr) { try { boolean possibleFloatingPointNumber = (nextchr == '.'); text[0] = (char)nextchr; textlen = 1; m_in.mark(); // All paths out of the try must unmark the stream!! PunctuationPrefixTree prefix = punctuationTable; PunctuationPrefixTree lastPunctuation = prefix; int lastPunctuationLength = 0; int index = nextchr - '!'; if (index < 0 || PunctuationPrefixTree.CHILDLEN <= index) prefix = null; else prefix = prefix.children[nextchr - '!']; nextchr = m_in.read(); if (possibleFloatingPointNumber && Character.isDigit((char)nextchr)) { m_in.clearMark(); return finishFloatingPointLiteral(nextchr); } this.append(nextchr); if (prefix != null && prefix.code != TagConstants.NULL) { lastPunctuation = prefix; lastPunctuationLength = textlen - 1; m_in.mark(); } while(prefix != null) { index = nextchr - '!'; if (index < 0 || PunctuationPrefixTree.CHILDLEN <= index) prefix = null; else prefix = prefix.children[nextchr - '!']; nextchr = m_in.read(); this.append(nextchr); if (prefix != null && prefix.code != TagConstants.NULL) { lastPunctuation = prefix; lastPunctuationLength = textlen - 1; m_in.mark(); } m_in.reset(); textlen = lastPunctuationLength; endingLoc = m_in.getLocation(); ttype = lastPunctuation.code; if (ttype != TagConstants.C_COMMENT&& ttype != TagConstants.EOL_COMMENT) nextchr = m_in.read(); return ttype; } catch (IOException e) { m_in.clearMark(); ErrorSet.fatal(m_in.getLocation(), e.toString()); return TagConstants.NULL; // Dummy }

24. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Passive commands if (x = 0; */ (assume x 0 < 0; x 1 = -x 0 ; x 2 = x 1 []assume ￢(x 0 < 0); x 2 = x 0 ); assert x 2 >= 0

25. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Passive commands if (x = 0; */ (assume x 0 < 0; assume x 1 == -x 0 ; assume x 2 == x 1 []assume ￢(x 0 < 0); assume x 2 == x 0 ); assert x 2 >= 0

26. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Weakest preconditions A Hoare triple {P} S {Q} says that if command S is started in a state satisfying P, then S terminates without error in a state satisfying Q The weakest precondition of a command S with respect to a postcondition Q, written wp(S, Q), is the weakest P such that {P} S {Q}

27. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Weakest preconditions wp(assert E, Q) = E && Q wp(assume E, Q) = E ==> Q wp(S;T, Q) = wp(S, wp(T,Q)) wp(S [] T, Q) = wp(S, Q) && wp(T, Q) wp(x := E, Q) = Q[ x := E ]

28. private int scanPunctuation(int nextchr) { try { boolean possibleFloatingPointNumber = (nextchr == '.'); text[0] = (char)nextchr; textlen = 1; m_in.mark(); // All paths out of the try must unmark the stream!! PunctuationPrefixTree prefix = punctuationTable; PunctuationPrefixTree lastPunctuation = prefix; int lastPunctuationLength = 0; int index = nextchr - '!'; if (index < 0 || PunctuationPrefixTree.CHILDLEN <= index) prefix = null; else prefix = prefix.children[nextchr - '!']; nextchr = m_in.read(); if (possibleFloatingPointNumber && Character.isDigit((char)nextchr)) { m_in.clearMark(); return finishFloatingPointLiteral(nextchr); } this.append(nextchr); if (prefix != null && prefix.code != TagConstants.NULL) { lastPunctuation = prefix; lastPunctuationLength = textlen - 1; m_in.mark(); } while(prefix != null) { index = nextchr - '!'; if (index < 0 || PunctuationPrefixTree.CHILDLEN <= index) prefix = null; else prefix = prefix.children[nextchr - '!']; nextchr = m_in.read(); this.append(nextchr); if (prefix != null && prefix.code != TagConstants.NULL) { lastPunctuation = prefix; lastPunctuationLength = textlen - 1; m_in.mark(); } m_in.reset(); textlen = lastPunctuationLength; endingLoc = m_in.getLocation(); ttype = lastPunctuation.code; if (ttype != TagConstants.C_COMMENT&& ttype != TagConstants.EOL_COMMENT) nextchr = m_in.read(); return ttype; } catch (IOException e) { m_in.clearMark(); ErrorSet.fatal(m_in.getLocation(), e.toString()); return TagConstants.NULL; // Dummy }

29. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Verification condition Universal background predicate – (∀t ･ t <: t) Type-specific background predicate – Bag <: java.lang.Object Verification condition: BP Univ && BP T ==> VC method

30. (BG_PUSH (AND (<: T_T |T_java.lang.Object|) (EQ T_T (asChild T_T |T_java.lang.Object|)) (DISTINCT arrayType |T_boolean| |T_char| |T_byte| |T_short| |T_int| |T_long| |T_float| |T_double| |T_.TYPE| T_T |T_java.lang.Object|))) (EXPLIES (LBLNEG |vc.T.abs.2.2| (IMPLIES (AND (EQ |elems@pre| elems) (EQ elems (asElems elems)) (< (eClosedTime elems) alloc) (EQ LS (asLockSet LS)) (EQ |alloc@pre| alloc)) (NOT (AND (EQ |@true| (is |x:2.21| T_int)) (OR (AND (OR (AND (< |x:2.21| 0) (LBLPOS |trace.Then^0,3.15| (EQ |@true| |@true|)) (EQ |x:3.17| (- 0 |x:2.21|)) (EQ |x:2.21 | |x:3.17|)) (AND (NOT (< |x:2.21| 0)) (LBLPOS |trace.Else^1,3.4| (EQ |@true| |@true|)) (EQ |x:2.21 | |x:2.21|))) (NOT (LBLNEG |Assert@4.8| (>= |x:2.21 | 0)))) (AND (OR (AND (< |x:2.21| 0) (LBLPOS |trace.Then^0,3.15| (EQ |@true| |@true|)) (EQ |x:3.17| (- 0 |x:2.21|)) (EQ |x:2.21 | |x:3.17|)) (AND (NOT (< |x:2.21| 0)) (LBLPOS |trace.Else^1,3.4| (EQ |@true| |@true|)) (EQ |x:2.21 | |x:2.21|))) (LBLNEG |Assert@4.8| (>= |x:2.21 | 0)) (NOT (LBLNEG |Exception@5.2| (EQ |ecReturn| |ecReturn|))))))))) (AND (DISTINCT |ecReturn|))) Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Verification condition class T { static int abs(int x) { if (x < 0) { x = -x; } //@ assert x >= 0; }

31. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Theorem prover: “Simplify” Nelson-Oppen cooperating decision procedures – conguence closure – linear arithmetic – partial orders – quantifiers Key features: – automatic: no user interaction – refutation based: searches for counterexamples – heuristics tuned for program checking – labels – time limit

32. Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Counterexamples and warnings Counterexample: labels: (|IndexTooBig@26.5| |vc.Bag.add.20.2| |trace.Then^0,21.23|) context: (AND (NEQ |tmp1!a:23.23| null) (NEQ this null) (EQ |alloc@pre| alloc) (EQ |tmp4!n:26.6| 0) … (<= alloc (vAllocTime |tmp3!a:26.4|)) ) Bag: add(int)... ----------------------------------------------------------- Bag.java:26: Warning: Array index possibly too large (IndexTooBig) a[n] = x; ^ Execution trace information: Executed then branch in "Bag.java", line 21, col 23. -----------------------------------------------------------

33. Experience: annotations Capture common design decisions Suggested immediately by warnings Overhead: 4-10% of source code ～1 annotation per field or parameter Most common annotations: – non_null – container element types

34. Experience: performance 50% of all methods:< 0.5 s 80% of all methods:< 1 s time limit:300 s total time for Javafe (～40kloc): 65 min.

35. Related work ESC/Modula-3 Full functional specification and verification – JML, LOOP, B, Penelope,... Languages and language features – Euclid, Eiffel, Escher, Guava, Vault, Cqual,... – LCLint, refinement types, Types against races,... Other checking techniques – Abstract interpretation, PREfix, SLAM, Bandera, Java PathFinder 2, Canvas, ESP, AST Toolkit, Metal

36. Conclusions Using weakest precondition semantics and automatic decision procedures for program analysis works! Cost effective?

37. Reading Read Hudak’s paper on functional programming. Read at least the initial part of Pierce’s paper on foundational calculi (the part on lambda calculus).