Software Quality: Testing and Verification II

2 1.A failure is an unacceptable behaviour exhibited by a system — The frequency of failures measures software reliability Low failure rate = high reliability — Failures result from violation of a requirement 2.A defect is a flaw that contributes to a failure — It might take several defects to cause one failure 3.An error is a software decision that leads to a defect Software Flaws are identified at three levels

3 Eliminating Failures: Testing vs Verification Testing = running the program with a set of inputs to gain confidence that the software has few defects Goal: reduce the frequency of failures When done: after the programming is complete Methodology: develop test cases; run the program with each test case Verification = formally proving that the software has no defects (in this case, the program is said to be “correct”) Goal: eliminate failures When done: before, during and after the programming is complete Methodology: write separate specifications for the code; prove that the code and the specifications are mathematically equivalent

4 Program Correctness The correctness of a program is based on a specific standard. That standard is called a specification. int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m; } E.g., a specification for the above program might be that it “finds the maximum value of any two integers.”

5 Formalizing a Specification A Formal specification is written as a logical expression called an assertion. An assertion describes the state of the program’s variables. Two key assertions are the program’s precondition and its postcondition. int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m; } A domain is a set of values over which a variable is well defined. The primitive types (int, float, boolean, etc.) and standard Java classes (String, Vector, HashMap, etc.) provide domains for reasoning about programs. Postcondition Q = what’s ensured to be true when the program finishes Precondition P = what’s required for the program to begin its run.

6 Pre- and Postconditions Postconditions describes what it will compute. For max, a postcondition is Q = m = max(a, b), where max is a mathematical function that delivers the larger of two integers. Preconditions describe minimum requirements for the program to run. For max, a and b can be any integers, so the precondition is P = true. Before proving a program’s correctness, we first write its specifications: {true} int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m; } {m = max(a, b)}

7 Another Example – computing a factorial (n!) {n > 1} int Factorial (int n) { int f = 1; int i = 1; while (i < n) { i = i + 1; f = f * i; } return f; } {f = n!} This example raises two issues. What happens if: 1. the program has a loop that never terminates? 2. The program terminates abnormally (e.g., an exception is raised)? Precondition P Postcondition Q

8 Defining away the problem: “partial correctness” Let’s assume that nothing bad ever happens… programs always terminate after a finite number of steps, and termination is always normal. Then such a program is partially correct if, for every set of input values that satisfies precondition P, the program computes a result that satisfies postcondition Q. E.g., Factorial is partially correct if for every value of n that satisfies n > 1, it computes f =n!

9 (Partial) Correctness Proofs Let’s think generally about any program or sequence of statements s, whose pre- and postconditions are P and Q. A “Hoare triple” is a predicate of the form {P} s {Q} which asserts that “execution of statements s, beginning in a state that satisfies P, results in a state that satisfies Q.” If we can prove that this Hoare triple is valid, (i.e., it is true for all assignments of values to variables in P, Q, and s) then the program s is said to be correct. But how can we construct such a proof?

10 Constructing a Correctness Proof If the program is a series of statements: s 1 ; s 2 ; …; s n We start with the Hoare triple {P} s 1 ; s 2 ; …; s n {Q}, and use inference rules for programs to derive an intermediate triple {P i } s i {P i+1 } for every statement s i in this sequence. When done, we also ensure that Note: This process is similar to a direct proof in logic, where we use inference rules to derive a series of assertions that logically link the premises to the conclusion. So, what are the inference rules for programs?

11 Inference Rules are tied to program statement types

12 Formal methods and real programs Major question: Where do formal methods fit in the software process? That is, 1.How do we integrate them into object oriented programs? 2.How do we write pre- and postconditions P and Q for –methods? –classes? –systems? 3.Once written, how are these used to ensure that software is correct?

13 Recall the spiral model for software design Requirements Specifications (use cases) Design Coding/ Integration Testing and Verification Deployment (v 1.0)Review Prototype v 1.1 v 1.0

14 Formal methods in the object oriented design process In OO design, we focus on classes and objects Methods and messages are subordinate Collections of objects have state, which is the set of all active objects and the values of their variables at any moment of run time. Formal specifications P and Q are therefore logical expressions about each object’s state. Tools for the formal design process Specifications : Java Modeling Language (JML) Design: Unified Modeling Language (UML and JML) Coding: Java and JML Verification: JML

15 Correctness of OO systems Where? Method level: pre- and post-conditions, loop invariants Class level: class invariant (class state) System level: intra-class invariants (system state) When? Specification and design phases: Write specifications for all classes and methods (UML/JML) Coding phase: Develop code from the specifications (UML/JML/Java) Validation phase: Prove (mathematically) that specifications and code are logically equivalent (JML Java)

16 What is JML? ( History –Emerged in early 2000s out of ESC/Java2 Goals —Integration of formal methods throughout the software process —Formal specification accessible to programmers —Direct support for design by contract —Integration with a real language (Java) JML allows us to mix specifications directly with the Java code —Preconditions —Postconditions —Loop invariants —Class invariants

17 JML Basics JML specifications are special comments in a Java program: for one-liners for multiple-liners The Hoare triple {P} s 1 ; s 2 ; …; s n {Q} is written in JML/Java as ( P and Q are written as Java boolean expressions, and use parameters, local, and class variables as arguments.) requires P ; ensures Q type method (parameters) { local variables s 1 ; s 2 ; …; s n }

18 JML Language Summary Note: p and e are also good old fashioned Java boolean expressions, possibly augmented by JML-specific operations.

19 Here’s a simple example, first as a Hoare triple: {n > 1} P (precondition) int Factorial (int n) { int f = 1; int i = 1; {1 < i  i < n  f = i!} R (loop invariant) while (i < n) { i = i + 1; f = f * i; } return f; } {f = n!} Q (postcondition)

20 And again with JML expressions for P, Q, and R: requires 1 <= n ; ensures \result == (\product int i; 1<=i && i<=n; i) static int Factorial (int n) { int f = 1; int i = 1; loop_invariant i <= n && f == (\product int j; 1 <= j && j <= i; while (i < n) { i = i + 1; f = f * i; } return f; } PQR

21 JML-based software tools (1 and 3 are built into Eclipse) 1. Compiling (use jmlc instead of javac ) Does syntactic and type checking, and byte code generation for all JML assertions and Java code 2. Static checking (ESC/Java2) 3. Runtime assertion checking (use jmlrac instead of java ) Checks truth of precondition P at entry to every call Checks truth of postcondition Q at exit from every call Checks truth of loop invariant R before every iteration Issues a Java Exception when any of these is not true Note: this is not formal verification (Checking truth for one instance of a call is not the same as checking truth for all instances. The latter is proof of correctness!) 4. Proof assistance tools (Daikon, LOOP)

22 JML Eclipse Environment 3. Runtime assertion checking 1. Compiling

23 Continuing our JML example, let’s wrap Factorial inside a simple class: public class myFactorial { requires 1 <= n; ensures \result == (\product int i; 1<=i && i<=n; static int Factorial (int n) { … } public static void main(String[] args) { int n = Integer.parseInt(args[0]); System.out.println("Factorial of " + n + " = " + Factorial(n)); }

24 … compile it with jmlc, and run it twice with jmlrac: % jmlc -Q myFactorial.java % jmlrac myFactorial 3 Factorial of 3 = 6 % jmlrac myFactorial -5 Exception in thread "main” org.jmlspecs.jmlrac.runtime.JMLEntryPreconditionError: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 3, character 15 when 'n' is -5 at myFactorial.checkPre$Factorial$myFactorial(myFactorial.java:240) at myFactorial.Factorial(myFactorial.java:382) at myFactorial.main(myFactorial.java:24) normal run abnormal run (throws a JML exception)

25 JML Exceptions

26 JML helps identify errors Example 1: Suppose we change the while loop from while (i < n) to while (i <= n) so that n! will be computed incorrectly. Here’s the result: % jmlrac myFactorial 3 Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 9, character 24 when 'n' is 3 at myFactorial.internal$Factorial(myFactorial.java:102) at myFactorial.Factorial(myFactorial.java:575) at myFactorial.main(myFactorial.java:211) invariant not satisfied

27 JML Example 2 Suppose we change the while loop from while (i < n) to while (i <=n ) and also remove the JML loop invariant. Now we get: % jmlrac myFactorial 3 Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLNormalPostconditionError: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 4, character 23 when 'n' is 3 '\result' is 24 at myFactorial.checkPost$Factorial$myFactorial(myFactorial.java:321) at myFactorial.Factorial(myFactorial.java:392) at myFactorial.main(myFactorial.java:24) postcondition not satisfied

28 JML Example 3 Disagreement between a JML specification and a program may signal an error in the specification. E.g., if the loop invariant had specified j <= i rather than j < i the following outcome would occur: % jmlrac myFactorial 3 Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 9, character 24 when 'n' is 3 at myFactorial.internal$Factorial(myFactorial.java:101) at myFactorial.Factorial(myFactorial.java:573) at myFactorial.main(myFactorial.java:209) invariant not satisfied

29 But beware… JML is no panacea jmlrac doesn’t trap all errors… here are two “normal” runs: % jmlrac myFactorial 21 Factorial of 21 = % jmlrac myFactorial 32 Factorial of 32 = Recall: (1) Java has no ArithmeticOverflow exception, but (2) Factorial(n) for n > 12 should give a result > Note: jmlrac computes the same wrong result when it checks the postcondition as the Factorial method computes, so that this error goes undetected. Conclusion: the program and its specifications are both wrong. wrong results!

30 Exception Handling in JML requires P ; ensures Q ; signals (exception) type method (parameters) { locals s 1 ; s 2 ; …; s n (includes “throw new exception ;”) } We can throw Java Exceptions, and then validate their circumstances in JML whenever they occur. whenever this happens JML executes this Two outcomes: 1) expression is true and normal Java exception handling proceeds, or 2) expression is false and JMLExceptionalPostconditionError is reported.

31 JML Example 4: Throwing and Checking Exceptions requires 1 <= n; ensures \result == (\product int i; 1<=i && i<=n; i); signals (ArithmeticException) n > static int Factorial (int n) { if (n > 12) throw new ArithmeticException(); else { … % jmlrac myFactorial 22 Exception in thread "main" java.lang.ArithmeticException at myFactorial.internal$Factorial(myFactorial.java:9) at myFactorial.Factorial(myFactorial.java:610) at myFactorial.main(myFactorial.java:213) Normal Java exception handling occurs, since n > 12 is true.

32 Additional Points about JML 1. We can sometimes avoid a signals clause by strengthening the precondition. E.g., for Factorial, we could have said: requires 1 <= n && n < 13 ; 2. Specifications are always declarative; they never affect the state of the program. 3. Runtime assertion checking is not proof, but it does provide a rigorous framework for debugging. 4. JML provides a language in which formal methods and Java programs can be integrated. 5. There’s a lot more to JML. We are particularly interested in: a. class level specifications? b. Tools for static checking of specifications? c. Tools for proving correctness?

33 Readings (at the course web site) Hoare, An axiomatic basis for computer programming, Communications of the ACM 12(10): An axiomatic basis for computer programming Leino, Hoare-Style Program Verification I Hoare-Style Program Verification I Leino, Hoare-Style Program Verification II Hoare-Style Program Verification II Gordon, Specification and Verification I Specification and Verification I Dwyer et al., Software SpecificationsSoftware Specifications Leavens and Cheon, Design by Contract with JML Design by Contract with JML Poll, Kiniry, and Cok, Introduction to JML Introduction to JML Burdy et al., Overview of JML Tools Overview of JML Tools JML Reference ManualReference Manual