1. 272: Software Engineering Fall 2012 Instructor: Tevfik Bultan Lecture 2: Software Verification with JPF, ALV and Design for Verification

2. Model Checking Evolution Earlier model checkers had their own input specification languages –For example Spin, SMV This requires translation of the system to be verified to the input langauge of the model checker –Most of the time these translations are not automated and use ad- hoc simplifications and abstractions More recently several researchers developed tools for model checking programs –These model checkers work directly on programs, i.e., their input language is a programming language –These model checkers use well-defined techniques for restricting the state space or use automated abstraction techniques

3. Explicit-State Model Checking Programs Verisoft from Bell Labs –C programs, handles concurrency, bounded search, bounded recursion. –Uses stateless search and partial order reduction. Java Path Finder (JPF) at NASA Ames –Explicit state model checking for Java programs, bounded search, bounded recursion, handles concurrency. –Uses techniques similar to the techniques used in Spin. CMC from Stanford for checking systems code written in C

4. Symbolic Model Checking of Programs CBMC –This is the bounded model checker we discussed earlier, bounds the loop iterations and recursion depth. –Uses a SAT solver. SLAM project at Microsoft Research –Symbolic model checking for C programs. Can handle unbounded recursion but does not handle concurrency. –Uses predicate abstraction and BDDs.

5. Java Path Finder Program checker for Java Properties to be verified –Properties can be specified as assertions static checking of assertions –It can also verify LTL properties Implements both depth-first and breadth-first search and looks for assertion violations statically Uses static analysis techniques to improve the efficiency of the search Requires a complete Java program It can only handle pure Java, it cannot handle native code

6. Java Path Finder, First Version First version –a translator from Java to PROMELA –Use SPIN for model checking Since SPIN cannot handle unbounded data –Restrict the program to finite domains A fixed number of objects from each class Fixed bounds for array sizes Does not scale well if these fixed bounds are increased Java source code is required for translation

7. Java Path Finder, Current Version Current version of the JPF has its own virtual machine: JVM-JPF –Executes Java bytecode can handle pure Java but can not handle native code –Has its own garbage collection –Stores the visited states and stores current path –Offers some methods to the user to optimize verification Traversal algorithm –Traverses the state-graph of the program –Tells MC-JVM to move forward, backward in the state space, and evaluate the assertion The rest of the slides on the current version of JPF

8. Storing the States JPF implements a depth-first search on the state space of the given Java program –To do depth first search we need to store the visited states There are also verification tools which use stateless search such as Verisoft The state of the program consists of –information for each thread in the Java program a stack of frames, one for each method called –the static variables in classes locks and fields for the classes –the dynamic variables (fields) in objects locks and fields for the objects

9. Storing States Efficiently Since different states can have common parts each state is divided to a set of components which are stored separately –locks, frames, fields Keep a pool for each component –A table of field values, lock values, frame values Instead of storing the value of a component in a state store an index at which the component is stored in the table in the state –The whole state becomes an integer vector JPF collapses states to integer vectors using this idea This strategy enables JPF to collapse and uncollapse parts of the states during the state space exploration

10. State Space Explosion State space explosion if one of the major challenges in model checking The idea is to reduce the number of states that have to be visited during state space exploration Here are some approaches used to attack state space explosion –Symmetry reduction search equivalent states only once –Partial order reduction do not search thread interleavings that generate equivalent behavior –Abstraction Abstract parts of the state to reduce the size of the state space

11. Symmetry Reduction Some states of the program may be equivalent –Equivalent states should be searched only once Some states may differ only in their memory layout, the order objects are created, etc. –these may not have any effect on the behavior of the program JPF makes sure that the order which the classes are loaded does not effect the state –There is a canonical ordering of the classes in the memory A similar problem occurs for location of dynamically allocated objects in the heap –If we store the memory location as the state, then we can miss equivalent states which have different memory layouts –JPF tries to remedy this problem by storing some information about the new statements that create an object and the number of times they are executed

12. Partial Order Reduction Statements of concurrently executing threads can generate many different interleavings –all these different interleavings are allowable behavior of the program A model checker has to check all possible interleavings that the behavior of the program is correct in all cases –However different interleavings may generate equivalent behaviors In such cases it is sufficient to check just one interleaving without exhausting all the possibilities –This is called partial order reduction

13. class S1 { int x;} class FirstTask extends Thread { public void run() { S1 s1; int x = 1; s1 = new S1(); x = 3; }} class Main { public static void main(String[] args) { FirstTask task1 = new FirstTask(); SecondTask task2 = new SecondTask(); task1.start(); task2.start(); }} class S2 { int y;} class SecondTask extends Thread { public void run() { S2 s2; int x = 1; s2 = new S2(); x = 3; }} state space search generates 258 states with symmetry reduction: 105 states with partial order reduction: 68 states with symmetry reduction + partial order reduction : 38 states

14. Action Language and Action Language Verifier Now, I want to talk about an infinite state model checker called Action Language Verifier (ALV) ALV is like Spin and SMV, it has its own input language called Action Language So, to verify something using ALV you first have to specify it in Action Language

15. Action Language Actions specify state changes (transitions) States correspond to valuations of variables –boolean –enumerated –integer (possibly unbounded) –heap variables (i.e., pointers) Parameterized constants –specifications are verified for every possible value of the constant Parameterized specifications –Enable verification of a protocols for arbitrary number of processes

16. Action Language Transition relation is defined using actions –Atomic actions: Predicates on current and next state variables –Action composition: asynchronous (|) or synchronous (&) Modular –Modules can have submodules –A module is defined as asynchronous and/or synchronous compositions of its actions and submodules

17. Actions in Action Language Atomic actions: Predicates on current and next state variables –Current state variables: reading, nr, busy –Next state variables: reading’, nr’, busy’ –Logical operators: not (!) and (&&) or (||) –Equality: = (for all variable types) –Linear arithmetic:, >=, <=, +, * (by a constant) An atomic action: !reading and !busy and nr’=nr+1 and reading’

18. What can we specify in Action Language? For example, we can specify a Read-Write lock implementation Then using Action Language Verifier, we can check if this read write lock satisfies a CTL property (like mutual-exclusion) integer nr; boolean busy; initial: !busy and nr=0; r_enter: [!busy] nr := nr+1; r_exit: nr := nr-1; w_enter: [!busy && nr=0] busy := true; w_exit: busy := false;

19. Read-Write Lock in Action Language module main() integer nr; boolean busy; restrict: nr>=0; initial: nr=0 and !busy; module ReaderWriter() enumerated state {idle, reading, writing}; initial: state=idle; r_enter: state=idle and !busy and nr’=nr+1 and state’=reading; r_exit: state=reading and nr’=nr-1 and state’=idle; w_enter: state=idle and !busy and nr=0 busy’ and state’=writing; w_exit: state=writing and !busy’ and state’=idle; ReaderWriter: r_enter | r_exit | w_enter | w_exit; endmodule main: ReaderWriter*(); spec: invariant(busy => nr=0) spec: invariant(busy => eventually(!busy)) endmodule S : Cartesian product of variable domains defines variable domains defines the set of states the set of states I : Predicates defining the initial states the initial states R : Atomic actions of a single process a single process R : Transition relation of a process, defined as asynchronous composition of its atomic actions R : Transition relation of main, defined as asynchronous composition of finite but arbitrary number of reader-writer modules

20. Arbitrary Number of Threads? How do we check arbitrary number of threads? Counting abstraction –Create an integer variable for each thread state –Each variable counts the number of threads in a particular state –Generate updates and guards for these variables based on the specification –Local states of the threads have to be finite Shared variables can be unbounded Counting abstraction is automated

21. Parameterized Read-Write Lock module main() integer nr; boolean busy; parameterized integer numReaderWriter; restrict: nr>=0 and numReaderWriter>=1; initial: nr=0 and !busy; module ReaderWriter() integer idle, reading, writing; initial: idle=numReaderWriter; r_enter: idle>0 and !busy and nr’=nr+1 and idle’=idle-1 and reading’=reading+1; r_exit: reading>0 and nr’=nr-1 and reading’=reading-1 and idle’=idle+1; w_enter: idle>0 and !busy and nr=0 and busy’ and idle’=idle-1 and writing’=writing+1; w_exit: writing>0 and !busy’ and writing’=writing-1 and idle’=idle+1 ReaderWriter: r_enter | r_exit | w_enter | w_exit; endmodule main: ReaderWriter(); spec: invariant(busy => nr=0) spec: invariant(busy => eventually(!busy)) endmodule

22. Action Language Verifier Action Language Verifier is an infinite state model checker that can verify properties of systems specified in Action Language The input Action Language specification is represented as a transition system: – S : The set of states – I  S : The set of initial states – R  S  S : The transition relation Properties of the input specification are expressed in temporal logics Invariant(p) : is true in a state if property p is true in every state reachable from that state –Also known as AG Eventually(p) : is true in a state if property p is true at some state on every execution path from that state –Also known as AF

23. Symbolic Model Checking Given a program and a temporal property p: Either show that all the initial states satisfy the temporal property p –set of initial states  truth set of p Or find an initial state which does not satisfy the property p –a state  set of initial states  truth set of  p We can check these in two ways: –Starting from the initial states and iteratively add states that are reachable from the current set of reachable states. We stop when there is nothing new to add. (This is called forward fixpoint computation). This computes all the reachable states. OR –Start from the bad states (  p) and iteratively add states that can reach the current set of states. We stop when there is nothing new to add. (This is called backward fixpoint computation). This computes all the states that can reach a bad state.

24. Invariant(p) pppp Initialstates initial states that violate Invariant(p) Backwardfixpoint Forwardfixpoint Initialstates states that can reach  p i.e., states that violate Invariant(p) reachable states of the system pppp Pre-condition ( backwardImage) of  p of  p reachable states that violate p Post-condition (forward image) of initial states Symbolic Model Checking Computes Fixpoints

25. Symbolic Model Checking Represent sets of states and the transition relation as logic formulas Forward and backward fixpoints can be computed by iteratively manipulating these formulas –Pre- and Post-condition computation (aka Forward, backward image): Existential variable elimination –Conjunction (intersection), disjunction (union) and negation (set difference), and equivalence check Requires use an efficient data structures for manipulation of logic formulas

26. Fixpoints May Not Converge For infinite state systems fixpoint computations may not converge (there is always something new to add, so we never stop). In fact many verification problems are undecidable for infinite state systems So we use conservative approximations

27. Conservative Approximations Compute a lower ( p  ) or an upper ( p + ) approximation to the truth set of the property ( p ) Action Language Verifier can give three answers: I p pppp 1) “The property is satisfied” I p 3) “I don’t know” 2) “The property is false and here is a counter-example” I p  p p p p sates which violate the property p+ pppp

28. Action Language Tool Set Action Language Parser Verifier OmegaLibraryCUDDPackage MONA Composite Symbolic Library PresburgerArithmeticManipulatorBDDManipulatorAutomataManipulator Action Language Specification Verified Counter example Don’t know

29. Read-Write Lock Verification with ALV IntegersBooleansCons. Time (secs.) Ver. Time (secs.) Memory (Mbytes) RW-4150.040.016.6 RW-8190.080.017 RW-161170.190.028 RW-321330.530.0310.8 RW-641651.710.0620.6 RW-P710.050.019.1

30. Read-Write Lock in Java class ReadWriteLock { private Object lockObj; private int totalReadLocksGiven; private boolean writeLockIssued; private int threadsWaitingForWriteLock; public ReadWriteLock() { lockObj = new Object(); writeLockIssued = false; } public void getReadLock() { synchronized (lockObj) { while ((writeLockIssued) || (threadsWaitingForWriteLock != 0)) { try { lockObj.wait(); } catch (InterruptedException e) { } } totalReadLocksGiven++; } } public void getWriteLock() { synchronized (lockObj) { threadsWaitingForWriteLock++; while ((totalReadLocksGiven != 0) || (writeLockIssued)) { try { lockObj.wait(); } catch (InterruptedException e) { // } } threadsWaitingForWriteLock--; writeLockIssued = true; } } public void done() { synchronized (lockObj) { //check for errors if ((totalReadLocksGiven == 0) && (!writeLockIssued)) { System.out.println(" Error: Invalid call to release the lock"); return; } if (writeLockIssued) writeLockIssued = false; else totalReadLocksGiven--; lockObj.notifyAll(); } } } How do we translate this to Action Language? Action Language Verifier Verification of Synchronization in Java Programs

31. Two Challenges in Software Model Checking State space explosion –Exponential increase in the state space with increasing number of variables and threads State space includes everything: threads, variables, control stack, heap Environment generation –Finding models for parts of software that are either not available for analysis, or are outside the scope of the model checker

32. Modular Verification Modularity is key to scalability of any verification technique –Moreover, it can help in isolating the behavior you wish to focus on, removing the parts that are beyond the scope of your verification technique Modularity is also a key concept for successful software design –The question is finding effective ways of exploiting the modularity in software during verification

33. Interfaces for Modularity How do we do modular verification? –Divide the software to a set of modules –Check each module in isolation How do we isolate a module during verification/testing? –Provide stubs representing other modules (environment) How do we get the stubs representing other modules? –Write interfaces Interfaces specify the behavior of a module from the viewpoint of other modules Generate stubs from the interfaces

34. Interfaces and Modularity: Basic Idea 1.Write interface specifications for the modules 2.Automatically generate stubs from the interface specifications 3.Automatically generated stubs provide the environment during modular verification

35. A Design for Verification Approach Our design for verification approach is based on the following principles: 1.Use of design patterns that facilitate automated verification 2.Use of stateful, behavioral interfaces which isolate the behavior and enable modular verification 3.An assume-guarantee style modular verification strategy that separates verification of the behavior from the verification of the conformance to the interface specifications 4.A general model checking technique for interface verification 5.Domain specific and specialized verification techniques for behavior verification

36. Controller Shared Concurrency Controller Pattern ThreadA ThreadB StateMachine Controller -var1 -var2 +action1() +action2() Action +blocking() +nonblocking() -GuardedExecute SharedStub +a() +b() Shared +a() +b() GuardedCommand +guard() +update() int GuardedCommand ControllerStateMachine +action1() +action2() used at runtime used during interface verification used both times Helper classes

37. Concurrency Controller Pattern Avoids usage of error-prone Java synchronization primitives: synchronize, wait, notify Separates controller behavior from the threads that use the controller –Supports a modular verification approach that exploits this modularity for scalable verification

38. class Action{ protected final Object owner; … private boolean GuardedExecute(){ boolean result=false; for(int i=0; i<gcV.size(); i++) try{ if(((GuardedCommand)gcV.get(i)).guard()){ ((GuardedCommand)gcV.get(i)).update(); result=true; break; } }catch(Exception e){} return result; } public void blocking(){ synchronized(owner) { while(!GuardedExecute()) { try{owner.wait();} catch (Exception e){} } owner.notifyAll(); } } public boolean nonblocking(){ synchronized(owner) { boolean result=GuardedExecute(); if (result) owner.notifyAll(); return result; } } class RWController implements RWInterface{ int nR; boolean busy; final Action act_r_enter, act_r_exit; final Action act_w_enter, act_w_exit; RWController() {... gcs = new Vector(); gcs.add(new GuardedCommand() { public boolean guard(){ return (nR == 0 && !busy);} public void update(){busy = true;}} ); act_w_enter = new Action(this,gcs); } public void w_enter(){ act_w_enter.blocking();} public boolean w_exit(){ return act_w_exit.nonblocking();} public void r_enter(){ act_r_enter.blocking();} public boolean r_exit(){ return act_r_exit.nonblocking();} } Reader-Writer Controller This helper class is provided. No need to rewrite it!

39. Controller Interfaces A controller interface defines the acceptable call sequences for the threads that use the controller Interfaces are specified using finite state machines public class RWStateMachine implements RWInterface{ StateTable stateTable; final static int idle=0,reading=1,writing=2; public RWStateMachine(){... stateTable.insert("w_enter",idle,writing); } public void w_enter(){ stateTable.transition("w_enter"); }... } writing reading idle r_enter r_exit w_exit w_enter

40. Interface Machine Thread 1Thread 2Thread n Thread 1 Controller Shared Data Interface Machine Thread 2 Interface Machine Thread n Thread Modular Interface Verification Concurrent Program Controller Behavior Modular Behavior Verification Modular Design / Modular Verification Interface

41. Behavior Verification Analyzing properties (specified in CTL) of the synchronization policy encapsulated with a concurrency controller and its interface –Verify the controller properties assuming that the user threads adhere to the controller interface Behavior verification with Action Language Verifier –We wrote a translator which translates controller classes to Action Language –Using counting abstraction we can check concurrency controller classes for arbitrary number of threads

42. Interface Verification A thread is correct with respect to an interface if all the call sequences generated by the thread can also be generated by the interface machine –Checks if all the threads invoke controller methods in the order specified in the interfaces –Checks if the threads access shared data only at the correct interface states

43. Interface Verification Interface verification with Java PathFinder –Verify Java implementations of threads –Correctness criteria are specified as assertions Look for assertion violations Assertions are in the StateMachine and SharedStub –Performance improvement with thread Isolation thread modular verification

44. Thread Isolation: Part 1 Interaction among threads Threads can interact with each other in only two ways: –invoking controller actions –Invoking shared data methods To isolate the threads –Replace concurrency controllers with controller interface state machines –Replace shared data with shared stubs

45. Thread Isolation: Part 2 Interaction among a thread and its environment Modeling thread’s calls to its environment with stubs –File I/O, updating GUI components, socket operations, RMI call to another program Replace with pre-written or generated stubs Modeling the environment’s influence on threads with drivers –Thread initialization, RMI events, GUI events Enclose with drivers that generate all possible events that influence controller access

46. Concurrent Program Controller Classes Thread Classes Controller Interface Machine Controller Behavior Machine Java Path Finder Action Language Verifier Thread Isolation Thread Class Counting Abstraction Interface Verification Behavior Verification Verification Framework

47. A Case Study: TSAFE Tactical Separation Assisted Flight Environment (TSAFE) functionality: 1.Display aircraft position 2.Display aircraft planned route 3.Display aircraft future projected route trajectory 4.Show conformance problems between planned and projected route

48. Server Computation Flight Database Graphical Client > 21,057 lines of code with 87 classes Radar feed > User EventThread Feed Parser Timer TSAFE Architecture

49. Reengineering TSAFE Found all the synchronization statements in the code ( synchronize, wait, notify, notifyAll ) Identified 6 shared objects protected by these synchronization statements Used 2 instances of a reader-writer controller and 3 instances of a mutex controller for synchronization In the reengineered TSAFE code the synchronization statements appear only in the Action helper class provided by the concurrency controller pattern

50. Behavior Verification Performance RW0.171.03 Mutex0.010.23 Barrier0.010.64 BB-RW0.136.76 BB-Mutex0.631.99 ControllerTime(sec)Memory (MB)P-Time (sec)P-Memory (MB) 8.1012.05 0.980.03 0.010.50 0.6310.80 2.056.47 P denotes parameterized verification for arbitrary number of threads

51. Interface Verification Performance ThreadTime (sec)Memory (MB) TServer-Main67.7217.08 TServer-RMI91.7920.31 TServer-Event6.5710.95 TServer-Feed123.1283.49 TClient-Main2.002.32 TClient-RMI17.0640.96 TClient-Event663.2133.09

52. Effectiveness in Finding Faults Created 40 faulty versions of TSAFE by fault seeding Each version had at most one interface fault and at most one behavior fault –14 behavior and 26 interface faults Among 14 behavior faults ALV identified 12 of them –2 uncaught faults were spurious Among 26 interface faults JPF identified 21 of them –2 of the uncaught faults were spurious –3 of the uncaught faults were real faults that were not caught by JPF

53. Falsification Performance TServer-RMI29.4324.74 TServer-Event6.889.56 TServer-Feed18.5194.72 TClient-RMI10.1242.64 TClient-Event15.6312.20 ThreadTime (sec)Memory (MB) RW-80.343.26 RW-161.6110.04 RW-P1.515.03 Mutex-80.020.19 Mutex-160.040.54 Mutex-p0.120.70 Concurrency ControllerTime (sec)Memory (MB)

54. Conclusions ALV performance –Cost of parameterized verification was somewhere between concrete instances with 8 and 16 threads –Falsification performance was better than verification Completeness of the controller properties –Effectiveness of behavior verification by ALV critically depends on the completeness of the specified properties Concrete vs. parameterized behavior verification –When no faults are found, the result obtained with parameterized verification is stronger –However for falsification we observed that concrete instances were as effective as parameterized instances

55. Conclusions JPF performance –Typically falsification performance is better than verification performance –In some cases faults caused execution of new code causing the falsification performance to be worse than verification performance Thread isolation –Automatic environment generation for threads result in too much non-determinism and JPF runs out of memory –Dependency analysis was crucial for mitigating this Deep faults were difficult to catch using JPF –Three uncaught faults were created to test this

56. Conclusions Unknown shared objects –The presented approach does not handle this problem –Using escape analysis may help We could not find a scalable and precise escape analysis tool Environment generation –This is the crucial problem in scalability of the interface verification –Using a design for verification approach for environment generation may help