Zing: Exploiting Program Structure for Model Checking Concurrent Software Tony Andrews Shaz Qadeer Sriram K. Rajamani Jakob Rehof Microsoft Research Yichen Xie Stanford

Outline Motivation for Zing Zing overview Exploiting structure for efficient model checking Reduction Summarization Compositional conformance checking

Problem Check if programs written in common programming languages (C, C++, C#, Java) satisfy certain safety properties Examples of properties: API usage rules – ordering of calls Absence of races Absence of deadlocks Protocol (state machines) on objects

Approach Extract abstract model from the program that captures all relevant portions of the program with respect to property of interest Systematically explore the state space of the extracted model. Example: SLAM Check if a sequential C program uses an interface correctly as specified by a safety property, using boolean program models

Sequential program in rich programming language (eg. C) Finite state machines Source code FSM abstraction model checker C data structures, pointers, procedure calls, parameter passing, scoping,control flow Software model checking Boolean program Data flow analysis implemented using BDDs SLAM Push down model Related work: BLAST, MAGIC,…

Source code abstraction model checker Zing 3 core constructs: 1.Procedure calls with call-stack 2.Objects with dynamic allocation 3.Threads with dynamic creation Inter-process communication: 1.Shared memory 2.Channels with blocking-receives, non-blocking sends, FIFO Concurrent program in rich programming language

Outline Motivation for Zing Zing overview Exploiting structure for efficient model checking Reduction Summarization Compositional conformance checking

Zing: Challenges and Approach Handling programming language features Compile Zing to an intermediate object model (ZOM) Build model checker on top of ZOM State explosion Expose program structure in ZOM Exploit program structure to do efficient model checking

State Heap: complex types … Process … Processes Zing Object Model: Internal StateView Globals: simple types & refs Stack IP Locals Params IP Locals Params …

Zing Object Model: External State View Simplified view to query and update state How many processes? Is process(i) runnable? Are two states equal? Execute process(i) for one atomic step Can write simple DFS search in 10 lines

private void doDfs(){ while(stateStack.Count > 0){ State s = (State) stateStack.Peek(); bool foundSuccessor = false; // find the next process to execute and execute it for (int p = s.LastProcessExplored + 1; p < s.NumProcesses; p++) { if(s.RunnableProcesses[p] { State newS = s.Execute(p); if (!stateHash.contains(newS)){ stateHash.add(newS); stateStack.push(newS); foundSuccessor = true; break; } if(!foundSuccessor) stateStack.Pop(); } DOESNT SCALE NEED TO EXPLOIT PROGRAM STRUCTURE !

Outline Motivation for Zing Zing overview Exploiting structure for efficient model checking Reduction Summarization Compositional conformance checking

Racy program: need to explore all interleavings! local int y = 0; x := x + 1; assert(x div 4); y = y+1; //initialize int x :=0; local int z = 0; x := x + 1; assert(x div 4); z = z+1;

Race-free program: need to explore two interleavings! local int y; acquire (m); x := x + 1; assert(x div 4); release (m); y = y+1; //initialize int x :=0; mutex m; local int z; acquire (m); x := x + 1; assert(x div 4); release (m); z = z+1;

Four atomicities S0S0 S1S1 S2S2 acq(this)x S0S0 T1T1 S2S2 x S7S7 T6T6 S5S5 rel(this)z S7S7 S6S6 S5S5 z S2S2 S3S3 S4S4 r=baly S2S2 T3T3 S4S4 y S2S2 T3T3 S4S4 x S2S2 S3S3 S4S4 x R: right movers lock acquire L: left movers lock release B: both right + left movers variable access holding lock N: non-movers access unprotected variable

Transaction S0S0. S5S5 R* N L*xY... S0S0. S5S5 R* N L* xY... Other threads need not be scheduled in the middle of a transaction Lipton 75: any sequence (R+B)*; (N+ ) ; (L+B)* is a transaction

Recall example:each thread has one transaction! local int y; acquire (m); x := x + 1; assert(x div 4); release (m); y = y+1; //initialize int x :=0; mutex m; local int z; acquire (m); x := x + 1; assert(x div 4); release (m); z = z+1;

Transaction-based reduction ZOM extended to expose mover-ness of each action Model checker maintains a state machine to track the phase of each transaction Continues scheduling one thread as long as it is inside a transaction! Current implementation: Classifies all heap accesses as non-movers Can improve the scalability using better analysis (ownership?)

Outline Motivation for Zing Zing overview Exploiting structure for efficient model checking Reduction Summarization Compositional conformance checking

Summarization for sequential programs Procedure summarization (Sharir-Pnueli 81, Reps- Horwitz-Sagiv 95) is the key to efficiency int x; void incr_by_2() { x++; } void main() { … x = 0; incr_by_2(); … x = 0; incr_by_2(); … } Bebop, ESP, Moped, MC, Prefix, …

Assertion checking for sequential programs Boolean program with: g = number of global vars m = max. number of local vars in any scope k = size of the CFG of the program Complexity is O( k 2 O(g+m) ), linear in the size of CFG Summarization enables termination in the presence of recursion

Assertion checking for concurrent programs There is no algorithm for assertion checking of concurrent boolean programs, even with only two threads [Ramalingam 00]

Our approach Precise semi-algorithm for verifying properties of concurrent programs based on model checking procedure summarization for efficiency Termination for a large class of concurrent programs with recursion and shared variables Generalization of precise interprocedural dataflow analysis for sequential programs

What is a summary in sequential programs? Summary of a procedure P = Set of all (pre-state post-state) pairs obtained by invocations of P int x; void incr_by_2() { x++; } void main() { … x = 0; incr_by_2(); … x = 0; incr_by_2(); … x = 1; incr_by_2(); … } x

What is a summary in concurrent programs? Unarticulated so far Naïve extension of summaries for sequential programs do not work

Choose N = 2 Summaries: m, (a[0],a[1]) i, m, (a[0],a[1]) 0, (0, 0) 2, 0, (0,0) 0, (0, 1) 1, 0, (0,0) 0, (1, 0) 0, 0, (0,0) 0, (1, 1) 0, 0, (0,1) If a procedure body is a single transaction, summarize as in a sequential program bool available[N]; mutex m; int getResource() { int i = 0; L0: acquire(m); L1: while (i < N) { L2: if (available[i]) { L3: available[i] = false; L4: release(m); L5: return i; } L6: i++; } L7: release(m); L8: return i; }

Transactional procedures In the Atomizer benchmarks (Flanagan-Freund 04), a majority of procedures are transactional

Choose N = 2 Summaries: pc,i,(m[0],m[1]),(a[0],a[1]) pc,i,(m[0],m[1]),(a[0],a[1]) L0, 0, (0,*), (0,*) L1, 1, (0,*), (0,*) L0, 0, (0,*), (1,*) L5, 0, (0,*), (0,*) L1, 1, (*,0), (*,0) L8, 2, (*,0), (*,0) L1, 1, (*,0), (*,1) L5, 1, (*,0), (*,0) What if a procedure body comprises multiple transactions? bool available[N]; mutex m[N]; int getResource() { int i = 0; L0: while (i < N) { L1: acquire(m[i]); L2: if (available[i]) { L3: available[i] = false; L4: release(m[i]); L5: return i; } else { L6: release(m[i]); } L7: i++; } L8: return i; }

What if a transaction 1.starts in caller and ends in callee? 2.starts in callee and ends in caller? void foo() { acquire(m); x++; bar(); x--; release(m); } void bar() { release(m); acquire(m); } int x; mutex m; 2 1

What if a transaction 1.starts in caller and ends in callee? 2.starts in callee and ends in caller? void foo() { acquire(m); x++; bar(); x--; release(m); } void bar() { release(m); acquire(m); } int x; mutex m; Solution: 1.Split the summary into pieces 2.Annotate each piece to indicate whether transaction continues past it 2 1

Two-level model checking Top level performs state exploration Bottom level performs summarization Top level uses summaries to explore reduced set of interleavings, and reuse work Maintains a stack for each thread Pushes a stack frame if annotated summary edge ends in a call Pops a stack frame if annotated summary edge ends in a return

Termination Theorem: If all recursive functions are transactional, then our algorithm terminates. The algorithm reports an error iff there is an error in the program. [Qadeer-Rajamani-Rehof POPL 2004]

Summarization-based reduction ZOM extended to expose procedure boundaries Summarization implemented over transactions In progress: Benchmarking Publication of implementation details

Outline Motivation for Zing Zing overview Exploiting structure for efficient model checking Reduction Summarization Compositional conformance checking

Application

Goal Check if all message-passing interactions are well- formed No deadlocks No unreceived messages This requirement is called stuck-freeness Exploit interface specifications Check this compositionally

Compositional conformance checking Interface A A B C Intferface B Interface C Interface A Conformance A Interface B Interface C

Compostional conformance checking DefectsPass Interface A A Interface BInterface C

Stuck-free conformance B Interface B A B Interface B Stuck-freeness preserved by all environments Stuck-free A Preserves stuck-freeness

CCS

Stuck-freeness

Goal

Candidates (that dont work)

Refusal and Readiness

Examples

Conformance

Stuck-free Conformance [Rajamani, Rehof CAV 2002] [Fournet, Hoare, Rajamani, Rehof CAV 2004]

Implementation Generalize conformance definition from CCS to Zing Expose sends and receives on external channels as observable actions Disallow shared memory between processes Can have multiple threads within each process communicating through shared memory Make all other actions internal Run specification and implementation in parallel and check for conformance. Finds several errors: InventoryReservation: missing timeout specification [C2] InventoryReservation: repeated input not specified [C1] ShoppingCart: stuck state Inventory: input not implemented in service [C2] InventoryChangeNotification: inputs not available after receipt of Done

Summary Model checking software Challenges: richness of programming language, state explosion, environment modeling Zing: a model checker for concurrent software Modular architecture Transaction based reduction Summaries for procedures in concurrent programs Compositional stuck-free conformance checking Zing available for download!