Automated and Modular Refinement Reasoning for Concurrent Programs Shaz Qadeer

The refinement approach P is safe P = P 0 refines P 1 … P k-1 refines P k is safe Abstraction Refinement Layered proof could be simpler both for human and computer The most natural specification of a program might be another program

Concurrent program primitive action Acquire(linear tid: int) [ assert tid != 0; assume lock == 0; lock := tid ] primitive action Release(linear tid: int) [ assert tid != 0  lock == tid; lock := 0 ] call A S1; S2 if (e) S1 else S2 while (e) S procedure P(…) { … } call P async call P call P1 || P2 gateaction

Safety Preemptive interleaved semantics Program must not fail gate of primitive atomic actions

Refines In general, any relation that preserves safety If X refines Y and Y is safe, then X is safe In our work, (non-primitive) actions are refined by procedures A partial map RS from procedures to actions for each P in dom(RS), replace every occurrence of call P with call RS(P)

Establishing refinement in two steps Reduce preemptive to cooperative semantics “yield e” for annotating cooperative semantics Check that for each P in dom(RS), body of P is simulated by skip*; RS(P); skip*

Preemptive to cooperative semantics primitive action Acquire(linear tid: int) right [ assert tid != 0; assume lock == 0; lock := tid ] primitive action Release(linear tid: int) left [ assert tid != 0; assert lock == tid; lock := 0 ] RM LM N,  Y B,L L B,R Right mover R Left moverL Both mover B Non moverN Yield Y Pairwise commutativity checks Gate 1  Gate 2 |- Act 1  Act 2  Act 2  Act 1

  R L Y RM LM N,  Y B,L L B,R procedure P(linear tid: int) requires tid != 0; { while (*) { call Acquire(tid); call Release(tid); yield true; }

Checking simulation by skip*; RS(P); skip* The sequence of yield-to-yield execution fragments from beginning to end of P must look like skip*; RS(P); skip* This check is compiled to an assertion Instrument body of P with a bit indicating whether RS(P) has occurred Save global variables at last yield in snapshot variables

Modular reasoning (in the large) A module has procedures and owns a subset of shared variables Atomic actions and yield invariants in a module cannot refer to variables owned by another module Each module is verified separately Ownership is allowed to change across refinement layers Procedures in a module verified together atomic actions, yield invariants, preconditions, postconditions, loop invariants verification conditions conjunctively decomposed for scalable parallel verification

Refinement requires invariants Ghost variables Linear variables for free invariants thread identifiers, permissions, disjoint memory Explicit non-interference reasoning yield e (location invariants ala Owicki-Gries) rely specifications via procedures that abstract over yield statements

Implementation Boogie extension for concurrency and intra-module reasoning BoogieAsm for module system and X86 code generation Summary of verification techniques Linear variables (types) Yield sufficiency (automata) Commutativity (logic) Atomicity refinement (logic) Non-interference (logic) Sequential correctness (logic)

Garbage Collector specification // y = x.f ReadField(linear tid: int, x: idx, f: fld, y: idx) atomic [ Root[y] := Mem[Root[x]][f] ] // x.f = y WriteField(linear tid: int, x: idx, f: fld, y: idx) atomic [ Mem[Root[x]][f] := Root[y] ] // x = new object Alloc(linear tid: int, x: idx) atomic [ var o: obj; assume  Alloc[o]; Alloc[o] := true; Root[x] := o ] Mem: [obj][fld]obj Root: [idx]obj Alloc: [obj]bool // Initialize data structures and start GC thread Initialize(consume gcTid: int, linear mutatorTids:[int]bool) atomic [ ]

Garbage Collector implementation A mark-sweep collector that extends Dijkstra et al. 78 Multiple parallel mutators Fast write barrier No read barrier Performance-related features that make verification difficult Mutator cooperates with collector during the mark phase Mark/Sweep/Idle phases separated by barriers Barrier for atomic root scan

Garbage Collector verification Simple high-level specification refined down to primitive operations (load, store, CAS) Six levels of refinement 2100 lines of code and specification Verifies in ~60s on my 4-core laptop Simplifying assumptions Naïve allocator (sequential search for free space) Logical objects each with the same number of fields Sequentially consistent execution

MarkAllGrays(linear tid:Tid) { while (true) { var isEmpty:bool, node:int := GraySetChoose(tid); if (isEmpty) { break; } for (var f:int := 0; f < numFields; f++) { var child:int := ReadFieldInMark(tid, node, f); if (memAddr(child)) { call GraySetInsertChildIfWhite(tid, node, child); } } call GraySetRemove(tid, node); } } WriteBarrier(linear tid:Tid, y:idx) { var rootVal:int := ReadRoot(tid, y); if (memAddr(rootVal)) { if (ReadMutatorPhase(tid) == MARK) { call GraySetInsertIfWhite(tid, rootVal); } } } Mark(linear tid:Tid) { call ResetSweepPtr(tid); while (true) { if (ScanRoots(tid)) { return; } call MarkAllGrays(tid); } } Alloc(consume tid_in:Tid, y:idx) returns(linear tid:Tid) { call tid := TestRootScanBarrier(tid_in); call UpdateMutatorPhase(tid); var ptr:int, absPtr:obj := AllocRaw(tid, y); } assert mutatorTidWhole(tid_in) && rootAddr(y) && tidOwns(tid_in, y); var o:obj; assume (memAddrAbs(o) && !allocSet[o]); allocSet[o] := true; rootAbs[y] := o; memAbs[o] :=...initial fields...; tid := tid_in; // x.f := y WriteField(linear tid:Tid, x:idx, f:fld, y:idx) { call WriteBarrier(tid, y); call WriteFieldRaw(tid, x, f, y); } assert mutatorTidWhole(tid) && fieldIndex(f) && rootAddr(x) && tidOwns(tid, x) && rootAddr(y) && tidOwns(tid, y) && memAddrAbs(rootAbs[x]); memAbs[rootAbs[x]][f] := rootAbs[y]; // y := x.f ReadField(linear tid:Tid, x:idx, f:fld, y:idx) { call ReadFieldRaw(tid, x, f, y); } assert mutatorTidWhole(tid) && fieldIndex(f) && rootAddr(x) && tidOwns(tid, x) && rootAddr(y) && tidOwns(tid, y) && memAddrAbs(rootAbs[x]); rootAbs[y] := memAbs[rootAbs[x]][f]; Initialize(consume gcTid:Tid, linear mutatorTids:[int]bool) {... async call GarbageCollect(gcTid); } Eq(linear tid:Tid, x:idx, y:idx) // x == y returns (eq:bool) {...} assert... eq := rootAbs[x] == rootAbs[y]; GarbageCollect(linear tid:Tid) { while (true) { call WaitForMutators(tid, Handshake(tid)); call Mark(tid); call WaitForMutators(tid, Handshake(tid)); call Sweep(tid); call Handshake(tid); } } ScanRoots(linear tid:Tid) returns (done:bool) { call CollectorRootScanBarrierStart(tid); call CollectorRootScanBarrierWait(tid); for (var i:int := 0; i < numRoots; i++) { var obj:int := ReadRootInRootScanBarrier(tid, i); if (memAddr(obj)) { call GraySetInsertIfWhite(tid, obj); } } call done := IsGraySetEmpty(tid); call CollectorRootScanBarrierEnd(tid); } WriteFieldRaw(linear tid:Tid, x:idx, f:fld, y:idx) { var valx:int := ReadRoot(tid, x); var valy:int := ReadRoot(tid, y); call WriteFieldGeneral(tid, valx, f, valy); } assert tid == GcTid; Color :=...; done := (forall v:int :: memAddr(v) ==> Color[v] != GRAY); assert mutatorTidWhole(tid) && rootAddr(x) && tidOwns(tid, x) && rootAddr(y) && tidOwns(tid, y) && fieldIndex(f) && memAddr(root[x]) && memAddrAbs(rootAbs[x]); memAbs[rootAbs[x]][f] := rootAbs[y]; mem[root[x]][f] := root[y]; assert mutatorTidWhole(tid) && rootAddr(y) && tidOwns(tid, y); if ( memAddr(root[y]) && Color[root[y]] == WHITE && mutatorPhase[tid] == MARK) { Color[val] := GRAY; } Sweep(linear tid:Tid) {... for (var i:int:= memLo; i < memHi; i++) { call SweepOneObject(tid); } }