Verifying Concurrent Programs with Relaxed Conflict Detection Tayfun Elmas, Ismail Kuru, Serdar Taşıran, Omer Subasi Koç University Istanbul, Turkey

Relaxed Conflict Detection Pattern – Traverse/Take snapshot of (large portion of) global state – Do local computation – Update small portion of global state Very common – Concurrent data structures – Parallelized optimization algorithms – Cloud computing – Performance optimizations for transactional memory Ignore WAR conflicts (Titos et al., HiPEAC ‘12) Early release/early discard (e.g., Kozyrakis et al., WTW ‘06) Performance problem: Most operations conflict Solution: Program so that some conflicts can be ignored Verification problem: How do you reason about correctness? 2

Motivating Example: Sorted Linked List Head 5 16 Insert 5 Insert 16

Motivating Example: Sorted Linked List Head 5 16 Insert 5 Insert 16

Head READ

Head 5 WRITE READ

Write- After- Read conflict Head 16 WRITE READ 5 WRITE

Write- After- Read conflict Head 16 WRITE READ Conventional TM conflict detection enforces conflict serializability – Does WriteSet(Tx1) intersect (ReadSet(Tx1) + WriteSet(Tx2)) ? – Any two concurrent insertions conflict! 5 WRITE

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic { *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; } 9 Transactional memory: Optimistic concurrency Track read and write accesses a transaction performs – ReadSet(Tx1) – WriteSet(Tx1) Enforce conflict serializability At commit time for Tx1 for all concurrent transactions Tx2, check WriteSet(Tx2)  (ReadSet(Tx1)  WriteSet(Tx1)) =  – Abort and retry Tx1 otherwise

Head 5 16 But, allowing both insertions OK even if we ignore WAR conflict Conflict serializability too strict Options: – Fine-grain, hand-crafted concurrency – Transactional memory + relaxed conflict detection

Relaxed conflict detection Programmer tells TM to ignore this kind of conflict Need to reason that this still results in a correct program Write- After- Read conflict Head 16 WRITE READ 5 WRITE

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic { *curr = listPtr->head; Read do { phase prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; Write prev->next = node; phase } 12

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic { *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 13 Strict conflict detection: Can reason about transaction code sequentially.

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 14 Ignore Write-After-Read conflicts: Writes by others interfere with transaction Cannot reason sequentially Ignore Write-After-Read conflicts: Writes by others interfere with transaction Cannot reason sequentially READ 6; WRITE 5

Arguing that the !WAR block is atomic list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 15 Would like these actions to be “right movers” Can commute to the right of any action by another thread Would like these actions to be “right movers” Can commute to the right of any action by another thread Atomicity of this block guaranteed by TM Write-write conflicts not ignored Atomicity of this block guaranteed by TM Write-write conflicts not ignored

Making !WAR block atomic list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr T1 = curr T1 ->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 16 node T2 ->next = curr T2 ; READ 6; WRITE 5

Making !WAR block atomic list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr T1 = curr T1 ->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 17 node T2 ->next = curr T2 ; Ignored WAR conflict: curr T1 = curr T1 ->next; does not move to the right of node T2 ->next = curr T2 ;

Abstraction list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { (curr T1 = curr T1 ->next)+; havoc prev; assume prev->next = curr; assume prev->key key; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 18 Abstract action right mover Now block is atomic. Sequential verification WRITE 5; READ 5, 6;

1. Sequentially verify the original code list_insert(list_t *listPtr, node_t *node) { *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 19

2. Apply program transformation list_insert(list_t *listPtr, node_t *node) { *curr = listPtr->head; do { (curr T1 = curr T1 ->next)+; havoc prev; assume prev->next = curr; assume prev->key key; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 20 Do global read abstractions  Abstract transaction becomes atomic.

3. Prove abstract code sequentially 21 list_insert(list_t *listPtr, node_t *node) { *curr = listPtr->head; do { (curr T1 = curr T1 ->next)+; havoc prev; assume prev->next = curr; assume prev->key key; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); }

Relaxed Conflict Detection Pattern – Traverse/Take snapshot of (large portion of) global state – Do local computation – Update small portion of global state Concurrent data structures, optimization algorithms, cloud computing Performance optimizations for transactional memory – Ignore WAR conflicts (Titos et al., HiPEAC ‘12) – Early release/early discard (e.g., Kozyrakis et al., WTW ‘06) Issue: Snapshot may be stale – But programmer thinks this is OK (as long as there is no write-write conflict with another operation) Problem: – How to verify properties of code running under a relaxed consistency model 22

WAR Conflict Pattern read x read y read x write x write y 23

WAR Conflict Pattern read x read y read x write x write y 24 WAR conflict OK

WAR Conflict Pattern read x read y read x write x write y 25 WAR conflict OK as long as no WAW conflict

WAR Conflict Pattern read x read y read x write x write y 26 WAR conflict OK as long as no WAW conflict read x read y read x write x write x WAR conflict WAW conflict

WAR Conflict Pattern read x read y read x write x write y 27 WAR conflict OK as long as no WAW conflict read x read y read x write x write x WAR conflict WAW conflict A B O R T

Labyrinth: Grid Routing FindRoute(p1,p2) – Route a wire on the grid connecting points p1 and p2 – Wires must not touch FindRoute operation – Take snapshot of grid – Find shortest path from p1 to p2 – Write path to shared memory if path does not overlap others Overlap = write-write conflict Stale snapshot OK as long as computed path does not overlap others – Same path could have been computed even with up-to-date snapshot 28

Write- After- Read conflict Head 16 WRITE READ 5 WRITE

snapshotGS := GlobSt n ; (x, xVal) := localComp(snapshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } 30 Non-problematic Interleaving GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; …

31 Actual Interleaving GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … snapshotGS := GlobSt 0 ; snapshotGS := GlobSt n ; (x, xVal) := localComp(snapshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } This is OK because localComp(GlobSt n, args) and localComp(GlobSt 0,args) would/might have produced same result.

Correctness Argument 32 Blue transaction’s snapshot is stale Blue and green accesses conflict But, end state consistent with –green executed serially, then –blue executed serially Head 16 WRITE READ 5 WRITE

QED: A Proof System for Concurrent Programs Shaz Qadeer Microsoft Research Redmond, WA Serdar Taşıran, Tayfun Elmas, Ali Sezgin Koç University Istanbul, Turkey [POPL ’09, TACAS ‘10, PADTAD ‘10, VSTTE ‘10]

Intuition for proof steps Abstract some actions Prove non-interference Commute Prove larger blocks atomic Abstract some actions Prove non-interference Commute Prove larger blocks atomic

Coarser Atomic Actions check P1P1 PnPn P2P2 Correct Difficult to prove Fine-grain concurrency Annotations at every interleaving point Easy to prove Larger atomic blocks Local, sequential analysis within atomic blocks

36 QED’s Idea of Abstraction  If for all : error s1s1 s1s1 1. If then s1s1 2. Else, if then s2s2 s1s1 s2s2 or error s1s1 s1s1      abstracted by 36

37 Flavors of Abstraction if (x == 1) y := y + 1; if ( * ) y := y + 1; Adding non-determinism Adding assertions (more behaviors that might go wrong) t := x; havoc t; assume x != t; skip; assert (lock_owner == tid); x := t + 1; 37 “Read abstraction”

QED Transformations: Reduction [ S1; S2] [ S1 ] ; [ S2 ] I,PI,P’ 38 If [S1] is a right mover or [S2] is a left mover P P’P’

39 Reduction  ;  ;  ...   1   2 ...   n   ;  ...  right-mover: For each execution: Exist equivalent executions:...     1   2 ...   n      1     2 ...   n      1   2 ...     n   ...  ;  39

Mover check in QED: Static, local, semantic First-order verification condition For each ;... Right-mover ? A AB ; BA B : S1S1 S2S2 S3S3 S1S1 T2T2 S3S3 AB BA All actions in program run by different thread

41 Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Need this to be a “Right mover” Must commute to the right of all other concurrent actions Atomicity ensured by strict conflict detection

42 Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Need this to be a “Right mover” Must commute to the right of all other concurrent actions Atomicity ensured by strict conflict detection Reasoning using Lipton’s reduction and abstraction QED-like proof (“A Calculus of Atomic Actions”, POPL ’09) Costly pairwise mover checks avoided

Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Not a “Right mover” Does not commute to the right of global state updates

Reads and Updates do not Commute READ 6; WRITE WRITE 5; READ 5 Want to read 6 again!

Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Not a “Right mover” Does not commute to the right of global state updates Idea: Abstract this read Add non-determinism so it can read GlobSt 1, GlobSt 2, …, GlobSt n

Abstraction​ intuition ABSTRACT READ 6; WRITE WRITE 5; ABSTRACT READ 6 Need to jump over READ 6; WRITE WRITE 5; READ 5 Want to read 6 again!

Abstraction​ intuition ABSTRACT READ 6; WRITE WRITE 5; ABSTRACT READ 6 Need to jump over READ 6; WRITE WRITE 5; READ 5 Want to read 6 again! curr = curr->next; abstracted by curr = curr->next*; but don’t go past key.

Abstracting Read Accesses Abstraction invariant Abs T,x : A transition invariant for variable of type T at address x l := x.f becomes “Read returns a non-deterministic value satisfying abstraction invariant” For the linked list insertion example Abs node,x (old_st, new_st) = Reach old_st (list->head, x) ==> Reach new_st (list->head, x) ∧ Reach new_st (x, old(x->next)) ∧ Sorted old_st (list) ∧ Sorted new_st (list) 48

Abstracting Accesses Annotate write accesses: – x.f := m becomes – Assertion: Proof obligation, ensures the abstraction invariant By construction, abstract reads commute to the right of annotated writes Assertions become proof obligations – Discharged using sequential reasoning – Abstraction invariant verified when operations are atomic Soundness guaranteed by theorem – Special case of QED soundness theorem – No pairwise mover check needed! 49

Ongoing Work: Obtaining the (Sequential) Specification When interpreted sequentially, the program may already have a specification – Spec already existed for Labyrinth and Genome Specifications (invariants) may be inferred mechanically Done for the Genome benchmark using the Celia tool – Invariant Synthesis for Programs Manipulating Lists with Unbounded Data A. Bouajjani, C. Dragoi, C. Enea, A. Rezine, and M. Sighireanu CAV'10. – On Inter-Procedural Analysis of Programs with Lists and Data A. Bouajjani, C. Dragoi, C. Enea, and M. Sighireanu PLDI'11. 50

Future Work: Obtaining the Abstraction Invariant Sometimes “true” is sufficient Sometimes easy: Full cells never become empty (Labyrinth) Otherwise, need an over-approximation for conflicting_updates* o read_access Can use annotation inference tool on the sequential code for while (*) conflicting_updates; read_access; 51

Summary Common pattern – Traverse/Take snapshot of (large portion of) global state – Do local computation – Update small portion of global state Conflict serializability too costly – Fine-grain locking – Relaxed conflict detection Static proof approach for such programs – Read abstraction to capture effects of conflicting writes – Enables sequential proof on abstract program Ongoing work: Automation for inferring spec, abstraction invariant 52

Ongoing Work: Port Entire Proof Argument to VCC VCC: Verification tool for concurrent C programs (MS Aachen) – Used to verify Microsoft Hyper-V hypervisor – Uses Boogie, Z3 Why VCC? – Can handle entire C – Larger agenda: Model and verify programs using novel concurrent programming constructs Programs with TM: race-freedom, static separation, assertions Programs with relaxed consistency models, e.g. snapshot isolation Key ideas in VCC – Ownership, permissions expressed using ghost variables All annotations expressed using program logic – Objects have two-state invariants VCC checks that all actions preserve two-state invariants of all objects Modular checks, converted to first-order logic queries for Z3. 53

Modeling Relaxed Conflict Detection in VCC Current proof argument – For each type, write an abstraction invariant – Relax read accesses using abstraction invariant Read accesses become right movers by construction – Relaxed code is atomic Discharge assertions used to justify right moverness – Verify desired properties on non-deterministic but sequential code Proof argument in VCC – Model TM and relaxed conflict detection using ownership, approval – Write two-state invariant about values returned by reads – Write two-state invariants about program objects – Verify that two-state invariants are admissible, preserved – Verify same specification as sequential version, Only rely on two-state invariant and atomicity within write block 54

Serdar Taşıran

Earlier Microsoft-Koç University Collaboration Goldilocks: A Race Detection Tool – Patented – Included in US and European university curricula – Published in CACM Research Highlights One of 24 articles selected in 2010 QED: A Verification Tool for Concurrent Programs – Winner: ACM Student Research Competition – Best tool built using Microsoft technology – Publicly available reduce abstract..... reduce check Correct... P1P1 PnPn P2P2 P1P1 PnPn Proof script Boogie 2, Z3 QEDPL program

Ongoing Research Tool support for new programming paradigms: – DesCloud: Cloud simulation on a desktop. – VCC verifier, relaxed conflict detection, relaxed consistency models Program Proof Tools – Handling relaxed hardware memory models in QED – Verifying TM implementations with QED Dynamic (Runtime) verification tools – Parallelizing race detection – Using TM to prevent and recover from races – Races in programs that use TM