Modular Verification of Multithreaded Software Shaz Qadeer Compaq Systems Research Center Shaz Qadeer Compaq Systems Research Center Joint work with Cormac Flanagan, Stephen Freund, Sanjit Seshia

Multithreaded software  Examples: operating systems, databases, web servers, Java libraries  Verification problem –for all inputs and interleavings, program satisfies specification  Verifying multithreaded programs is difficult –large and complex

Modular verification of multithreaded programs  thread: verify each thread separately using specifications of other threads  procedure: in each thread, verify each procedure separately using specifications of called procedures

Method  Reduce verification of multithreaded programs to verification of sequential programs  Analysis of sequential programs well-studied –dataflow analysis –abstract interpretation –theorem proving with verification conditions

Action  predicate on unprimed and primed variables –(x’ = y+1  y’ = y)  x := y+1 –(p  x’ = x)  p y –true   skip

Atomic operation  actions p and q  p  q : all transitions allowed by p must also be allowed by q  (x’ = y+1  y’ = y)  x’  y’ succeeds  (x’ = y+1  y’ = y)  x’ = y’ fails  skip  p  assert p  p abbreviates p  true

Outline  modular verification –without procedure calls –with procedure calls  verification of Mercator

Simple multithreaded program Thread1 acq; x := x+1; assert x > 0; rel; Thread2 acq; x := 0; rel; m = i, if held by thread i 0, if unheld Init: x = 0

Simple multithreaded program Thread1 ( m=0  m’=1) m ; x := x+1; assert x > 0; ( m’=0) m ; Thread2 ( m=0  m’=2) m ; x := 0; ( m’=0) m ; Init: x = 0

Owicki-Gries-Lamport method Thread1 L1: m=0  x  0 (m=0  m’=1) m ; L2: m=1  x  0 x := x+1; L3: m=1  x  1 assert x > 0; L4: m=1  x  1 (m’=0) m ; L5: m=0  x  0 Thread2 M1: m=0  x  0 (m=0  m’=2) m ; M2: m=2  x  0 x := 0; M3: m=2  x  0 (m’=0) m ; M4: m=0  x  0  Sequential correctness  Non-interference

Why is Thread 1 correct? Thread 1 view ( m=0  m’=1) m ; ** x := x+1; * assert x > 0; * ( m’=0) m ; 1-abstraction A 1  ; ( m=0  m’=1) m  A 2 ; A 1  ; x := x+1  A 2 ; A 1  ; assert x > 0  A 2 ; A 1  ; ( m’=0) m  A 2 ;  (m’=0  x’  0)  (m = i  m’ = i)  (m = i  x’ = x) Ai Ai  Jones 83

Why is Thread 1 correct? Thread 1 view ( m=0  m’=1) m ; ** x := x+1; * assert x > 0; * ( m’=0) m ; 1-abstraction A 1 reflexive- transitive ! Can replace A 1  by A 1 !  A 1 ; (m=0  m’=1) m  A 2 ; A 1 ; x := x+1  A 2 ; A 1 ; assert x > 0  A 2 ; A 1 ; (m’=0) m  A 2 ;  (m’=0  x’  0)  (m = i  m’ = i)  (m = i  x’ = x) Ai Ai  Jones 83

Sequential checker Summary of method Thread 1 Thread 2 Thread n 1-abs 2-abs n-abs Sequential checker Sequential checker... yes

Assume-guarantee decomposition  Misra-Chandy 81 –processes communicating over channels  Jones 83 –parallel shared-memory programs  Abadi-Lamport 85 –temporal logic  Alur-Henzinger 96, McMillan 97 –Mealy machines (hardware) ...

Outline  modular verification –without procedure calls –with procedure calls  verification of Mercator

Simple multithreaded program Thread1 acq( ); x := x+1; assert x > 0; rel( ); Thread2 acq( ); x := 0; rel( ); Init: x = 0

Mutex implementation is not atomic ! acq( ) { t := i; while (t  0) ACS(m,0,t) ; } ACS(m,v,t)  if (m=v) then m,t := t,m rel( ) { m := 0; } What is the specification of acq( ) and rel( )?

Modular verification of sequential programs  verify each procedure separately using specifications of called procedures requires x > 0 modifies x ensures x’ = x+1 inc( ) { x := x+1; }. assert x > 0; (x’ = x+1) ;. assume x > 0; x_pre := x; y_pre := y; x := x+1; assert x = x_pre+1; assert y = y_pre;

Pre/post-conditions not enough ! Thread1 acq( ); x := x+1; assert x > 0; rel( ); acq( ) { t := 1; while (t  0) ACS(m,0,t) ; } modifies x, m ensures m=1  x  0 abstraction boundary A 1 ; t := 1  A 2 ; while (t  0) { A 1 ; ACS(m,0,t)  A 2 ; } A 1 ; assert m=1  x  0 ;  (m’=0  x’  0)  (m = i  m’ = i)  (m = i  x’ = x) Ai Ai 

Specification of acq() for Thread i Impl: acq( ) { t := i; while (t  0) ACS(m,0,t) ; } Spec: skip * ; ( m=0  m’=i) m skip ( m=0  m’=i) m m0m0 m=0 Often, witness can be guessed automatically ! Need witness to synchronize spec actions with impl actions !

Verification of acq() for Thread 1 true ; t := 1  skip; true ; while (t  0) { true ; ACS(m,0,t)  } Spec: skip * ; ( m=0  m’=1) m  m  0  skip  m=0  (m=0  m’=1) m ; In general, assumption could involve variables in scope.

Specification of rel() for Thread i Impl: rel( ) { m := 0; } Spec: ( m’=0) m

Generalization of pre/post-conditions requires p modifies U ensures q f( ) {... } assert p ; true U  ; assume q f( ) {... }

Why is Thread 1 correct? Thread1 skip * ; ( m=0  m’=1) m ; x := x+1; assert x > 0; ( m’=0) m ; Thread1 acq( ); x := x+1; assert x > 0; rel( ); A 1 ; { skip  A 2 ; A 1 ; }  ; (m=0  m’=1) m  A 2 ; A 1 ; x := x+1  A 2 ; A 1 ; assert x > 0  A 2 ; A 1 ; (m’=0) m  A 2 ;  (m’=0  x’  0)  (m = i  m’ = i)  (m = i  x’ = x) Ai Ai 

Summary  for each thread i –write assumption –write specifications for procedures –check each specification by inlining specifications of called procedures –check assumptions of other threads

Outline  modular verification –without procedure calls –with procedure calls  verification of Mercator

Implementation  Extended Static Checker for Java (Detlefs- Leino-Nelson-Saxe 98)  verification conditions proved by Simplify (Nelson 81)

More details  atomic operation is read/write of a heap variable  optimizations for reducing assumptions between atomic operations  assume sequential consistency  witness guessed automatically

Mercator (Heydon-Najork 99)  multithreaded web crawler written in Java  in production use at Altavista  synchronization based on Java monitors and reader-writer locks

Mercator architecture RAMDISK... TCTC T1T1 T NUM Checkpointing Thread Worker Threads URL set URL queue

Reader-writer locks  lock L 1 : –checkpointing thread acquires in write mode –worker threads acquire in read mode  lock L 2 : –protects data structure on disk –worker thread acquires in write mode to write and in read mode to read

Race detection  checkpointing thread –must hold L 1 in write mode  worker thread –must hold L 1 in read mode –reading: must hold L 2 in read mode –writing: must hold L 2 in write mode

Verification  analyzed ~1500 LOC  Mercator’s reader-writer locks implement specification –acq_read( ), acq_write( ), rel_read(), rel_write( )  absence of races –threads obey protocol for accessing shared data  found bug introduced by us

Annotation cost  55 annotations  Scales with complexity of synchronization patterns, not size of program

Verification of java.util.vector  synchronization with Java monitors  ~450 LOC  4 annotations  found known race (Flanagan-Freund 00)

Annotation overhead CategoryFraction of Total Read/Write Constraints & Data Invariants 6/55 Specification of Readers-Writer Locks 21/55 Other Interface Specifications 16/55 Environment Assumptions 12/55

File system Web server Disk Buffer cache Mutex module Mutex { /* Data */ boolean b; int m; /* Code */ acq() { } rel() { } } create() {. acq();. }

module Mutex { /* Data */ boolean b; int m; invariant  b  m = 0; /* Code */ spec …… acq() {. assert b;. } spec …… rel() { } }

Programs S  SeqProg ::=  atomic operation | S 1 ; S 2 composition | S 1  S 2 choice | S 1  iteration | call p procedure call P  ParProg ::= S 1 ||... || S n

Modular verification of multithreaded programs  one procedure active for each thread  interleaving of atomic actions of active procedures  atomic action is unit of induction  pre/post-conditions not enough  need abstraction of each atomic action of a procedure  specification is an abstract program