1. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 1 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Verification & Testing Dynamic Algorithms for Concurrency Problems Roderick Bloem Sources: Savage, Burrows, Nelson, Sobalvarro, Anderson, Eraser: A Dynamic Race Detector for Multithreaded Programs. ACM Transactions on Computer Systems 15, 1997 Visser et al, Model Checking Programs, Model Checking Programs, Automated Software Engineering 10, 2003

2. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 2 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Deadlocks & Race Conditions Deadlocks show themselves when a program hangs Race conditions cause unexpected results Hard to find because they often occur only with a specific scheduling. Often not found during testing but as low-frequency (high-impact) bugs at client site. Hard to reproduce. Today: Algorithms that find these problems without looking at all schedulings.

3. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 3 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Dynamic Tools for Concurrency Problems We want something that is better than testing We need techniques that we can use for any program we can run! We’re willing to sacrifice precision: we will get unnecessary warnings, undiscovered bugs Subject: dynamic methods to find concurrency errors – deadlocks and race conditions Dynamic methods: Result depends on exact run (inputs and scheduling) Try to minimize dependence on scheduling (better than testing!)

4. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 4 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Background: Java Synchronization Mechanism 1: object-level locks class C{ void synchronized f(){…} void synchronized g(){…} } For each object, only one thread can execute a synchronized method When f() or g() are entered, the object is locked. Methods f() and g() can not execute in different threads at the same time. A thread never blocks on a lock held by itself Mechanism 2: block-level locks void f(){ C o = new C();... synchronized(o){... } } When reaching synchronized(0), execution is suspended until lock is free, then o is locked. Lock is released at end of block Note: Two mechanisms use the same per-object locks

5. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 5 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Java Thread States (Monitors) running scheduler synchronized() x.wait() lock x released (exit from block/method) x.notify() ready suspended on lock x waiting for notify on x See any OS textbook for a description of monitors

6. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 6 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Java Synchronization wait(): release lock, wait for notify notifyAll(): wake up all waiting threads notify(): wakes up one thread waiting on object’s lock (not very useful) yield(): be nice, give someone else the turn Typical use: synchronized void doWhenCondition(){ while(!condition) wait(); …perform action… } synchronized void changeCondition(){ …change the condition… notifyAll(); …do some computation… }//lock released here

7. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 7 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Java Synchronization Java keeps a list of waiting threads per object When a notify comes –one of the threads (say, T) is removed from the list If no objects are on the list, the notify is forgotten –Thread T waits for the object lock to be freed, –T acquires the lock –T continues execution notifyAll() does the same for all threads

8. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 8 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Java Thread States running scheduler synchronized() x.wait() lock is freed lock x released (exit from block/method) x.notify() Wait and notify act on the lock for the object this. ready suspended on lock x waiting for notify on x

9. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 9 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Locking Example int available = 0; thread 1: public synchronized int get() { while (!available) { try { wait(); } catch (InterruptedException e) { } } available = false; notifyAll(); return contents; //still locked! } thread 2: public synchronized void put(int value) { while (available) { try { wait(); } catch (InterruptedException e) { } } contents = value; available = true; notifyAll(); }

10. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 10 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Explicit Locks ReentrantLock l = new ReentrantLock(); l.lock(); … l.unlock(); Note: synchronized locks are just locks on “this”

11. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 11 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Deadlock A deadlock is a circular wait For locks, this is called lock reversal: Thread one holds lock A, waits for lock B Thread two holds lock B, waits for lock A or with three threads: Thread one holds lock A, waits for lock B Thread two holds lock B, waits for lock C Thread three holds lock C, waits for lock A

12. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 12 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Deadlock Example Alice a = new Alice(); Bob b = new Bob(); class Alice{ synchronized void hug(){ b.hug() ; }} class Bob{ synchronized void hug(){ a.hug(); }}

13. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 13 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Deadlock Example ReentrantLock ALock = new ReentrantLock; ReentrantLock BLock = new ReentrantLock; class Alice{ void hug(){ ALock.lock(); Block.lock(); work… Block.unlock() ALock.unlock(); }} class Bob{ void hug(){ BLock.lock(); Alock.lock(); work… Alock.unlock(); BLock.unlock(); }} Thread 1 calls Alice.hug() Thread 1 calls ALock.lock() [T1 holds AlLock] Thread 2 calls Bob.hug Thread 2 calls Block.lock(); [T1 holds AlLock, T2 holds BLock] Thread 1 calls Block.lock() [T1 holds ALock waits for BLock, T2 holds BLock] Thread 2 calls Alock.lock() [T1 holds ALock waits for BLock, T2 holds BLock, waits for ALock] (deadly embrace)

14. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 14 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Gate Locks A gate lock prevents a deadlock by protecting the areas with lock reversal ReentrantLock gateLock; class Alice{ void hug(){ gateLock.lock(); ALock.lock(); Block.lock(); Block.unlock() ALock.unlock(); gateLock.unlock(); }} class Bob{ void hug(){ gateLock.lock(); BLock.lock(); Alock.lock(); Alock.unlock() BLock.unlock(); gateLock.unlock(); }}

15. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 15 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Lock Tree Algorithm Dynamic algorithm to find deadlocks Lock reversal: only for deadlocks with two threads Dynamic: may miss possibilities (statements not executed at all or not in every possible order) False warnings: other mechanisms may prevent deadlock (e.g., shared variable) Only looks at locks In a tree keep track the order in which locks are acquired and released; see if there are reversals

16. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 16 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Lock Tree Algorithm Build trees during runtime –each tree has a current node –If lock acquired create new child and move to it –If node released, move up one level After termination, analyze trees. Possible deadlock if 1.T1 contains a node Li with ancestor Lj 2.T2 tree contains a node Lj with ancestor Li 3.There is no gate lock: node Lk which is A gate lock is a lock that is 1.an ancestor of Li and Lj in T1 and 2.an ancestor of Li and Lj in T2 Limitations Works for deadlocks involving two threads only Works only for properly nested locks

17. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 17 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Lock Tree Thread 1: L1.lock(); L3.lock(); L2.lock(); L2.unlock(); L4.lock(); L4.unlock(); L3.unlock() L1.unlock(); L4.lock(); L2.lock(); L3.lock(); L3.unlock() L2.unlock(); L4.unlock(); Thread 2: L1.lock(); L2.lock(); L3.lock(); L3.unlock(); L2.unlock L1.unlock(); L4.lock(); L3.lock(); L2.lock(); L2.unlock(); L3.unluck(); L4.unlock(); Let’s draw lock tree by executing T1 first and then T2

18. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 18 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Lock Tree Where are the potential deadlocks?

19. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 19 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Deadlocks Potential deadlocks in the last example, –L3L4 left versus L4L3 right is a problem –L3L2 left versus L2L3 right is not: protected by L1 –L2L3 left versus L3L2 right is not: protected by L4 To get deadlock: 1.Execute T2, stop when L4 acquired, 2.Execute T1 until deadlock. Note: executing T1 first then T2 will not give deadlock. By executing one scheduling we found a problem in a different scheduling!

20. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 20 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Limitations 1.Dependence on execution: If suspicious code is never executed, we do not find deadlock 2.Deadlocks do not have to be due to locks 3.Deadlocks can be prevented without using locks (trick for 2,3: build your own lock.)

21. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 21 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Limitations: An undetected Deadlock class Lock{} Lock lock; int a = 0, b = 0; class Alice{ void hug(){ synchronize(lock){ while(a==0) lock.wait(); } a = 0; b = 1; synchronize(lock){ lock.notifyAll; } } class Bob{ void hug(){ synchronize(lock){ while(b==0) lock.wait(); } b = 0; a = 1; synchronize(lock){ lock.notifyAll(); }

22. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 22 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Limitations: LockTree detects False Deadlock class Lock{} Lock lock; int a = 0; // the gate lock class Alice{ ReentrantLock ALock = …; void hug(){ synchronize(lock){ while(a==0) lock.wait(); } ALock.lock(); Block.lock(); Block.unlock() ALock.unlock(); a = 0; synchronize(lock){ lock.notifyAll(); } class Bob{ ReentrantLock Block = …; void hug(){ synchronize(lock){ while(a==1) lock.wait(); } Block.lock(); Alock.lock(); Alock.unlock() Block.unlock(); a = 1; synchronize(lock){ lock.notifyAll() }

23. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 23 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Data Races

24. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 24 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Data Race A data race exists when: Two threads access variable concurrently At least one access is write Nothing prevents simultaneous access When a data race occurs, the result depends on the interleaving Not necessarily bad Thermometer writes to int temp, GUI reads: no locks needed But be careful: Writes to ints are atomic, so this works if temp is a long or a structure, you need locking How do you usually prevent race conditions?

25. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 25 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Eraser Check locking behavior For any piece of shared data, is there at least one lock that is always held on access? This condition is sufficient but not necessary for correctness, Dynamic algorithm –Computes locks held during one run –May not find all problems –May warn when no problem exists –What it finds depends on the run!

26. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 26 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Bank Account (Grandma’s Disappearing Money) class Acct{ private long balance; private long acctNr; Acct(){ acctNr = Acct.getNewNr(); balance = 0; } long getAcctNr(){ return acctNr; } long getBalance(){ return balance; } void deposit(long amount){ long current; current = balance; current += amount; balance = current; }

27. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 27 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Data Race void deposit(long amount){ long current; current = this.balance; current += depositAmount; this.balance = current; } Initial balance is 0, deposit 100 twice. Final balance: 100 instead of 200. Thread 1 (You): account1.deposit(100) current = balance; (0) current += amount; (100) balance = current; (100) Thread 2 (Grandma): account1.deposit(100) current = balance; (0) current += amount; (100) balance = current; (100) Where did Grandma’s money go?? Same problem occurs if you use balance +=amount.

28. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 28 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Eraser – Simple Version At any point in time, a thread t holds a set of locks: locks(t) Associate with each variable v a set of lock candidates, C(v) For each variable v { C(v) = all_locks; } // called when thread t reads variable v read(t,v){ C(v) := C(v)  locks(t); if C(v) =  then issue warning; } // same for write(t,v) Note: minimal dependence on order of scheduling! Results only depends on execution paths taken (which may in turn depend on scheduler)

29. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 29 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Example Thread 1Thread 2locks(T1)locks(T2)C(v)  {l1, l2} l1.lock(); {l1} v := v + 1; {l1} l1. unlock()  l2.lock() {l2} v := v + 1;  : warning! l2.unlock() 

30. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 30 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Bank Account, 2 class Acct{ private long balance; private long acctNr; private ReentrantLock l; Acct(){ acctNr = Acct.getNewNr(); balance = 0; l = new Lock(); } long getAcctNr(){ return acctNr; } long getBalance(){ long currentBalance; l.lock(); currentBalance = balance; l.unlock(); return currentBalace; } void deposit(long amount){ long current; l.lock(); current = balance; current += amount; balance = current; l.unlock(); } } Does this solve our problem?

31. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 31 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Remaining Problems Program is now correct but Eraser does not understand: 1.Initialization is not protected But: initialization is never simultaneous with anything else! 2.Account number not protected Also, an efficiency problem: Two threads reading account data have to wait for each other. Note: we should exclude simultaneous read/writes, but simultaneous reads are OK. We will solve problem 1 & 2 first

32. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 32 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Initialization & Read-Shared Virgin: new data Exclusive: only one thread has access (initialization mode) Shared: read-only, after initialization finished shared-modified: at least one writer and one reader Start computing lock sets when second thread accesses variable Report warnings when moving to shared-modified & lock set empty Side effect: increased dependency on scheduler. (When do we leave Exclusive?)

33. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 33 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Eraser, version II //called when thread t reads var v read(t,v){ case state(v) of{ VIRGIN: read before write!; EXCLUSIVE: if( t != threadid(v) ){ state(v) = SHARED; locks(v) = locks(t); } SHARED: locks(v) = locks(v)  locks(t); SHARED-MODIFIED: locks(v) = locks(v)  locks(t); if(locks(v) =  ) emit warning; endcase } Per variable keep: state when exclusive: thread id when shared: lock set //called when thread t writes var v write(t,v){ case state(v) of{ VIRGIN: state(v) = EXCLUSIVE; threadid(v) = t; EXCLUSIVE: if(t != threadid(v)){ state(v) = SHARED-MODIFIED; locks(v) = locks(t); } SHARED: state(v) = SHARED-MODIFED; locks(v) = locks(v)  locks(t); if(locks(v) =  ) emit warning; SHARED-MODIFIED: locks(v) = locks(v)  locks(t); if(locks(v) =  ) emit warning; endcase }

34. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 34 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Problem 2: Read/Write Locks Let’s solve problem 2: simultaneous reads should be allowed Read-write locks allow for multiple simultaneous readers, a write is never simultaneous with another read or write. Useful if you have many reads, regular writes. (Tricky to implement: prevention of starvation for writers) Lock l = new ReentrantReadWriteLock(); // acquire/release l in read mode l.readLock().lock(); l.readLock().unlock(); // acquire/release l in write mode l.writeLock().lock(); l.writeLock().unlock();

35. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 35 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Bank Account, 3 class Acct{ private long balance; private long acctNr; private ReentrantReadWriteLock l; Acct(){ acctNr = Acct.getNewNr(); balance = 0; l = new ReentrantReadWriteLock(); } long getAcctNr(){ return acctNr; } long getBalance(){ long currentBalance; l.readLock().lock(); currentBalance = balance; l.readLock().unlock(); return currentBalace; } void deposit(long amount){ long current; l.writeLock().lock(); current = balance; current += depositAmount; balance = current; l.writeLock().unlock(); } }

36. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 36 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Problem Lockset does not work properly Bank account is correct, but –write lock is not always held and –always holding read lock is not enough (a write with just a read lock would be a problem)

37. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 37 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Lockset for Read/Write Locks Let locks(t) be the set of locks held by t Let write_locks(t) be the set of write locks held by t For each variable v { C(v) = all_locks; } read(t,v){ C(v) := C(v)  locks(t); if C(v) =  then issue warning; } wite(t,v){ C(v) := C(v)  write_locks(t); if C(v) =  then issue warning; }

38. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 38 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Example Thread 1rlockswlocks Thread 2 rlockswlocksC(v)  all locks l.rdl.lk() {l} l.rdl.lk() {l} read v {l} read v {l} l.rdl.ulk()   l.wl.lk() {l} write v {l} l.wl.ulk()  l.rl.lk() {l} write v  : warning!

39. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 39 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Remaining False Alarms Memory reuse: a private memory manager may use a location for one purpose first, then for another purpose. Locks will be different Private locks. Benign races Solution: annotations –EraserReuse() –Eraser{Read/Write}{Lock/Unlock}() –EraserIgnore{On/Off}()

40. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 40 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Implementation For a C compiler Modify binary code Instrument each load and store operation Instrument calls to storage allocator for dynamic memory Not for local variables (locals are indexed off the stack pointer ) Warning accompanied by line number and stack traces Based on DEC’s ATOM, similar to Purify & Valgrind

41. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 41 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Storing Lock Sets Each lock set is a sorted list Locks sets are stored in table and referred to by index Locks sets are entered in hash map which associates index to set. When a lockset is constructed, and not in hash map, new index is constructed Intersections are cached and otherwise are linear in the size of the lock set For every data word there is a shadow word which keeps state (virgin, exclusive, shared, shared-modified, 2 bits) and in the other 30 bits: –If state is exclusive, the thread id of exclusive owner –otherwise, the lock set index

42. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 42 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Memory Layout

43. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 43 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Example (Black Board) We have locks 1, 2, 3. One variable, v –v is virgin t1 writes v, no locks –v is exclusive, owned by thread 1 t2 writes v with locks {1,2} –v is shared-modified, lock set T2 t2 writes v with locks {1,2} –v is shared modified, lock set T2 t2 writes v with locks {1,2} –v is shared modified, lock set T2 (cached intersection) t2 writes v with locks {2,3} –v is shared-modified, lock set T4 Final State lock set table: T1: {1,2,3} T2: {1,2} T3: {2,3} T4: {2} Lock hash {1,2,3}  1 {1,2}  2 {2,3}  3 {2}  4 Intersection cache T1  T2 = T2 T2  T2 = T2 T2  T3 = T4

44. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 44 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Experience Two AltaVista servers: –mhttpd: http server, 5kLoC in C, 100 locks, 250 lock sets –Ni2: indexing engine, 20kLoC in C, 900 locks, 3600 lock sets No bugs and no false alarms after adding 19 annotations

45. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 45 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Experience Vesta cache server for software configuration management. 30kLoC, 26 locks, 70 lock sets. One bug: fingerprint is a field with associated bit FPvalid. This was accessed without a lock. This is a bug because the fingerprint and FPvalid need not be consistent. More bugs in Petal, and lots of bugs (10%) in undergrad homework

46. Institute for Software Technology Professor Horst Cerjak, 19.12.2005 46 Roderick Bloem V&T 02Dynamic Algorithms for Concurrency Conclusions Dynamic algorithms May give false alarms May not find all problems Locktree finds possible deadlocks Eraser finds possible race conditions Little dependence on scheduling: Can find bug in one scheduling by executing another one: better than testing.