50.530: Software Engineering Sun Jun SUTD

Week 8: Race Detection

Agenda A remainder of concurrent bugs. Common ways of coordinating threads/processes How to detect potential concurrent bugs?

Concurrent Bugs 0 1 Thread1 0 1 Thread2 count count = 0 count = 1 count = 2  count should be 2?

Concurrent Bugs Thread1 r1 i1 w Thread2 r2 i2 w r2 i2 w1 r1 i1 w2 count=1 Is it OK if it could be 1 or 2?

r2 i2 w1 r1 i1 w2 What do we do to make sure the red transitions won’t happen? Coordinate the threads based on timing “You wait for 5 minutes, I will start right away” Coordinate the threads based on timing “You wait for 5 minutes, I will start right away”

Using Thread.sleep() Example: a program with two threads, one printing 1,2,3,5 and the other printing 4 repeatedly such that the print out is 1,2,3,4,5 …

Using Thread.sleep() Thread1 0 print 1,2, wait for 10 seconds Thread2 0 wait for 5 seconds 1 2 print 4 print 5

Expected Behavior RightThreadLeftThread time print 1,2,3 5 sleeping sleepingprint 4 print 5,1,2,3 sleeping print 4 print 5,1,2,3 sleeping print 4 print 5,1,2,3 sleeping This would work except it doesn’tSleepExample.java

r2 i2 w1 r1 i1 w2 What do we do to make sure the red transitions won’t happen? Coordinate the threads based on locking If an object is to be shared by multiple threads, make sure the object is locked and there is only one key to unlock it. Coordinate the threads based on locking If an object is to be shared by multiple threads, make sure the object is locked and there is only one key to unlock it.

Using Locks Thread1 ThirdBlood.java 0 acquire lock release lock r1 i1 w1 Thread2 0 acquire lock release lock r2 i2 w2

Rules for Locks Update related state variables in a single atomic operation For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held. Every shared, mutable variable should be guarded by exactly one lock. Make it clear to maintainers which lock that is. For every invariant that involves more than one variable, all the variables involved in that invariant must be guarded by the same lock.

r2 i2 w1 r1 i1 w2 What do we do to make sure the red transitions won’t happen? Coordinate the threads by sending messages “I am working on this. I will send you a message when I am done.” Coordinate the threads by sending messages “I am working on this. I will send you a message when I am done.”

wait() and nofity() Producer/Consumer Pattern Producer Thread 1 Producer Thread 2 … Consumer Thread 1 Consumer Thread 2 … BoundedBuffer addItem removeItem BufferFixed.java

Race Conditions A race condition occurs when two or more threads can access shared data at the same time and at least one of the threads is writing. Thread 1 Thread 2 count++

Deadlock public class LeftRightDeadlock { private final Object left = new Object (); private final Object right = new Object (); public void leftRight () { synchronized (left) { synchronized (right) { doSomething(); } public void rightLeft () { synchronized (right) { synchronized (left) { doSomethingElse(); } Thread A Thread B lock left lock right try to lock right wait for lock left wait forever

Example public void transferMoney (Account from, Account to, int amount) { synchronized (from) { synchronized (to) { if (from.getBalance() < amount) { //raiseException } else { from.debit(amount); to.credit(amount) } Is it deadlocking?

Example public void transferMoney (Account from, Account to, int amount) { synchronized (from) { synchronized (to) { if (from.getBalance() < amount) { //raiseException } else { from.debit(amount); to.credit(amount) } How can transferMoney deadlock? Thread A: transferMoney(myAccount, yourAccount, 1) Thread B: transferMoney(yourAccount, myAccount, 1) Check out: DemonstrateDeadlock.java

Research Discussion Would Delta Debugging work for concurrent programs? Or the bug localization methods? Would Delta Debugging work for concurrent programs? Or the bug localization methods?

ERASER: A DYNAMIC DATA RACE DETECTOR FOR MULTI-THREADED PROGRAMS Savage et al. SOSP’97

Static vs. Dynamic Analysis Static (program) analysis: is the analysis of computer programs that is performed without actually executing the programs. – Often would result in false alarms Dynamic (program) analysis: is the analysis of computer programs that is performed by executing programs on a real or virtual processor. – Often not exhaustive

Dynamic Analysis Instrument the program Obtain execution traces Analyze the trace and wonder: Is there a potential problem like race condition? Analyze the trace and wonder: Is there a potential problem like race condition? e.g., adding print statement everywhere e.g., which are sequences of statements

Example Trace 1. Thread 1: count++ 2. Thread 2: count++ Can we spot any problem from this trace? Answer: Yes, it seemed that nothing would prevent us from reordering statement 1 and 2. Since they modify the same variable, a race condition is found! Answer: Yes, it seemed that nothing would prevent us from reordering statement 1 and 2. Since they modify the same variable, a race condition is found!

Example Trace 1. Thread 1: lock(mu) 2. Thread 1: count++ 3. Thread 1: unlock(mu) 4. Thread 2: lock(mu) 5. Thread 2: count++ 6. Thread 2: unlock(mu) Can we spot any problem from this trace? Answer: No, because of the happens-before relationship. Happens-before

Example Trace 1. Thread 1: count++ 2. Thread 1: send message to thread 2 3. Thread 1: v = v + x; 4. Thread 2: y = z + x; 5. Thread 2: get message from thread 1 5. Thread 2: count++ Can we spot any problem from this trace? Answer: No, because of the happens-before relationship. Happens-before

Happens-Before Approach Input: a sequence of statements Output: true if there is a race condition For any pair of statements s and t (assuming s is “earlier”) if (s and t are from two different threads and s and t access some shared memory and either s modifies the memory or t does) { if (there is no happens-before relation from s to t) { report potential race condition; } Input: a sequence of statements Output: true if there is a race condition For any pair of statements s and t (assuming s is “earlier”) if (s and t are from two different threads and s and t access some shared memory and either s modifies the memory or t does) { if (there is no happens-before relation from s to t) { report potential race condition; }

Example Trace Thread 1: obj1.methodA() Thread 2: obj2.methodB() Can we spot any problem from this trace? It depends on whether obj1 and obj2 are disjoint.

Example Trace Thread 1: obj1.methodA() Thread 2: obj2.methodB() A B The question is: whether A and B are disjoint?

Define Happens-Before Given a sequence of statements If statement f and u (u is after f in the sequence) are from the same thread, f happens-before u. If f sends a message and u receives the message, f happens-before u. If f unlocks an object and u locks the same object, f happens-before u. … If f happens before u and u happens-before y, f happens-before y. Is this complete?

Subtle Happens-Before Do these form a race condition? Trace Thread 1: count++ Thread 2: while (x <= 0) { Thread 2: Thread.yield() Thread 2: endwhile Thread 1: x++ Thread 2: while (x <= 0) { Thread 2: endwhile Thread 2: count++

Subtle Happens-Before Trace Thread 1: count++ Thread 2: while (x <= 0) { Thread 2: Thread.yield() Thread 2: endwhile Thread 1: x++ Thread 2: while (x <= 0) { Thread 2: endwhile Thread 2: count++

Happens-Before Algorithm In order to get precise happens-before relation, we often need to analyze the whole program. Race condition detection based on happens- before relation is very costly – no known efficient implementation so far. Race condition detection based on happens- before relation is not exhaustive even with a perfect happens-before relation.

Exercise 1 Trace Thread 1: count++ Thread 1: lock(mu) Thread 1: x++ Thread 1: unlock(mu) Thread 2: lock(mu) Thread 2: x++ Thread 2: unlock(mu) Thread 2: count++ Is there a race condition according to the happens-before approach? Is there a race condition?

Lockset Algorithm Recall (rules for locks): For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held. The lockset algorithm is designed to check if the rules is properly implemented. If not, it reports potential race condition.

Lockset Algorithm For each shared variable x set lockset(x) = *; Endfor For each access to x by thread t lockset(x) = {locks held by t at the time} intersect lockset(x) if (lockset(x) is an empty set) { report race condition; } * is a special flag denoting the set of all possible locks.

Example TraceLocks held by thread 1 and 2Lockset(count) Thread 1: lock(mu1){} for thread 1; {} for 2{mu1, mu2} Thread 1: count++{mu1} for thread 1; {} for 2{mu1} Thread 1: unlock(mu1){mu1} for thread 1; {} for 2{mu1} Thread 2: lock(mu2){} for thread 1; {} for 2{mu1} Thread 2: count++{} for thread 1; {mu2} for 2{} Thread 2: unlock(mu2)

Exercise 2 Trace Thread 1: count++ Thread 1: lock(mu) Thread 1: x++ Thread 1: unlock(mu) Thread 2: lock(mu) Thread 2: x++ Thread 2: unlock(mu) Thread 2: count++ How does Lockset find the race in this example?

Patching Lockset Locking may not be necessary in some scenarios Initialization: shared variables are frequently initialized without holding a lock Read-share data: Some shared variables are written during initialization only and are read- only thereafter. These can be safely accessed without locks. Read-write locks: Read-write locks allow multiple readers to access a shared variable, but allow only a single writer to do so.

Improving Lockset For each shared variable, we only monitor its status according to the state machine and check for race condition only at state “Shared-Modified”

Improving Lockset When a variable is in Shared-Modified state, we do the checking slightly differently. For each read-access of x by thread t { lockset(x) = {locks held by t at the time} intersect lockset(x) if (lockset(x) is an empty set) { report race condition; } For each write-access of x by thread t { lockset(x) = {locks held by t at the time in write mode} intersect lockset(x) if (lockset(x) is an empty set) { report race condition; }

Example: On Variable count TraceLocks heldWrite Locks heldStatusLockset(count) T1: lock(mu, R){} for T1 and T2 Virgin* T1: x = count+1{mu} for T1; {} for T2 {} for T1; {} for T2 Virgin* T1: unlock(mu){mu} for T1; {} for T2 {} for T1; {} for T2 Virgin* T2: lock(mu, W){} for T1; {} for T2 {} for T1; {} for T2 Virgin* T2: count++{} for T1; {mu} for T2 {} for T1; {mu} for T2 Exclusive* T2: unlock(mu){} for T1; {mu} for T2 {} for T1; {mu} for T2 Exclusive*

Exercise 3 Trace Thread 1: count++ Thread 1: lock(mu) Thread 1: x++ Thread 1: unlock(mu) Thread 2: lock(mu) Thread 2: x++ Thread 2: unlock(mu) Thread 2: count++ Draw the table for this example.

Efficiency of Lockset Implemented at the binary level Maintain a status and lockset for each memory location Performance penalty: applications with lockset implementation slows down 10 to 30 times.

Effectiveness False Alarms are produced in some scenarios. Memory reuse: the same memory is reused for different purposes later (with different locking policy). Private locks: user defines their locks without using the ones from the library Benign races: Race condition is found but deemed to be benign.

False Alarm: Example Which is worse for users: false alarms or missing some true races?

HYBRID DYNAMIC DATA RACE DETECTION O’Callahan et al. PPoPP’03

Example class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; … main.childThread = null; } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; … main.childThread = null; } Is there a race condition?

Example: Lockset Algorithm class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; … main.childThread = null; } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; … main.childThread = null; } What would the lockset algorithm report?

Example: Lockset Algorithm class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; … main.childThread = null; } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; … main.childThread = null; } What would the (not improved) lockset algorithm report? Two potential races: one globalFlag and the other on childThread. The former is false alarm, why?

The Idea Race condition detection with Lockset Race condition detection with Lockset Define a limited easy-to-check happens-before relation Define a limited easy-to-check happens-before relation Filter false alarms with the happens-before relation Potential race conditions Filtered race conditions

Exercise 4 class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class Main { int globalFlag; ChildThread childThread; void execute () { globalFlag = 1; childThread = new ChildThread(this); childThread.start(); … synchronized (this) { childThread.interrupt(); } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; synchronized (main) { main.childThread = null; } class ChildThread extends Thread { Main main; ChildThread (Main main) { this.main = main; } void run() { if (main.globalFlag == 1) …; synchronized (main) { main.childThread = null; } What would the hybrid approach report?

RACE DIRECTED RANDOM TESTING OF CONCURRENT PROGRAMS Koushik Sen. PLDI’08

Motivation Hybrid dynamic race detection: 39 out of 51 reported race conditions for tomcat are false alarms Static race detection (from Stanford, 2006) 13 out of 19 reported race conditions for hedc (a software) are false alarms Would you be happy to use these tools?

Motivation Thread1 r1 i1 w Thread2 r2 i2 w r2 i2 w1 r1 i1 w2 count=1 Even if we are sure there is a race condition, users might not be convinced if you can’t show it!

Question … Thread n … Thread n … … Thread k k1 k2 kn How many possible traces are there? How about we show the problematic trace by testing the system many times?

Scheduling threads Scheduler Thread1 Thread2 Thread3Thread4 How can we control the scheduler so that it would show the error, if there is?

Proposal: PaceFuzzer 1.Use hybrid dynamic race detection to find potential race conditions, in the form of pairs of statements (s, t). 2.PaceFuzzer executes the program with a random scheduler 3.Control the scheduler such that it keeps delaying s and t 4.Report race condition if s and t can be executed by different thread next to each other

Example What are the race conditions according to the hybrid algorithm? Assume there are two test cases, one in which thread 1 runs to finish first and the other in which thread 2 runs to finish first. What are the race conditions according to the hybrid algorithm? Assume there are two test cases, one in which thread 1 runs to finish first and the other in which thread 2 runs to finish first.

Example Test 1: thread 1 runs first and then thread 2 – Lockset based algorithm: race condition between statement 1 and 10 race condition between statement 5 and 7 – happen-before filtering The race condition between 1 and 10 is removed before statement 4 happens-before 8

Exercise 5 Test 2: thread 2 runs first and then thread 1 – Lockset based algorithm: – happen-before filtering: What are the race conditions reported in this case?

Example: Case (1, 10) Statement 1 is enabled, so delay it and let thread 2 go.

Example: Case (1, 10) Statement 1 is enabled, so delay it and let thread 2 go.

Example: Case (1, 10) Statement 1 is enabled, so delay it and let thread 2 go.

Example: Case (1, 10) Statement 1 is enabled, so delay it and let thread 2 go.

Example: Case (1, 10) Thread 2 has finished; thread 1 proceeds to execute statement 1 and more

Example: Case (1, 10) Report: No real race condition found! False alarm!

Example: Case (5, 7) Statement 7 is enabled, so delay it and let thread 1 go.

Example: Case (5, 7) Report: Real race condition identified!

The Algorithm For each pair of statements (s, t) { postponed = {} while (some thread is enabled) { randomly pick one enabled thread which is not in postponed; if execute the picked thread would execute s or t { if a thread in postponed is about to execute the other statement { report race condition and terminate; } else { add the thread to postponed; } else { execute the picked thread for one step; } report deadlock if some thread is not finished; }

Exercise 6 lock(L); x = 1; lock(L); x = 1; Apply the algorithm with this modified example for (2, 10).

A Supportive Example Without using RaceBuzzer Unlikely x = 1 will be delayed for long and hence no ERROR will be see. With high-probability 8 and 10 will be separated by lock(L) and hence happens- before will fail.

Empirical Study

Conclusion There are proposals for detecting race conditions. The algorithms/tools must work with large programs. The algorithms/tools are often neither sound nor complete – Sound: a race condition found is always a real- one. – Complete: all race conditions are found.

Example Initially: x = y = 0 thread 1 { x = random.nextInt(1,100000); count++; } thread 2 { if (x==3771) { count++; } Would any dynamic analysis approach be able to find the race condition? And be sure it is not a false alarm? Would any dynamic analysis approach be able to find the race condition? And be sure it is not a false alarm?

Can We Do Better? Recall the rules – Update related state variables in a single atomic operation – For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held. – Every shared, mutable variable should be guarded by exactly one lock. Make it clear to maintainers which lock that is. – For every invariant that involves more than one variable, all the variables involved in that invariant must be guarded by the same lock. – … Can we do better using other rules?

Example Thread 1 lock(mu) x++; y++; unlock(mu) Thread 2 lock(mu) x++; unlock(mu) lock(mu) y++; unlock(mu) Assume that we have an invariant x==y.