1. 50.530: Software Engineering
Sun Jun
SUTD

2. Week 6: Race Detection

3. Agenda A remainder of concurrent bugs.
Common ways of coordinating threads/processes
How to detect potential concurrent bugs?

4. Concurrent Bugs 1 Thread1 Thread2 count++ 00 count = 0  count = 1 011
Thread1
Thread2
count++ 00
count = 0 
count = 1 01 10
count = 1 11
count = 2
count should be 2?

5. Concurrent Bugs 00 Thread1 Thread2 r2 01 10 r1 r2 r1 1 1 02 11 20 i201 10 r1 r2 r1 1 1 02 11 20 i2 i1 i2 03 12 21 30 i1 2 2 13 22 31 w1 w2 w1 3 3 23 32
count=1 w2 33
Is it OK if it could be 1 or 2?

6. 00 r2
What do we do to make sure the red transitions won’t happen? 01 10 r1 02 11 20 i2 03 12 21 30 i1
Coordinate the threads based on timing
“You wait for 5 minutes, I will start
right away” 13 22 31 w1 23 32 w2 33

7. Using Thread.sleep()
Example: a program with two threads, one printing 1,2,3,5 and the other printing 4 repeatedly such that the print out is
1,2,3,4,5 …

8. Using Thread.sleep() Thread1 Thread2 print 1,2,3 wait for 5 seconds 1
print 1,2,3
wait for 5 seconds 1 1
print 4
wait for 10 seconds 2 2
print 5 3

9. This would work except it doesn’t
Expected Behavior
LeftThread
RightThread
time
print 1,2,3
sleeping 5
sleeping
print 4
print 5,1,2,3 10 15
sleeping
print 4
print 5,1,2,3 20 25
sleeping
print 4
print 5,1,2,3 30
This would work except it doesn’t
SleepExample.java

10. 00 r2
What do we do to make sure the red transitions won’t happen? 01 10 r1 02 11 20 i2 03 12 21 30 i1
Coordinate the threads based on locking
If an object is to be shared by multiple threads, make sure the object is locked and there is only one key to unlock it. 13 22 31 w1 23 32 w2 33

11. Using Locks ThirdBlood.java Thread1 Thread2 acquire lock acquire lock
acquire lock
acquire lock 1 1 r1 r2 2 2 i1 i2 3 3 w1 w2 4 4
release lock
release lock 5 5
ThirdBlood.java

12. Rules for Locks
Update related state variables in a single atomic operation
For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held.
Every shared, mutable variable should be guarded by exactly one lock. Make it clear to maintainers which lock that is.
For every invariant that involves more than one variable, all the variables involved in that invariant must be guarded by the same lock.

13. 00 r2
What do we do to make sure the red transitions won’t happen? 01 10 r1 02 11 20 i2 03 12 21 30 i1
Coordinate the threads by sending messages
“I am working on this. I will send you a message when I am done.” 13 22 31 w1 23 32 w2 33

14. wait() and nofity() Producer/Consumer Pattern BufferFixed.java
BoundedBuffer
Producer Thread 1
Producer Thread 2 …
Consumer Thread 1
Consumer Thread 2 …
addItem
removeItem
BufferFixed.java

15. Race Conditions
A race condition occurs when two or more threads can access shared data at the same time and at least one of the threads is writing.
Thread 1
Thread 2
count++
count++

16. Deadlock Thread A Thread B lock left lock right try to lock right
public class LeftRightDeadlock {
private final Object left = new Object ();
private final Object right = new Object ();
public void leftRight () {
synchronized (left) {
synchronized (right) {
doSomething(); }
public void rightLeft () {
doSomethingElse();
Thread A
Thread B
lock left
lock right
try to lock right
wait for lock left
wait forever
wait forever

17. Example
public void transferMoney (Account from, Account to, int amount) {
synchronized (from) {
synchronized (to) {
if (from.getBalance() < amount) {
//raiseException }
else {
from.debit(amount);
to.credit(amount)
Is it deadlocking?

18. Example
public void transferMoney (Account from, Account to, int amount) {
synchronized (from) {
synchronized (to) {
if (from.getBalance() < amount) {
//raiseException }
else {
from.debit(amount);
to.credit(amount)
How can transferMoney deadlock?
Thread A: transferMoney(myAccount, yourAccount, 1)
Thread B: transferMoney(yourAccount, myAccount, 1)
Check out: DemonstrateDeadlock.java

19. Research Discussion
Would Delta Debugging work for concurrent programs?
Or the bug localization methods?

20. Eraser: A Dynamic Data Race Detector for Multi-threaded programs
Savage et al. SOSP’97
Eraser: A Dynamic Data Race Detector for Multi-threaded programs

21. Static vs. Dynamic Analysis
Static (program) analysis: is the analysis of computer programs that is performed without actually executing the programs.
Often would result in false alarms
Dynamic (program) analysis: is the analysis of computer programs that is performed by executing programs on a real or virtual processor. 
Often not exhaustive

22. Dynamic Analysis Instrument the program
e.g., adding print statement everywhere
e.g., which are sequences of statements
Obtain execution traces
Analyze the trace and wonder:
Is there a potential problem like race condition?

23. Example Trace 1. Thread 1: count++ 2. Thread 2: count++
Can we spot any problem from this trace?
Answer: Yes, it seemed that nothing would prevent us from reordering statement 1 and 2.
Since they modify the same variable, a race condition is found!

24. Example Trace 1. Thread 1: lock(mu) 2. Thread 1: count++
3. Thread 1: unlock(mu)
4. Thread 2: lock(mu)
5. Thread 2: count++
6. Thread 2: unlock(mu)
Happens-before
Happens-before
Happens-before
Can we spot any problem from this trace?
Answer: No, because of the happens-before relationship.

25. Example Trace 1. Thread 1: count++
2. Thread 1: send message to thread 2
3. Thread 1: v = v + x;
4. Thread 2: y = z + x;
5. Thread 2: get message from thread 1
5. Thread 2: count++
Happens-before
Can we spot any problem from this trace?
Answer: No, because of the happens-before relationship.

26. Happens-Before Approach
Input: a sequence of statements <s1, s2, …>
Output: true if there is a race condition
For any pair of statements s and t (assuming s is “earlier”)
if (s and t are from two different threads and
s and t access some shared memory and
either s modifies the memory or t does) {
if (there is no happens-before relation from s to t) {
report potential race condition; }

27. Example Trace Thread 1: obj1.methodA() Thread 2: obj2.methodB()
Can we spot any problem from this trace?
It depends on whether obj1 and obj2 are disjoint.

28. Example Trace Thread 1: obj1.methodA() Thread 2: obj2.methodB() A B
The question is: whether A and B are disjoint?

29. Define Happens-Before
Given a sequence of statements
<a, b, c, …>
If statement f and u (u is after f in the sequence) are from the same thread, f happens-before u.
If f sends a message and u receives the message, f happens-before u.
If f unlocks an object and u locks the same object, f happens-before u. …
If f happens before u and u happens-before y, f happens-before y.
Is this complete?

30. Subtle Happens-Before
Trace
Thread 1: count++
Thread 2: while (x <= 0) {
Thread 2: Thread.yield()
Thread 2: endwhile
Thread 1: x++
Thread 2: count++
Do these form a race condition?

31. Subtle Happens-Before
Trace
Thread 1: count++
Thread 2: while (x <= 0) {
Thread 2: Thread.yield()
Thread 2: endwhile
Thread 1: x++
Thread 2: count++

32. Happens-Before Algorithm
In order to get precise happens-before relation, we often need to analyze the whole program.
Race condition detection based on happens-before relation is very costly – no known efficient implementation so far.
Race condition detection based on happens-before relation is not exhaustive even with a perfect happens-before relation.

33. Exercise 1 Trace Thread 1: count++ Thread 1: lock(mu) Thread 1: x++
Thread 1: unlock(mu)
Thread 2: lock(mu)
Thread 2: x++
Thread 2: unlock(mu)
Thread 2: count++
Is there a race condition according to the happens-before approach?
Is there a race condition?

34. Lockset Algorithm Recall (rules for locks):
For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held.
The lockset algorithm is designed to check if the rules is properly implemented. If not, it reports potential race condition.

35. Lockset Algorithm For each shared variable x set lockset(x) = *;
Endfor
For each access to x by thread t
lockset(x) = {locks held by t at the time} intersect lockset(x)
if (lockset(x) is an empty set) {
report race condition; }
* is a special flag denoting the set of all possible locks.

36. Example Trace Locks held by thread 1 and 2 Lockset(count)
Thread 1: lock(mu1)
{} for thread 1; {} for 2
{mu1, mu2}
Thread 1: count++
{mu1} for thread 1; {} for 2
{mu1}
Thread 1: unlock(mu1)
Thread 2: lock(mu2)
Thread 2: count++
{} for thread 1; {mu2} for 2 {}
Thread 2: unlock(mu2)

37. Exercise 2 Trace Thread 1: count++ Thread 1: lock(mu) Thread 1: x++
Thread 1: unlock(mu)
Thread 2: lock(mu)
Thread 2: x++
Thread 2: unlock(mu)
Thread 2: count++
How does Lockset find the race in this example?

38. Patching Lockset Locking may not be necessary in some scenarios
Initialization: shared variables are frequently initialized without holding a lock
Read-share data: Some shared variables are written during initialization only and are read-only thereafter. These can be safely accessed without locks.
Read-write locks: Read-write locks allow multiple readers to access a shared variable, but allow only a single writer to do so.

39. Improving Lockset
For each shared variable, we only monitor its status according to the state machine and check for race condition only at state “Shared-Modified”

40. Improving Lockset
When a variable is in Shared-Modified state, we do the checking slightly differently.
For each read-access of x by thread t {
lockset(x) = {locks held by t at the time} intersect lockset(x)
if (lockset(x) is an empty set) {
report race condition; }
For each write-access of x by thread t {
lockset(x) = {locks held by t at the time in write mode} intersect lockset(x)

41. Example: On Variable count
Trace
Locks held
Write Locks held
Status
Lockset(count)
T1: lock(mu, R)
{} for T1 and T2
Virgin *
T1: x = count+1
{mu} for T1;
{} for T2
{} for T1;
T1: unlock(mu)
T2: lock(mu, W)
T2: count++
{mu} for T2
Exclusive
T2: unlock(mu)

42. Exercise 3 Trace Thread 1: count++ Thread 1: lock(mu) Thread 1: x++
Thread 1: unlock(mu)
Thread 2: lock(mu)
Thread 2: x++
Thread 2: unlock(mu)
Thread 2: count++
Draw the table for this example.

43. Efficiency of Lockset Implemented at the binary level
Maintain a status and lockset for each memory location
Performance penalty: applications with lockset implementation slows down 10 to 30 times.

44. Effectiveness False Alarms are produced in some scenarios.
Memory reuse: the same memory is reused for different purposes later (with different locking policy).
Private locks: user defines their locks without using the ones from the library
Benign races: Race condition is found but deemed to be benign.

45. False Alarm: Example
Which is worse for users: false alarms or missing some true races?

46. Hybrid Dynamic Data RACE Detection
O’Callahan et al. PPoPP’03
Hybrid Dynamic Data RACE Detection

47. Example class Main { int globalFlag; ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start(); …
synchronized (this) {
childThread.interrupt(); }
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main; }
void run() {
if (main.globalFlag == 1) …; …
main.childThread = null;
Is there a race condition?

48. Example: Lockset Algorithm
class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start(); …
synchronized (this) {
childThread.interrupt(); }
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main; }
void run() {
if (main.globalFlag == 1) …; …
main.childThread = null;
What would the lockset algorithm report?

49. Example: Lockset Algorithm
class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start(); …
synchronized (this) {
childThread.interrupt(); }
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main; }
void run() {
if (main.globalFlag == 1) …; …
main.childThread = null;
What would the (not improved) lockset algorithm report?
Two potential races: one globalFlag and the other on childThread.
The former is false alarm, why?

50. The Idea Race condition detection with Lockset
Define a limited easy-to-check
happens-before relation
Potential race conditions
Filter false alarms with the happens-before relation
Filtered race conditions

51. Exercise 4 What would the hybrid approach report? class Main {
int globalFlag;
ChildThread childThread;
void execute () {
globalFlag = 1;
childThread = new ChildThread(this);
childThread.start(); …
synchronized (this) {
childThread.interrupt(); }
class ChildThread extends Thread {
Main main;
ChildThread (Main main) {
this.main = main; }
void run() {
if (main.globalFlag == 1) …;
synchronized (main) {
main.childThread = null;
What would the hybrid approach report?

52. Race Directed Random testing of concurrent programs
Koushik Sen. PLDI’08
Race Directed Random testing of concurrent programs

53. Motivation Hybrid dynamic race detection:
39 out of 51 reported race conditions for tomcat are false alarms
Static race detection (from Stanford, 2006)
13 out of 19 reported race conditions for hedc (a software) are false alarms
Would you be happy to use these tools?

54. Motivation 00 Thread1 Thread2 r2 01 10 r1 r2 r1 1 1 02 11 20 i2 i1 i201 10 r1 r2 r1 1 1 02 11 20 i2 i1 i2 03 12 21 30 i1 2 2 13 22 31 w1 w2 w1 3 3 23 32
count=1 w2
Even if we are sure there is a race condition, users might not be convinced if you can’t show it! 33

55. Question Thread1 Thread 2 Thread k …
How about we show the problematic trace by testing the system many times? 11 21 k1 12 22 k2 … … … 1n 2n kn
How many possible traces are there?

56. Scheduling threads Scheduler
How can we control the scheduler so that it would show the error, if there is?

57. Proposal: PaceFuzzer
Use hybrid dynamic race detection to find potential race conditions, in the form of pairs of statements (s, t).
PaceFuzzer executes the program with a random scheduler
Control the scheduler such that it keeps delaying s and t
Report race condition if s and t can be executed by different thread next to each other

58. Example
What are the race conditions according to the hybrid algorithm?
Assume there are two test cases, one in which thread 1 runs to finish first and the other in which thread 2 runs to finish first.

59. Example Test 1: thread 1 runs first and then thread 2
Lockset based algorithm:
race condition between statement 1 and 10
race condition between statement 5 and 7
happen-before filtering
The race condition between 1 and 10 is removed as statement 4 happens-before 8

60. Exercise 5 Test 2: thread 2 runs first and then thread 1
Lockset based algorithm:
happen-before filtering:
What are the race conditions reported in this case?

61. Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.

62. Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.

63. Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.

64. Example: Case (1, 10)
Statement 1 is enabled, so delay it and let thread 2 go.

65. Example: Case (1, 10)
Thread 2 has finished; thread 1 proceeds to execute statement 1 and more

66. Example: Case (1, 10)
Report: No real race condition found! False alarm!

67. Example: Case (5, 7)
Statement 7 is enabled, so delay it and let thread 1 go.

68. Example: Case (5, 7)
Report: Real race condition identified!

69. The Algorithm For each pair of statements (s, t) { postponed = {}
while (some thread is enabled) {
randomly pick one enabled thread which is not in postponed;
if execute the picked thread would execute s or t {
if a thread in postponed is about to execute the other statement {
report race condition and terminate; }
else {
add the thread to postponed;
execute the picked thread for one step;
if (enabled == postponed) { remove one from postponed };
report deadlock if some thread is not finished;

70. Exercise 6
lock(L);
x = 1;
Apply the algorithm with this modified example for (2, 10).

71. Empirical Study

72. Conclusion There are proposals for detecting race conditions.
The algorithms/tools must work with large programs.
The algorithms/tools are often neither sound nor complete
Sound: a race condition found is always a real-one.
Complete: all race conditions are found.

73. Example Initially: x = y = 0 thread 1 { x = random.nextInt(1,100000);
count++; }
thread 2 {
if (x==3771) {
count++; }
Would any dynamic analysis approach be able to find the race condition?
And be sure it is not a false alarm?

74. Can We Do Better? Recall the rules
Update related state variables in a single atomic operation
For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held.
Every shared, mutable variable should be guarded by exactly one lock. Make it clear to maintainers which lock that is.
For every invariant that involves more than one variable, all the variables involved in that invariant must be guarded by the same lock. …
Can we do better using other rules?

75. Example Thread 2 Thread 1 lock(mu) lock(mu) x++; x++; unlock(mu) y++;
Assume that we have an invariant x==y.