1. Barrier Synchronization
Nir Shavit
Multiprocessor Synchronization
Spring 2003

2. Ideal Parallel Computation1 1 1
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

3. Ideal Parallel Computation2 2 2 1 1 1
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

4. Real-Life Parallel Computation
zzz… 1 1
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

5. Real-Life Parallel Computation2
zzz… 1 1
Uh, oh
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

6. Barrier Synchronization
barrier
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

7. Barrier Synchronization1 1 1
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

8. Barrier Synchronization
Until every thread has left here
No thread enters here
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

9. Why Do We Care? Mostly of interest to Elsewhere
Scientific & numeric computation
Elsewhere
Garbage collection
Rare in systems programming
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

10. Barriers public class Barrier { int count; int size;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

11. Number threads participating
Barriers
public class Barrier {
int size;
int count;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Number threads participating
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

12. Number threads not yet arrived
Barriers
public class Barrier {
int size;
int count;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Number threads not yet arrived
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

13. Barriers public class Barrier { int size; int count; Initialization
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Initialization
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

14. Barriers public class Barrier { int size; int count; The method
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
The method
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

15. If I’m last, reset everything
Barriers
public class Barrier {
int size;
int count;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
If I’m last, reset everything
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

16. Wait for rest to catch up
Barriers
public class Barrier {
int size;
int count;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Wait for rest to catch up
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

17. What’s wrong with this protocol?
Barriers
public class Barrier {
int count;
int size;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
What’s wrong with this protocol?
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

18. Reuse Barrier b = new Barrier(n); while ( mumble() ) { work();
b.await() }
Do work
repeat
synchronize
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

19. Waiting for Phase 1 to finish
Barriers
public class Barrier {
int count;
int size;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Waiting for Phase 1 to finish
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

20. Waiting for Phase 1 to finish
Barriers
public class Barrier {
int count;
int size;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Phase 1 is so over
Waiting for Phase 1 to finish
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

21. Barriers public class Barrier { int count; int size;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchDec()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Prepare for phase 2
ZZZZZ….
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

22. Waiting for Phase 2 to finish Waiting for Phase 1 to finish
Barriers
public class Barrier {
int count;
int size;
public Barrier(int n){
this.size = this.count = n; }
public void await() {
if (count.FetchInc()==1) {
this.count = this.size;
} else {
while (this.count != 0) {}
}}}}
Waiting for Phase 2 to finish
Waiting for Phase 1 to finish
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

23. Basic Problem One thread “wraps around” to start phase 2
While another thread is still waiting for phase 1
One solution:
Always use two barriers
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

24. Sense-Reversing Barriers
public class Barrier {
int count, size;
boolean sense = true;
public void await(boolean mySense) {
if (count.FetchDec()==1) {
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

25. Sense-Reversing Barriers
public class Barrier {
int count, size;
boolean sense = true;
public void await(boolean mySense) {
if (count.FetchDec()==1) {
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Completed odd or even-numbered phase?
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

26. Sense-Reversing Barriers
public class Barrier {
int count, size;
boolean sense = true;
public void await(boolean mySense) {
if (count.FetchDec()==1) {
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Thread working on odd or even-numbered phase?
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

27. Sense-Reversing Barriers
public class Barrier {
int count, size;
boolean sense = true;
public void await(boolean mySense) {
if (count.FetchDec()==1) {
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
If I’m last, reverse the sense on the way out
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

28. Sense-Reversing Barriers
public class Barrier {
int count, size;
boolean sense = true;
public void await(boolean mySense) {
if (count.FetchDec()==1) {
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Otherwise, wait for sense to flip
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

29. Combining Tree Barriers
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

30. Combining Tree Barriers
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

31. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense) {
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

32. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense) {
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Parent barrier in tree
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

33. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense) {
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Thread working on odd or even-numbered phase?
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

34. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense) {
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Am I last?
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

35. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense){
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Wait on parent barrier, if any
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

36. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense){
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Prepare for next phase
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

37. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense) {
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
Wake up others at this level
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

38. Combining Tree Barrier
public class CBarrier {
int count, size;
CBarrier parent;
public void await(boolean mySense) {
if (this.count.FetchDec()==1){
if (this.parent != null)
this.parent.await();
this.count = this.size;
this.sense = mySense
} else {
while (this.sense != mySense) {}
}}}}
I’m not last, wait for release
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

39. Combining Tree Barrier
No sequential bottleneck
FetchDec calls proceed in parallel
Low memory contention
Same reason
Cache behavior
Local spinning on bus-based architecture
Not so good for distributed
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

40. Remarks Everyone spins on sense field Sequential bottleneck
Local spinning on bus-based (good)
Network hot-spot on distributed architecture (bad)
Sequential bottleneck
Sequence of FetchInc() calls
Not really scalable
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

41. Tournament Tree Barrier
If tree nodes have fan-in 2
Don’t need to call FetchDec()
Winner chosen statically
At level i
If i-th bit of id is zero, move up
Otherwise keep back
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

42. Tournament Tree Barriers
winner
loser
winner
loser
winner
loser
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

43. Tournament Tree Barriers
All flags blue
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

44. Tournament Tree Barriers
Loser thread sets winner’s flag
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

45. Tournament Tree Barriers
Loser spins on own flag
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

46. Tournament Tree Barriers
Winner spins on own flag
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

47. Tournament Tree Barriers
Winner sees own flag, moves up, spins
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

48. Tournament Tree Barriers
Bingo!
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

49. Tournament Tree Barriers
Sense-reversing: next time use blue flags
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

50. Tournament Barrier class TBarrier { boolean flag; TBarrier partner;
TBarrier parent;
boolean top; … }
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

51. Notifications delivered here
Tournament Barrier
class TBarrier {
boolean flag;
TBarrier partner;
TBarrier parent;
boolean top; … }
Notifications delivered here
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

52. Other thead at same level
Tournament Barrier
class TBarrier {
boolean flag;
TBarrier partner;
TBarrier parent;
boolean top; … }
Other thead at same level
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

53. Parent (winner) or null (loser)
Tournament Barrier
class TBarrier {
boolean flag;
TBarrier partner;
TBarrier parent;
boolean top; … }
Parent (winner) or null (loser)
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

54. Tournament Barrier Am I the root? class TBarrier { boolean flag;
TBarrier partner;
TBarrier parent;
boolean top; … }
Am I the root?
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

55. Tournament Barrier void await(boolean mySense) { if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

56. Tournament Barrier The root, c’est moi
void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
The root, c’est moi
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

57. Tournament Barrier I am already a winner void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
I am already a winner
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

58. Wait for partner to show up
Tournament Barrier
void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
Wait for partner to show up
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

59. Tournament Barrier Synchronize upstairs void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
Synchronize upstairs
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

60. Tournament Barrier Notify partner void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
Notify partner
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

61. My thread is such a loser
Tournament Barrier
void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
My thread is such a loser
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

62. Tournament Barrier Tell partner I’m here void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
Tell partner I’m here
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

63. Wait for partner to release me
Tournament Barrier
void await(boolean mySense) {
if (this.top) {
return;
} else if (this.parent != null) {
while (this.flag[round] != mySense) {};
this.parent.await(mySense);
this.partner.flag[round] = mysense;
} else {
this.partner.flag[round] = mySense;
}}}
Wait for partner to release me
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

64. Remarks No need for read-modify-write calls
Each thread spins on fixed location
Good for bus-based architectures
Good for distributed architectures
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

65. Dissemination Barrier
At round i
Thread A notifies thread A+2i (mod n)
Requires log n rounds
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

66. Dissemination Barrier+1 +2 +4
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

67. Dissemination Details
Use two copies of fields
Avoids interference
Use sense-reversing
Avoid reinitializing fields
Thread Arguments
Parity of round
Sense (flips when round becomes 0)
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

68. Dissemination Barrier
public class DBarrier {
private boolean flags[2][logn];
private DBarrier partners[2][logn]; }
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

69. Dissemination Barrier
public class DBarrier {
private boolean flags[2][logn];
private DBarrier partners[2][logn]; }
Use two copies to avoid interference
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

70. Dissemination Barrier
public class DBarrier {
private boolean flags[2][logn];
private DBarrier partners[2][logn]; }
One element per round
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

71. Dissemination Barrier
public class DBarrier {
private boolean flags[2][logn];
private DBarrier partners[2][logn]; }
Notify me here
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

72. Dissemination Barrier
public class DBarrier {
private boolean flags[2][logn];
private DBarrier partners[2][logn]; }
Barrier for thread A + 2i
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

73. Dissemination Barrier
void await(int parity, boolean mySense) {
for (int i=0; i<logn; i++) {
this.partner[parity].flags[i] = mySense;
while (this.flags[i] != mySense) {} }
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

74. Dissemination Barrier
void await(int parity, boolean mySense) {
for (int i=0; i<logn; i++) {
this.partner[parity].flags[i] = mySense;
while (this.flags[i] != mySense) {} }
Parity of current phase
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

75. Dissemination Barrier
void await(int parity, boolean mySense) {
for (int i=0; i<logn; i++) {
this.partner[parity].flags[i] = mySense;
while (this.flags[i] != mySense) {} }
Flips when parity becomes 0
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

76. Dissemination Barrier
void await(int parity, boolean mySense) {
for (int i=0; i<logn; i++) {
this.partner[parity].flags[i] = mySense;
while (this.flags[i] != mySense) {} }
Notify thread A+2i
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

77. Dissemination Barrier
void await(int parity, boolean mySense) {
for (int i=0; i<logn; i++) {
this.partner[parity].flags[i] = mySense;
while (this.flags[i] != mySense) {} }
Wait for thread A-2i
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

78. Remarks Every thread spins in the same place
Good for distributed implementations
Works even if n not a power of 2
4-Feb-19
M. Herlihy & N. Shavit (c) 2003

79. Clip Art
4-Feb-19
M. Herlihy & N. Shavit (c) 2003