John M. Mellor-Crummey Algorithms for Scalable Synchronization on Shared- Memory Multiprocessors Joseph Garvey & Joshua San Miguel Michael L. Scott

Dance Hall Machines?

Various insns known as fetch_and_ф insns: test_and_set, fetch_and_store, fetch_and_add, compare_and_swap Some can be used to simulate others but often with overhead Some lock types require a particular primitive to be implemented or to be implemented efficiently Atomic Instructions

type lock = (unlocked, locked) procedure acquire_lock (lock *L) while test_and_set (L) == locked ; procedure release_lock (lock *L) *L = unlocked Test_and_set: Basic

$ P $ P $ P Memory

type lock = (unlocked, locked) procedure acquire_lock (lock *L) while 1 if *L == unlocked if test_and_set (L) == unlocked return procedure release_lock (lock *L) *L = unlocked Test_and_set: test_and_test_and_set

$ P $ P $ P Memory

type lock = (unlocked, locked) procedure acquire_lock (lock *L) delay = 1 while test_and_set (L) == locked pause (delay) delay = delay * 2 procedure release_lock (lock *L) *L = unlocked Test_and_set: test_and_set with backoff

$ P $ P $ P Memory

type lock = record next_ticket = 0 now_serving = 0 procedure acquire_lock (lock *L) my_ticket = fetch_and_increment(L->next_ticket) while 1 if L->now_serving == my_ticket return procedure release_lock (lock *L) L->now_serving = L->now_serving + 1 Ticket Lock

Memory next_ticket now_serving $ P my_ticket $ P $ P

type lock = record slots = array [0…numprocs – 1] of (has_lock, must_wait) next_slot = 0 procedure acquire_lock (lock *L) my_place = fetch_and_increment (L->next_slot) // Various modulo work to handle overflow while L->slots[my_place] == must_wait ; L->slots[my_place] = must_wait procedure release_lock (lock *L) L->slots[my_place + 1] = has_lock Array-Based Queuing Locks

Memory next_slot slots $ P my_place $ P $ P

type qnode = record qnode *next bool locked type lock = qnode* procedure acquire_lock (lock *L, qnode *I) I->next = Null qnode *predecessor = fetch_and_store (L, I) if predecessor != Null I->locked = true predecessor->next = I while I->locked ; MCS Locks procedure release_lock (lock *L, qnode *I) if I->next == Null if compare_and_swap (L, I, Null) return while I->next == Null ; I->next->locked = false

MCS Locks L1-R 2-B 3-B 2-R 3-R3-E 4-B 5-B 4-R procedure release_lock (lock *L, qnode *I) if I->next == Null if compare_and_swap (L, I, Null) return while I->next == Null ; I->next->locked = false

MCS Locks Memory lock qnodes $ P lock next locked next locked next locked $ P lock next locked next locked next locked $ P lock next locked next locked next locked

Results: Scalability – Distributed Memory Architecture

Results: Scalability – Cache Coherent Architecture

Butterfly’s atomic insns are very expensive Butterfly can’t handle 24-bit pointers Results: Single Processor Lock/Release Time Times are in μsTest_and_setTicketAnderson (Queue)MCS Butterfly (Distributed) Symmetry (Cache coherent) 7.0NA

Results: Network Congestion Busy-wait LockIncrease in Network Latency Measured From Lock NodeIdle Node test_and_set1420%96% test_and_set w/ linear backoff882%67% test_and_set w/ exp. backoff32%4% ticket992%97% ticket w/ prop backoff53%8% Anderson75%67% MCS4%2%

Atomic insns >> normal insns && 1 processor latency is very important  don’t use MCS If processes might be preempted  test_and_set with exponential backoff Which lock should I use? fetch_and_store supported? fetch_and_increment supported? Yes No test_and_set w/ exp backoff Ticket MCS YesNo

Centralized Barrier P0 P1 P2 P 

Software Combining Tree Barrier P0 P1 P2 P    P0 P1 P2 P3

 Tournament Barrier P0 P1 P2 P3      P0P1P2P3 W C L W L L

Dissemination Barrier P0 P1 P2 P3         P0P1P2P3

New Tree-Based Barrier P0 P1 P2 P     3

Summary BarrierSpaceWakeupLocal SpinningNetwork Txns CentralizedO(1)broadcastnoO(p) or O(∞) Software Combining TreeO(p)treenoO(p × fan-in) or O(∞) TournamentO(plogp)treeyesO(p) DisseminationO(plogp)noneyesO(plogp) New Tree-BasedO(p)treeyes2p - 2

Results – Distributed Shared Memory BarrierSpaceWakeupLocal SpinningNetwork Txns CentralizedO(1)broadcastnoO(p) or O(∞) Software Combining TreeO(p)treenoO(p × fan-in) or O(∞) TournamentO(plogp)treeyesO(p) DisseminationO(plogp)noneyesO(plogp) New Tree-BasedO(p)treeyes2p - 2

Results – Broadcast-Based Cache-Coherent BarrierSpaceWakeupLocal SpinningNetwork Txns CentralizedO(1)broadcastnoO(p) or O(∞) Software Combining TreeO(p)treenoO(p × fan-in) or O(∞) TournamentO(plogp)treeyesO(p) DisseminationO(plogp)noneyesO(plogp) New Tree-BasedO(p)treeyes2p - 2

Results – Local vs. Remote Spinning BarrierNetwork Latency (local)Network Latency (remote) New Tree-Based10% increase124% increase Dissemination18% increase117% increase

Barrier Decision Tree Multiprocessor? Dissemination Barrier Centralized Barrier New Tree-Based Barrier (tree wakeup) New Tree-Based Barrier (central wakeup) Distributed Shared Memory Broadcast-Based Cache-Coherent

No dance hall No need for complicated hardware synch Need a full set of fetch_and_ф Architectural Recommendations