1. Distributed Shared Memory (part 1)

2. Distributed Shared Memory (DSM) mem0 proc0 mem1 proc1 mem2 proc2 memN procN network... shared memory

3. Shared memory programming Standard – pthread synchronizations –Barriers –Locks –Semaphores

4. Sequential SOR for some number of timesteps/iterations { for (i=0; i<n; i++ ) for( j=1, j<n, j++ ) temp[i][j] = 0.25 * ( grid[i-1][j] + grid[i+1][j] grid[i][j-1] + grid[i][j+1] ); for( i=0; i<n; i++ ) for( j=1; j<n; j++ ) grid[i][j] = temp[i][j]; }

5. Parallel SOR with Barriers (1 of 2) void* sor (void* arg) { int slice = (int)arg; int from = (slice * (n-1))/p + 1; int to = ((slice+1) * (n-1))/p + 1; for some number of iterations { … } }

6. Parallel SOR with Barriers (2 of 2) for (i=from; i<to; i++) for (j=1; j<n; j++) temp[i][j] = 0.25 * (grid[i-1][j] + grid[i+1][j] + grid[i][j-1] + grid[i][j+1]); barrier(); for (i=from; i<to; i++) for (j=1; j<n; j++) grid[i][j]=temp[i][j]; barrier();

7. Differences between SMP and Software DSM Delay: tradeoffs, such as block size Software => traps: cost of read/write misses Goals of caches: multiprocessor = performance, dist. system = transparency bus vs. long networks: reliance on serialization and broadcast.

8. Consequent differences in protocols and applications Bigger block size –Cost amortization, higher hit ratio for larger blocks? –Reduced overhead But therefore... –Migration vs. Replication –False sharing increases DSM protocol more complex: Must handle lost, corrupted, and out-of-order packets Above, coupled with cost of traps, => SDSM consistency cost much higher!

9. Results of high consistency costs Manage sharing more carefully Align data to page boundaries

10. Consistency Models Sequential Consistency –All processors observe the same order –Must correspond to some serial order –Only ordering constraint is that reads/writes of P1 appear in the same order, but no restrictions on relative ordering between processors.

11. Common consistency protocols Write update –Multicast update to all replicas Write invalidate –Invalidate cached copies in p2, p3 –Cache miss if p2/p3 access X Valid data from other cache

12. Conventional Implementation As proposed by Li & Hudak, TOCS ‘86. Use virtual memory to implement sharing. Shared memory divided up by virtual memory pages. Use single-writer, multiple-reader write- invalidate coherence protocol. Keep pages in one of three states: –invalid, read-only, read-write

13. Example proc0proc1proc2procN shared memory

14. Example: Read Access Hit proc0proc1proc2procN read

15. Example: Write Access Hit proc0proc1proc2procN write

16. Example: Read Access Miss proc0proc1proc2procN read

17. Example: Read Fault proc0proc1proc2procN read fault

18. Example: Replication on Read proc0proc1proc2procN read

19. Example: Write Access Miss proc0proc1proc2procN write

20. Example: Write Fault proc0proc1proc2procN write fault

21. Example: Write Invalidation proc0proc1proc2procN write

22. Example: Write Access to Read-Only proc0proc1proc2procN write

23. Example: Write Fault proc0proc1proc2procN write fault

24. Example: Write Invalidation proc0proc1proc2procN write

25. How to Remember Locations? Broadcast on miss (as in SMP). Static home. Dynamic home or owner.

26. Ownership and Owner Location Owner is the last writer. Owner maintains copyset. Every processor maintains probable owner (not always the real owner).

27. Ownership Location Every read or write miss is sent to (local) probable owner. If owner, handle appropriately, else forward to probable owner.

28. Ownership Modification If write miss, new writer becomes owner, and all forwarders set probable owner to requester. If read miss, set probable owner to responding processor.

29. Example Initially, owner(page0) = p0, and probable owner(page0) = p0 everywhere. Write miss by p1, sends message to its probable owner (p0), handled there, new owner = p1, probable owner(0) on p0 = 1. Read miss by p2, sends message to probable owner (p0), forwarded to probable owner (p1), handled there, probable owner(0) on p2 becomes p1.

30. Implement synchronizations Use messages to implement synchronizations

31. Barriers Designate one processor as barrier manager. When a process waits at a barrier, it sends an arrival message to the barrier manager and waits. When barrier manager has received all messages, it sends a departure message to all processes.

32. Locks Designate one process as the lock manager for a particular lock. When a process acquires a lock, it sends an acquire message to the manager and waits. Manager forwards message to last acquirer. If lock free, send lock grant message. If lock held, hold on to request until free, and then send lock grant message.

33. Problem: False Sharing Concurrent access to different data within the same consistency unit. With page as consistency unit, lots of opportunity for false sharing. Two flavors: –read-write –write-write

34. Read-Write False Sharing x y

35. Read-Write False Sharing (Cont.) w(x) r(y) r(x) synch w(x)

36. Read-Write False Sharing (Cont.) w(x) r(y) r(x) synch w(x)

37. Write-Write False Sharing w(x) w(y) r(x) synch w(x)

38. Summary Software shared memory on distributed memory hardware. –Uses virtual memory. Home migration to improve locality –important because of high latencies. Sequential consistency suffers from false sharing