1. Ivy
Eva Wu

2. Table of Content Ivy – Design, Architecture
Distributed Manager Algorithms
Performance, Problems, and Potential Improvements

3. Ivy First DSM system – Apollo Workstation
Implemented at the Yale University in mid-to-late 80’s
Implement multiprocessor cache coherency protocol in software
Single writer and multiple readers
Page management implementation
Centralized manager
Fixed distributed manager
Dynamic distributed manager
Provide a shared memory system across a group of workstations
Provide abstraction of 2 classes of memory: private and shared
Shared memory – easier to write parallel programs with than using message passing

4. Ivy Architecture Ownership of a page moves across nodes
Sequential consistency
Must invalidate before writing a page
Simulates FIFO
Sequential consistency – all other nodes/processes see the results of every memory operation performed by any processor in the same order
Provides uniprocessor behavior on multiprocessor

5. Granularity Size of unit transfer Ivy – 1Kbyte page for access
Advantage: large blocks – fewer page numbers of transfer (locality)
Disadvantage – false sharing
Ivy – 1Kbyte page for access
False sharing – when system participant attempts to periodically access data that will never be altered by another party, but that data shares a cache block with data that is altered, the caching protocol may force the first participant to reload the whole unit despite a lack of logical necessity
Sum_a – may need to continually re-read x from main memory

6. Centralized Manager Only manager knows all the copies
Contains a table (Info) with one entry for each page
Owner – most recent write access processor
Copy_set – list of all the processor that have copies of the page
Lock – for synchronizing requests
Each processor has a page table (PTable) – for accessibility
Access
Lock

7. Read Page fault for p1 in C C sends a read request to manager
Manager sends read forward to A; manager adds C to copy_set
A sends p1 to C; p1 in C is marked read-only
C sends read confirmation to manager
Copy_set – set of nodes with a copy of a piece of data
Last write is considered the “owner,” and has the current copy_set
2 messages for manager processor – one to owner, another from the owner
4 messages for nonmanager – one to manager, one to owner, one from owner, one for confirmation

8. Write Page fault for p1 on B B sends write request to manager
Manager sends invalidate to all processors in copy_set (C)
C sends invalidate confirm to manager
Manager clears copy_set and sends write forward to A
A sends p1 to B and clears access
B sends write confirmation to manager
Potential bottleneck/hotspot – all requests must go to manager
Page is not writeable without invalidation operation if there is a read copy
Invalidated all copies – send a message to each processor in the copy_set, wait for acks from everyone
Confirmation message – completion of a request, manager can give the page to someone else – synchronization

9. Eventcount
Process synchronization mechanism – based on shared virtual memory
Four primitive operations: init(), read(), await(value), and advance()
Atomic operation
Any process can use eventcount after initiailization
Eventcount operations are local when a page is received by a processor
Init() – initialized an eventcount
Read() – returns value of eventcount
Await() – suspends the calling process itself until the value of the eventcount reaches the value specified
Advance() – increments the values of the eventcount by one and wakes up awaiting processes

10. Improved Centralized Manager
The owner, instead of the manager, keeps the copy_set of a page
PTable: access, lock, and copy_set
Manager still answers where the page owner is
Copy_set is sent along with the data
Owner is responsible for invalidation
Decentralized synchronization – centralized manager no longer the hot spot
Eliminate confirmation operation to manager
Copy set field is valid only if the processor that holds the page table is the owner of the page
Might still have bottleneck – manager must respond to every page fault

11. Fixed Distributed Manager
Every processor has a predetermined set of pages to manage
One manager per processor
Manager is responsible for pages specified by fixed mapping function H
Page fault on p
Faulting processor asks processor H(p) where the true page owner is
Processor H(p) finds true page owner using centralized manager algorithm
Straightforward approach – distribute the pages evenly in a fixed manner to all processors
Hashing function H defined by H(p) = p mod N
More general definition – H(p) = (p/s) mod N
P – number of pages, N – number of processors, s – number of pages per segment

12. Broadcast Distributed Manager
No manager
PTable: access, lock, copy_set, and owner
Owner behaves similar to a manager and keeps the copy_set
Requesting processor sends a broadcast message
Disadvantage: all processes have to process each broadcast request
Each processor manages those pages it owns
Broadcast read request – true owner of the pages responds by adding processor P to the page’s copy set field and sending a copy of the page to P
Broadcast write request – true owner of the page gives up ownership and sends back the page and its copy set. The requesting processor invalidates all the copies.

13. Broadcast Distributed Manager Read
Add P1 to copy set and send copy of page 0
Broadcast P0 P1 P2 P3 P4
Request Page 0
Page 0

14. Broadcast Distributed Manager Write
Page 0 and its copy set
Broadcast P0 P1 P2 P3 P4
Write request
Page 0
Page 0

15. Dynamic Distributed Manager
Manager = owner
A page does not have a fixed owner or manager
Each process keeps tracks of the probable owner (probOwner)
Every host keeps track of the page ownership in its local page table PTable
Owner field is replaced with field: probOwner
If the processor is the true owner, it proceeds as in the centralized manager algorithm

16. probOwner Value either true owner or “probable” owner of the page
Page fault – sends a request to the processor in probOwner field
If correct, then proceeds as in centralized manager algorithm
If incorrect, then forward the message to the “probable” owner
Initially, all probOwners are set to a default processor
Updates when
Invalidation request
Relinquishes ownership
Forwards a page fault request
Job of the page fault handlers and their servers to maintain this field as the program runs
When forward a request, it does not need to send a reply to the requesting processor

17. Dynamic Distributed Manager Read
Request Read P3 P4 P0 P1 P2
probOwner
Page 0
Long link to find the true owner

18. Dynamic Distributed Broadcasts
Improved the dynamic distributed manager algorithm by enforcing a broadcast message
Announce the true owner after every M page faults
M steadily increases as number of processors get large
Program converges when M is very large

19. Dynamic Distributed Broadcasts Read
Current Owner
Request
Broadcast P0 P1 P2 P3 P4
probOwner
Page 0

20. Dynamic Distributed Copy Set
Copy set data to be stored as a tree
Root: owner
Bi-directional
Directed from root: copy_set
Directed from leaves: probOwner
Read fault: probOwner to the owner
Write fault: Invalidates all copies starting at owner and propagate to the copy_sets
Propagation of invalidation – “divide and conquer”
If balanced, O(log(m)) for m read copies
Read fault now only needs to find a single processor that holds a copy of the page – a lock is needed on processors having read copies of the page to synchronize sending copies of the page in the presence of other read or write faults

21. Double Fault Read first, then write – page fault twice
Solution – sequence numbers
Process can send its page sequence number to the owner. The owner then decide whether a transfer is needed
Only avoids transaction

22. Performance Works well when there is little sharing
Cannot handle false sharing
Sequential consistency required large amounts of communication
Ping-pong effect

23. Potential Improvements
Allow multiple writers by allowing certain users to keep private copies
Do not share the entire page to reduce false sharing

24. Questions?