Lecture 21 Synchronization

Directory Based Protocols All accesses to potentially shared memory locations are sent to all processors Not a very scalable proposition computation and communication Amdahl’s law costs of synchronization For correct operation: hardware: cache coherence software: synchronization hw and sw: memory consistency all three affect performance, too © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Where are the synchronization Variables Distinct Special Memory Locks in registers accessed by special instructions Fast access to a set of special registers, often via a separate bus Alliant FX/80, CRAY X-MP, and Sequent Balance SLIC (System Link and Interconnect) gates, and SGI MIPS-based machines. Often implemented with distributed copies updating each other via a separate synch bus Restricted to privileged users or allocated by OS via library functions May provide library functions to access a set of virtual virtual synch variables multiplexed on a small set of physical synch variables. This however, defeats the original goal of fast access. © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Where are the synchronization Variables (Cont.) Mapped Special memory Looks in registers mapped to user address space Sequent Balance ALM (Atomic Lock Memory) and Alliant FX/2800 Provides a limited number of physical synch variables managed the same way as virtual memory Virtual memory provides protection among unprivileged users Removes the need for library functions to access locks Often implemented as a special memory module accessed via the normal memory bus. © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Where are the synchronization Variables Normal Normal Memory Potentially all virtual memory could contain synch variables Locks in normal memory and indistinguishable from normal data manipulated with special atomic instructions Encore Multimax and Sequent Symmetry © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Where are the synchronization variables? Tagged Normal Memory Locks in tag fields of normal memory accessed by special synch instruction HEP full/empty bits and IBM 801 © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Example of an implicitly synchronized program Processor 0 Processor 1 A = 1; If (B == 0) { Critical sec. A = 0; } B = 1; If (A == 0){ Critical Sec. B = 0; } If Processor 0 reorders the write to A and the read from B, both processors can end up in the critical section at the same time. There is no local basis to know that write to A and read from B must be strictly ordered due to the needs of multiprocessor execution © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

How do caches react to synchronization Non-cached Use memory management to prevent synch variables from entering the cache Often implemented by placing synch variables in non-cacheable pages Bypass cache Does not require special OS and library functions to place synch variables into non-cacheable pages. Allows synch variables to enter caches if accessed by normal instructions. RMW bypasses the cache: it goes to the bus even if the synch variable is the local cache RMW invalidates address from all caches and does not place synch variable into the local cache. 68030, 88X00, i860, 80486, SPARC © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

How do caches react to synchronization Exclusive copy Read-invalidate bus transaction to acquire exclusive copy Synch performed locally on exclusive copy Problem with test-and-set ping-pong effects due to spin lock Sequent Symmetry and recent Encore Multimax © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

How do write buffers react to synchronization Write buffers for write-through caches Write buffers for write-back caches Write buffers for bus arbitration Invalidation/update buffers for snooping caches Invalidation/update buffers Cache often has only one access port shared between CPU and bus snooper Normally CPU has priority, invalidate requests are queued in a buffer to wait for a free cache cycle © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Sender Synchronization Flush all local write buffers before releasing a lock Ensure that all data generated before lock release are visible to the other processors The sender CPU could continue execution in its local cache while its write buffers are being flushed. However, its write buffers will not accept new entries until they are all emptied. In this case, a processor simply makes sure its local invalidation/update buffer is empty before acquiring a lock. Requires a special instruction to release a lock (rather than a normal memory write). E.g. release consistency in Stanford DASH © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Receiver Synchronization Flush all write buffers in the system when acquiring a lock Ensure that all potential supplier of data to be used after acquiring the lock have flushed out their data. The receiver cannot continue execution until all write buffers are empty. Should be avoided due to its performance penalty. © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

How does system bus support synchronization? Exclusive bus lock system bus locked out for the entire RMW transaction Split RMW system bus freed between read and write memory board locks the RMW location for the entire RMW transaction Requires intelligent memory board controller. Need to decide the granularity of locking: whole board, individual locations, somewhere in between. E.g., in VAX 6200, a small associative store is used to record the locked locations in each memory board. © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

How does system bus support synchronization? Exchange with memory out-of-order RMW RMW addr → New data → old data eliminates one address cycle cache coherence protocol modified if synch variables allowed in cache for update protocols Load-store-fixed implied write eliminate one more data transmission © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois

Where is the synchronization done? Lock by holding the bus, CPU does computation Locus: processor (where the computation is done) Domain: bus and single ported memory (exclusivity) Bypass local cache Invalidate all other caches Other processors still have access to their local caches Lock the memory board, memory board does computation Locus: memory Domain: memory board Only one Read-modify-write request Invalidate all caches Intelligence on memory board No locking on bus © S. J. Patel, 2001 ECE 412, Fall 2001, University of Illinois