Dynamic Verification of Cache Coherence Protocols Jason F. Cantin Mikko H. Lipasti James E. Smith

6/30/2001Workshop on Memory Performance Issues Introduction Multiprocessors are used for a variety commercial and mission-critical tasks Reliability is a growing concern Coherence is a fundamental feature of shared-memory MPs High design complexity Relatively low interconnect reliability

6/30/2001Workshop on Memory Performance Issues Introduction: Cache Coherence Protocols Notoriously difficult to design and verify Often conceptually simple, but with complex implementations for efficiency and handling special cases Multiple finite state machines operating concurrently

6/30/2001Workshop on Memory Performance Issues Introduction: A Simple Example MSI Protocol “Architected” State Invalid / Not Present Shared (readable) Modified (read/write) IS M Bus_RdX, Replace Read Write Bus_RdX, Replace Write Bus_Rd

6/30/2001Workshop on Memory Performance Issues M S Pend Rd I I Pend WB M Pend RdX S Pend RdX I Bus_RdX, Replace Read Bus_Av Replace Bus_Av Bus_Rd Write Bus_RdX Write Bus_RdX Introduction: Simple Example with a Bus MSI Protocol “Implementation” State Transient states for pending operations Arcs to satisfy requests while operations pending

6/30/2001Workshop on Memory Performance Issues Problem In practice, implementations can have dozens of states Atomic memory operations Split transaction buses Protocol optimizations Complexity grows exponentially with added states Random testing: Low Coverage Exhaustive testing: Too time consuming

6/30/2001Workshop on Memory Performance Issues Dynamic Verification Check the implementation at runtime It is easier to check a computation than to do the actual computation, provided there is a delay between the computation and the check (Rotenberg, AR-SMT) Simplified version of a processor implementation can be used for online verification (Austin, DIVA)

6/30/2001Workshop on Memory Performance Issues Dynamic Verification of Cache Coherence A distributed form of dynamic verification for multiprocessor memory systems Simplified version of protocol added to each node Maintains architected state Check completed transitions and actions against simple protocol Additional messages (assertions) sent between nodes to ensure coherence

6/30/2001Workshop on Memory Performance Issues Conceptual View for Superscalar Processors (DIVA) Single, centralized check processor Receives instructions serially in program order from implementation Physical registers Complex Execution Processor Check Processor Arch. registers Prediction Tables Committed results R.O.B. Arch. registers

6/30/2001Workshop on Memory Performance Issues Conceptual View for Coherence Distributed checking hardware Transitions received in parallel, in completion order Shared Logical Bus Shared Validation Bus Implementation Protocol Simple Protocol Completed Transitions

6/30/2001Workshop on Memory Performance Issues High Level Organization Cache Controller P Shared logical bus (addresses, data, control) Memory DV-CC Checker Validation bus (assertions to be checked)

6/30/2001Workshop on Memory Performance Issues Benefits Detects hardware faults Redundant computation Including intermittent network failures Detects design mistakes Checker is simple and easy to verify

6/30/2001Workshop on Memory Performance Issues Drawbacks Time is required for checking, but… May be overlapped with other activities Simple protocol requires fewer transitions Assertions consume bandwidth May need second bus / network Additional hardware But not much

6/30/2001Workshop on Memory Performance Issues DV for coherence in an SMP Architected state stored in a second tag array Transactions sent to the checker when architected state changes Address Initial State and Final States Input (Request, Snoop Responses, etc) Action (Send Data, Respond Shared, etc)

6/30/2001Workshop on Memory Performance Issues DV for coherence in an SMP (2) Checker compares the initial state of a transition against the architected state Final state and action recomputed and compared to implementation’s result Assertions broadcast to other nodes to check coherence and confirm completion of transactions Watchdog timer detects deadlock, livelock, and other omission failures

6/30/2001Workshop on Memory Performance Issues Init. stateFinal stateInputActionAddress Next State Logic =? Arch. Tag State Action Logic =? Error Detection / Diagnosis Transition From Implementation Protocol Update Tags Validation Bus Asst Send Buffer Checking a State Transition Watchdog timer

6/30/2001Workshop on Memory Performance Issues Arch. Tag State Error Detection / Diagnosis Watchdog timer Validation Bus Assert Recv Buffer OK Address Remote State Checking an Assertion

6/30/2001Workshop on Memory Performance Issues When to Broadcast Assertions For MSI: 1. I  S(readable copy loaded) 2. I  M(writeable copy loaded) 3. S  M(upgrade) 4. M  I(writeback) Note: The M  S transition results from remote reads, and doesn’t require an extra assertion. Replacements (S  I) are not considered here.

6/30/2001Workshop on Memory Performance Issues Preliminary Data (4-way SMP) Most memory references do not change cache state (checker need not have high bandwidth)

6/30/2001Workshop on Memory Performance Issues Preliminary Data (4-way SMP)

6/30/2001Workshop on Memory Performance Issues Future Work Performance impact for a real SMP protocol implementation In progress Directory-based protocols Dynamically verifying memory models Recovery Can stall to avoid error propagation Can write checkpoints periodically

6/30/2001Workshop on Memory Performance Issues In Summary Dynamic verification can be applied to multiprocessor systems (in a distributed manner) Improves fault-tolerance, and design verification may be relaxed More to come