1. Using Formal Specifications to Monitor and Guide Simulation: Verifying the Cache Coherence Engine of the Alpha 21364 Microprocessor Serdar Tasiran Systems Research Center, HP Labs (formerly Compaq) Yuan Yu (Microsoft Research, formerly Compaq) Brannon Batson, Scott Kreider, (Intel, formerly Compaq)

2. The Problem Given A complex protocol specified formally A hardware implementation Verify that All executions of implementation are consistent with protocol spec An “implementation verification” problem Properties of protocol verified separately

3. Alpha 21364 (EV7) System Block Diagram 1 M IO 0 M 2 M 3 M 5 M 4 M 6 M 7 M 9 M 8 M 10 M IO 11 M IO Distributed shared memory Up to 256 processors, 32 GB per processor

4. EV7 Cache Coherence: Implementation in hardware (~20K lines of HDL code) C Z1 mem system data buffers R core & cache Z0 mem EV7 SVDB, FB0, FB1 Cache coherence protocol spec (~3K lines of TLA, written by architects and us) Executable Not list of properties Does the hardware implement the protocol spec correctly?

5. Why is the problem difficult? Thousands of state variables per processor Parallelism, deep pipelining, speculative execution, redundancy,... Need 4+ processors for an interesting system Out of reach of automatic formal methods Limited to several hundred state variables Decomposition methods difficult for non-specialists, large design teams Complete verification of hardware against protocol not practical Simulation only viable approach Even simulation is expensive

6. Validation Guided by Formal Spec Coverage Simulation Correctness checking using formal spec Automated input generation Coverage analysis on formal spec

7. Validation Guided by Formal Spec Coverage Simulation Correctness checking using formal spec “Automated” input generation Coverage analysis on formal spec 1

8. Contributions: Spec written in formal language Properties of spec can be verified formally Model checker checks properties satisfied during simulation:  More reliable than hand-written code  More flexible than automatically generated assertions Must relate Implementation state  Spec state Devised two-phase mapping approach Applicable to complex designs by non-specialists 1: Formal Spec + Model Checker as Monitor

9. Validation Guided by Formal Spec Coverage Simulation Correctness checking using formal spec “Automated” input generation Coverage analysis on formal spec 2

10. Contributions 2: Coverage analysis and input generation using formal protocol spec Formal spec encapsulates design intent Full coverage = All scenarios exercised Spec at same level of abstraction as existing coverage data Model checker used to Measure coverage, detect gaps Generate simulation input traces to reach coverage holes Determine if unexercised scenario actually possible

11. Outline The cache-coherence protocol The EV7 cache-coherence engine Spec-guided simulation Conclusion

12. The EV7 Coherence Protocol Distributed shared memory Each address belongs to a “home node” but may be in other caches Directory-based protocol Cache states: Modified (Dirty), Exclusive (Clean), Shared, Invalid Directory states: Local, Shared, Exclusive, Incoherent Directory distributed with memory at each node CPU requests that miss in local caches are sent to home node Home node may forward request to other nodes Directory In Flight Table (DIFT) keeps track of pending requests 1 M IO 0 M 2 M 3 M 5 M 4 M 6 M 7 M 9 M 8 M 10 M IO 11 M IO

13. Example: write, remote sharers Conditions: home is remote, directory state is shared Actions: read-exclusive request to home home sends invalidation requests to sharers, sends data back to requester with invalidation count (early exclusive reply) sharing nodes reply to requester with invalidation acknowledgements requester proceeds when data arrives, but must “stall” incoming requests and potential writeback of line until all InvalAcks are received ReadModSharedInv HRS BlkExclusive InvalAck SS

14. TLA Description for Protocol Temporal Logic of Actions [Leslie Lamport] Formal language for writing high-level specs of concurrent, reactive systems Very expressive. Incorporates First-order logic, set theory, temporal operators Sets, queues, records, tuples, … EV7 protocol description is a TLA formula

15. One Protocol Action ReadModSharedInv HRS S S BlkExclusiveCnt Preconditions Messages sent and state variables updated

16. Outline The cache-coherence protocol The EV7 cache-coherence engine Spec-guided simulation Conclusion

17. Alpha 21364 Chip Block Diagram L2 Data Array L2 Tag Array Core L2 Cache Controller L1 Cache Router Memory Controller 0 Memory Controller 1 Data Buffers Data Address & Control IPx4 RDRAM I/O

18. Directory in Flight Table (DIFT) Cache State Lookup (from cache controller) Directory State from Memory CoherenceEngine New directory to Memory to Memory Forwards and responses to other CPUs DIFT Request Front end of memory controller Tracks up to 32 in-flight transactions

19. DIFT block diagram From CBox To RBox zc_dft_acc zc_dft_ros Address File (zx_dft_af*) addr maf pid vdba vdbv fb directory Event File cmd src ack vic lpr From CBoxFrom Back End New Request Decode Proto File rd fwd rsp inv wr vcp akp Protocol Logic (zx_dft_plc) Next DIFT State New Directory Logic (zx_dft_plt) New Directory Output (aod) To Back End Issue Logic (zx_dft_isp*) Grant logic to everywhere Address Output Logic (ao*) Messages to the Ring Rd/Wr Requests to the Zbox middle DIFT Free List DIFT Conflict Array

20. Outline The cache-coherence protocol The EV7 cache-coherence engine Spec-guided simulation Conclusion

21. Validation Guided by Formal Spec Coverage Simulation Correctness checking using formal spec “Automated” input generation Coverage analysis on formal spec 1

22. Formal Spec as Simulation Monitor Spec State-Space f abs Implementation State-Space f abs : Abstraction mapping f abs Model checker (TLC) checks if transition is legal

23. Refinement Mapping Issues Protocol transactions look instantaneous at spec level but in the implementation they happen over many clock cycles, interleaved with other actions Want designers, developers to write the mapping Implementation has parallelism, pipelining, speculative execution, redundancy AND many processors Burch, Dill style “flushing” not practical Action i1 (p j1, a k1 ) Action i2 (p j2, a k2 ) Action i3 (p j3, a k3 ) time

24. The Refinement Mapping Preconditions Messages sent and state variables updated Action i1 (p j1, a k1 ) Action i2 (p j2, a k2 ) Action i3 (p j3, a k3 ) time

25. A two-phase recipe for refinement mappings Collect “tokens”, e.g., cache state looked-up, invalidate message sent, directory state written 1.Implementation state  Intermediate state Determine interfaces: write, read ports state machines that relate to protocol state Watch Messages crossing interfaces Updates to state machines Record in intermediate state C Z1 mem system data buffers R core & cache Z0 mem IO EV7 Action i3 (p j3, a k3 ) time Record in intermediate state for processor p j3 address a k3

26. A two-phase recipe for refinement mappings All tokens collected. “Fire” action. 2. Intermediate state  Protocol spec state For each protocol transaction, check when  All preconditions hold  All state updates happen, all required messages sent Update abstract state Action i3 (p j3, a k3 ) time Preconditions and implementation state updates related to Action i3 (p j3, a k3 ) Abstract state gets updated here

27. A two-phase recipe for refinement mappings 1.Implementation state  Intermediate state Hardware signal transitions  Protocol events (tokens) 2.Intermediate state  Protocol spec state Protocol event sequences  Protocol transactions (actions) Component implementers can write step 1 System architects can put together step 2 Distinguishes protocol errors from component implementation errors Well-defined, clean interface Easier to keep implementation and spec consistent throughout design process Modular description makes reasoning easier

28. Validation Guided by Formal Spec Coverage Simulation Correctness checking using formal spec “Automated” input generation Coverage analysis on formal spec 2

29. Formal Spec as Coverage Model Spec State-Space f abs Implementation State-Space f abs : Abstraction mapping f abs Model checker (TLC) records visited states

30. Model Checker Tracks and Improves Coverage Spec State-Space Non-covered state Path generated by model checker Identify parts of spec not exercised by simulation Path in spec state space = unexamined scenario Problems: Spec has too many states Not feasible to track coverage, generate paths for each Want to explore “qualitatively distinct” scenarios

31. Coverage Metric Defined on Spec States Coverage State-Space Spec State-Space c0c0 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 c8c8

32. Coverage Metric Examples All possible directory state transitions: Invalid  Exclusive  Shared  SharedMask All legal combinations of Request type Source of request (Local or remote) Cache state Directory state All possible transitions of some protocol state field in the DIFT WaitingForAck  WaitingForVictim, …

33. Conclusions Novel approach uses formal spec and model checker to monitor simulation to identify coverage gaps to guide input generation Found valuable by architects and verification engineers EV7 verification engineers want to use model checker to analyze their coverage data EV8 design started with formal specification first! First attempt at verifying industrial implementation of this scale and complexity against formal spec

34. Action i1 (p j1, a k1 ) Action i2 (p j2, a k2 ) Action i3 (p j3, a k3 ) time Action i3 (p j3, a k3 ) time Preconditions and implementation state updates related to Action i3 (p j3, a k3 ) Abstract state gets updated here

35. ReadModSharedInv HRS S S BlkExclusiveCnt

36. 1 M IO 0 M 2 M 3 M 5 M 4 M 6 M 7 M 9 M 8 M 10 M IO 11 M IO