1. Dynamic Verification of Sequential Consistency
Albert Meixner Daniel J. Sorin
Dept. of Computer Dept. of Electrical and
Science Computer Engineering
Duke University Duke University

2. Introduction Multithreaded systems becoming ubiquitous
Commercial workloads rely heavily on parallel machines
Reliability and availability are crucial
Backward Error Recovery can provide high availability
Recover to known good state upon error
But can only recover from errors detected in time
Memory system is of special interest
Complex – Many components, large transistor count
Numerous error hazards

3. Memory System Error Detection
Must cover all memory system components
DRAMs, caches, controllers, interconnect, and write buffers
Mechanisms for individual components exist
Storage structures: ECC
Interconnect: checksums, sequence numbering
Cache and memory controllers: replication
Adding detection to all components is hard
Complicates design of every component
Requires good intuition of interactions and possible errors
 Want comprehensive, end-to-end error detection

4. Dynamic Verification Dynamic verification
Correct system operation constantly monitored at runtime
End-to-end scheme
Detects transient errors, design bugs, and manufacturing errors
Differs from statically verifying that design is bug-free
High level invariants are checked, instead of individual components
Simplified design of system components
Can detect any low-level error that violates invariant

5. Memory Consistency Memory consistency model
Formal specification of memory system behavior in a multithreaded system
Defines order in which memory accesses from different CPUs can become globally visible
Many consistency models exist, we focus on one
Verifying memory consistency = Verifying correctness of the memory system
Ideal invariant for dynamic verification

6. Sequential Consistency (SC)
Requires appearance of total global order of all loads and stores in system
Each load must receive value of most recent store in total order to the same address
Program order of all processors is preserved in total order
SC is most intuitive consistency model
Good for programmers
Speculation can make SC almost as fast as more relaxed models
Our contribution: Dynamic Verification of Sequential Consistency (DVSC)

7. Outline
Introduction
DVSC-Direct
DVSC-Indirect
Results
Conclusion

8. DVSC-Direct Program Order Program Order Global Order CPU 1 t=1.1
LD A→1
t=2.1
ST B←2
t=3.1
LD A→2
LD A→1
ST B←2
LD A→2
CPU 2
Program Order
t=1.2
LD C→1
t=2.2
ST A←2
t=3.2
LD C→1
LD C→1
ST A←2
LD C→1
Global Order
Verifier
LD A→2
ST B←2
LD C→1
LD C→1
ST A←2
LD A→1

9. DVSC-Indirect: Idea
Verify conditions sufficient for Sequential Consistency
In-order performance of memory operations
Cache coherence
Conditions formally defined and proven by Plakal et al. [SPAA 1998]
Two mechanisms
On-chip checker for in-order performance
Distributed checker for cache coherence

10. In-Order Performance Verification
A load of block B receives the value of…
…the most recent local store to B or most recent global store to B performed after all local stores
Trivially observed on in-order processor with coherent caches
Modern processors execute out-of-order
Results of ooo-execution are considered speculative until in-order re-execution and verification
DVSC-Indirect uses DIVA checker core by Austin [Micro 1999]
Could substitute other mechanisms

11. Cache Coherence
All processors observe the same order of stores to a given memory location
Difficult because the same memory location can exist in different caches
Maintained by a coherence protocol
Different protocols: MOSI, MSI, MOESI, Token Coherence, …
Different maintenance mechanisms: directory, snooping
Verification uses “divide and conquer”
Verify conditions provably sufficient for cache coherence
Initially defined for proof of sequential consistency by Plakal et al. [SPAA1998]

12. Cache Coherence Verification
Coherence Conditions
Cache accesses are contained in an epoch
Stores in read-write epochs
Loads in read-write or read-only epochs
Read-write epochs do not overlap other epochs
Block data at beginning of epoch equals block data at end of last read-write epoch
Verification
Check if accesses are in appropriate epoch during DIVA-replay
Collect epoch information at every node and send to verifier
Verifier checks epoch history for overlaps and data propagation
Epoch
The time interval between obtaining and losing permissions on a block.

13. Implementation Overview
CPU
Core
CPU
Core
CPU
Core
DIVA
DIVA
DIVA
Cache
Record
Epochs
Cache
Record
Epochs
Cache
Record
Epochs
Interconnect
Memory
Collect
Epochs
Memory
Collect
Epoch
Memory
Collect
Epochs
Verify
Epochs
Verify
Epochs
Verify
Epochs
Epoch
History
Epoch
History
Epoch
History

14. At the Cache Controller
All caches keep track of active epochs in the Cache Epoch Table (CET)
Epoch Inform sent to the memory controller when epoch ends
Begin and end data are hashed
Every DIVA cache access checks CET for active epoch
Ensure access is contained in epoch
Verification off the critical path
Second order performance effect from bandwidth usage
Epoch Inform
CET
Type: read-write or
read-only
Begin time
Begin data
End time
End data

15. At the Memory Controller
Check for epoch overlaps and correct value propagation
Generally requires entire block history → O(N) space
If epoch informs are processed in order…
Need end value of last read-write epoch for propagation check
Need end time of last read-write and last read-only epoch for overlap check
O(1) space
Epochs arrive almost in order
Fix remaining re-orderings in priority queue before verifications
Epoch state in Memory Epoch Table (MET)
Last end time of read-only epoch and read-write epoch, last value

16. Experimental Evaluation
Empirically determine error detection capability
Error injection into caches, controller, interconnect, switches, etc.
Quantify error-free overhead
Increase in interconnect bandwidth consumption
Potential decrease in application performance

17. Simulation Methodology
Full-system simulation of 8-CPU UltraSPARC SMP
Simics functional simulation
GEMS-based timing simulation
2 GB RAM, 4-way 32KB I+D L1, 4-way 1MB L2
SafetyNet for backward error recovery
MOSI-Directory and MOSI-Snooping
Benchmarks
Apache 2
Static web-server
SpecJBB
3-Tier Java system
OLTP
Online transaction system with DB2
Slashcode
Dynamic website with perl and mysql
Barnes
Barnes-Hut from SPLASH2

18. Bottleneck Link Bandwidth - Directory

19. Error-Free Runtime - Directory
slower

20. Conclusions
DVSC-Direct and DVSC-Indirect enable end-to-end verification of the memory system
DVSC-Indirect imposes acceptable hardware and performance overhead
An extension of DVSC-Indirect to relaxed consistency is currently under development

21. Questions?