September 1999Compaq Computer CorporationSlide 1 of 16 Verification of cache-coherence protocols with TLA+ Homayoon Akhiani, Damien Doligez, Paul Harter, Leslie Lamport, Joshua Scheid, Mark Tuttle, Yuan Yu Compaq Computer Corporation

September 1999Compaq Computer CorporationSlide 2 of 16 TLA+ A formal specification language based on set theory, first-order logic, temporal logic Hierarchical style clarifies written –specifications: becomes –proofs: becomes Engineers find reading easy, writing not too hard CASE 2. CASE 3. QED

September 1999Compaq Computer CorporationSlide 3 of 16 Used TLA+ to demonstrate formal methods to engineering Analyzed cache-coherence protocols for –EV6: Alpha processor –EV7: Alpha processor Built TLC, a model-checker for TLA+ Analyzed proposals for industry standards –PCI-X, …

September 1999Compaq Computer CorporationSlide 4 of 16 Cache coherence protocols Goal: prove the cache coherence protocol is correct. processor cache memory x=2 Alpha memory model defines ordering of reads and writes to x. Cache coherence protocol enforces the Alpha memory model. cache x=2 cache x=1 processor

September 1999Compaq Computer CorporationSlide 5 of 16 EV6 cache coherence in “three easy steps”+“two-man years” Model Alpha memory model. (200 lines) Model complete protocol. (2000 lines, 3 months) Prove implementation (5500 lines, 4+ months, incomplete) Model abstract protocol. (500 lines) Prove implementation (550 lines, 2 months, informal)

September 1999Compaq Computer CorporationSlide 6 of 16 Step 1: Alpha memory model We specified the Alpha memory memory model: –The official specification is an informal description of the allowed sequences of reads and writes. –We needed a precise, state-based specification. –We specified a slightly simplified memory model. Compare the specifications: –Official, English specification: 12 pages –Logical, precise specification: 200 lines

September 1999Compaq Computer CorporationSlide 7 of 16 Step 2: Model abstract protocol protocol = abstract protocol + implementation junk Surprisingly, –abstract protocol’s correctness was far from obvious –we discovered a bug… in the memory model Proved hardest part of correctness: –35-line invariant based on 300 lines of definitions –550-line proof, cases nested 10 levels deep

September 1999Compaq Computer CorporationSlide 8 of 16 Obstacle 1: find a single, complete description –English documents: 20 documents, 4-inch stack –Lisp simulator: crucial to understanding some details Obstacle 2: algorithm complexity –60 different kinds of messages –15 “quarks” could combine to model all 60 messages Protocol: 9 man-months, 1900 lines of TLA+ Partial proof: 7 man-months, 1000-line invariant Step 3: Model complete protocol

September 1999Compaq Computer CorporationSlide 9 of 16 Results: one bug Quite unexpected to find only one bug! Heavy simulation had found the easy bugs Demonstrating our bug requires –four processors –two memory locations –fifteen messages Hand proof appears essential to finding this bug: –extensive simulation did not find it –state space too large for exhaustive model checking

September 1999Compaq Computer CorporationSlide 10 of 16 Lessons learned The designers had no trouble reading our spec. The level of rigorous analysis resulting even from a partial proof delighted the designers The demonstration convinced engineers to consider doing the same thing on their own... The basic methodology worked as expected Tools, even simple tools, are essential…

September 1999Compaq Computer CorporationSlide 11 of 16 TLC model checker State machine in rich subset of TLA+ (Initial, NextState) Configuration file making state machine finite Invariant Minimal state trace from an initial state to a bad state Check for Invariant false Deadlock

September 1999Compaq Computer CorporationSlide 12 of 16 TLC implementation Require no changes to TLA+ specifications –use the richness of TLA+, no primitive language –use configuration files instead Interpret specifications, don’t compile them –better user interaction possible Use explicit state representation, not BDDs –BDD encoding of TLA+ formulas difficult –use canonical state representation + fingerprinting –use efficient disk-based state set and queue implem.

September 1999Compaq Computer CorporationSlide 13 of 16 TLC status 20,000 lines of Java Compaq internal distribution available now Performance is good, sometimes slow: threaded and distributed implementations now exist. Liveness checking/livelock detection coming Coverage analysis is desired: What does lack of an error mean: a correct spec or a buggy spec?

September 1999Compaq Computer CorporationSlide 14 of 16 EV7 cache coherence First intense application of TLC model checker First TLA+ specification written by engineers Specification is 1800 lines Specification accepted by TLC w/o modification State space reduced 50% by adding 15 lines to remove a lot of symmetry in state space

September 1999Compaq Computer CorporationSlide 15 of 16 Results 73 bugs found (90% found by TLC): –37 minor: typos, type errors, etc –12 bugs: wrong message/wrong state –14 missing cases –7 spurious cases (dead code) –3 miscellaneous (1 TLA+, 1 MC, 1 spec design) War story: Find bug B by hand; find bug B’ like B by simulation; find bug B’’ in bug-fix for B; find “???” written in original documentation!

September 1999Compaq Computer CorporationSlide 16 of 16 Lessons learned Learning TLA+ is not a major task, but writing good specifications still requires experience EV6 verification was –humbling: only one error actually found –encouraging: the basic method works as expected EV7 verification was very satisfying: –TLA+ specifications can be written by engineers –TLC can handle industrial-sized specifications Formal specification belongs in design process…