Huayang Guo 1,2, Ming Wu 1, Lidong Zhou 1, Gang Hu 1,2, Junfeng Yang 2, Lintao Zhang 1 1 Microsoft Research Asia 2 Columbia University Practical Software Model Checking via Dynamic Interface Reduction

Building reliable distributed systems is hard Machine failure Message lost Message reorder Thread interleaving Non-determinism leads to tricky bugs Crash Thr1 Thr2 Async I/O

Implementation-level software model checkers MaceMC (NSDI ’ 07), MoDist (NSDI ’ 09) Directly check implementations No need to construct abstract model beforehand Crash Thr1 Thr2 Async I/O State Space Explorer …

State space explosion MPS: Product-level Paxos Never fully explored 3 nodes 34 years for MoDist …

Dynamic Interface Reduction (DIR) Effective 34 years  18 hours (Fully explored MPS-3) Exponential Reduction: 100K : 1 states for MPS and Berkeley DB w/ replication Automatic, no manual efforts required Provably sound and complete Easy to integrate with legacy MCs DeMeter: DIR with MoDist and MaceMC MC specific modifications: ≤ 1k loc 5

Outline Insight Challenges Dynamic Interface Reduction Evaluation Related work Conclusion 6

Insight Distributed systems: componentized Local non-determinism isolated Empirically, 99.9% do not propagate (Berkeley DB) Previous work: Check components together |m1|*|m2|*|m3| DIR: Check components separately |m1|+|m2|+|m3| 7 Thr1 Thr2 Async I/O Thr3 Thr4 Interface behavior m1 m2 m3

Challenges and Solutions How to discover/construct interface behavior of component? Manually or statically construct interface process Impractical for complex software system How to guarantee Completeness: find all bugs Soundness: no false positives Our solution: Dynamically discover interface behaviors Combine discovered interface behaviors Track dependencies 8

DIR Overview 9 Global Explorer Explore global interface behaviors Local Explorers Component1 Component2 Component3 Explore local states Interface behavior

Example 10 Sum Ckpt Client Primary/Secondary //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Send(P,2); total+=n; Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Client Primary Secondary

Produce initial global trace 11 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer -- Produce initial global trace.

Construct message trace 12 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer -- Bold statements form the message trace.

Project message trace 13 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer -- Project global message trace to components. Pri.Recv(Cli, 1) Pri.Send(Sec, 1) Pri.Recv(Cli, 2) Pri.Send(Sec, 2) Primary Sec.Recv(Pri, 1) Sec.Recv(Pri, 2) Secondary Cli.Send(Pri, 1) Cli.Send(Pri, 2) Client

Local explorer for Primary 14 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer Pri.Recv(Cli, 1) Pri.Send(Sec, 1) Pri.Recv(Cli, 2) Pri.Send(Sec, 2) Local explorer for Primary Pri.Ckpt Pri.Sum Pri.Ckpt Pri.Sum Pri.Recv(Cli, 1) Pri.Send(Sec, 1) Pri.Recv(Cli, 2) Pri.Send(Sec, 2) Pri.Sum Pri.Recv(Cli, 1) Pri.Send(Sec, 1) Pri.Recv(Cli, 2) Pri.Send(Sec, 2)

Local explorer for Client 15 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer Cli.Send(Pri, 1) Cli.Send(Pri, 2) Local explorer for Client Cli.Choose(2) = 0 Cli.Send(Pri, 1) Cli.Send(Pri, 3) Cli.Choose(2) = 1 Branching Trace

Composition 16 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Existing global message trace: Cli.Send(Pri, 1) Cli.Send(Pri, 3) Branching local message trace: dependence ==

Composition 17 Client(Cli) Primary/Secondary(Pri/Sec) //Main thread //Checkpoint thread if (Choose(2)==0){ while (n=Recv()) { Lock(); Send(P,1); Lock(); Log(total); Ckpt Send(P,2); total+=n; Sum Unlock(); } else { Unlock(); Send(P,1); if (isPrimary) Send(P,3); Send(S,n); } Cli.Choose(2) = 0 Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Ckpt Pri.Sum Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Sec.Ckpt Sec.Sum Cli.Send(Pri, 2) Pri.Recv(Cli, 2) Pri.Sum Pri.Send(Sec, 2) Sec.Recv(Pri, 2) Sec.Sum Global explorer Cli.Send(Pri, 1) Pri.Recv(Cli, 1) Pri.Send(Sec, 1) Sec.Recv(Pri, 1) Cli.Send(Pri, 3) New global message trace:

Evaluation Experiment Setup D E M ETER -M O D IST : MPS, an deployed product implementation of Paxos Berkeley DB (BDB) D E M ETER -M ACE MC: Chord, peer-to-peer DHT implementation 18

Evaluation Effectiveness of Dynamic Interface Reduction App-n : n is the number of distributed nodes Reduction Ratio: |M w/o DIR | / |M w DIR | 19 AppMPS-2MPS-3BDB-2BDB-3Chord-2Chord-3 Reduction Speedup x1000 x100 DeMeter-ModistDeMeter-MaceMC

Related Work Compositional model checking E.M.Clarke et. al. (Symposium on Logic in Computer Science 1989) Partial-order reduction C.Flanagan and P.Godefroid (POPL ’ 05) Model checking network system R.Guerraoui and M.Yabandeh (NSDI ’ 11) 20

Conclusion Distributed systems  componentized Local non-determinism does not propagate Dynamic interface reduction Effective, automatic, easy Provably sound and complete DeMeter – enable DIR for legacy MCs 21