1. debugging declarative models using core extraction Robert Seater with Ilya Shlyakhter, Daniel Jackson, Manu Sridharan, Mana Taghdiri December 20, 2005

2. 2 logical modeling subject model analysis feedback hard? accurate? guarantees? scalable?

3. 3 styles of logical modeling operational models (model checking) ›prescriptive (how to build a tree) ›temporal properties ›state machines ›natural for hardware declarative models ›descriptive (what a tree looks like) ›partial descriptions ›topological properties ›structured data ›natural for software alloy language ›first order relational logic + transitive closure ›encodes to SAT

4. 4 example: file system module FileSystem sig Object { parent: lone Dir } sig Dir extends Object { contents: set Object } sig File extends Object {} one sig Root extends Dir {} fact DefineContents { contents = ~parent } fact Partioning { File + Dir = Object } pred WellFormed () { no Root.parent all o: Object | o in Root.*contents } assert Acyclic { WellFormed() => all o: Object | o !in o.^contents } check Acyclic for 5 --no counterexample

5. 5 logical modeling subject model analysis feedback hard? accurate? guarantees? scalable?

6. 6 logical modeling: alloy subject model analysis feedback hard? accurate? guarantees? scalable? arbitrary FOL OO-like syntax structured data sat solvers symmetry breaking sound scope-complete

7. 7 logical modeling: alloy subject model analysis feedback hard? accurate? guarantees? scalable? arbitrary FOL OO-like syntax structured data sat solvers symmetry breaking sound scope-complete Did you write the model you think you wrote?

8. 8 2 types of errors underconstraint – allow erroneous behaviors ›easy to identify ›easy to locate ›harmless if missed - stronger result! overconstraint - disallow important behaviors ›hard to identify ›hard to locate ›dangerous if missed - may mask errors! extreme: no behaviors violate property because no behaviors exist (simple liveness) dangerous case: missing only error-revealing behaviors

9. 9 harmless underconstraint module FileSystem sig Object { parent: lone Dir } sig Dir extends Object { contents: set Object } sig File extends Object {} one sig Root extends Dir {} fact DefineContents { contents = ~parent } fact Partioning { File + Dir = Object } pred WellFormed () { no Root.parent all o: Object | o in Root.*contents } assert Acyclic { WellFormed() => all o: Object | o !in o.^contents } check Acyclic for 5 --fact Partioning { File + Dir = Object } --no counterexample

10. 10 relevant underconstraint module FileSystem sig Object { parent: lone Dir } sig Dir extends Object { contents: set Object } sig File extends Object {} one sig Root extends Dir {} fact DefineContents { contents = ~parent } fact Partioning { File + Dir = Object } pred WellFormed () { no Root.parent all o: Object | o in Root.*contents } assert Acyclic { WellFormed() => all o: Object | o !in o.^contents } check Acyclic for 5 --all o: Object | o in Root.*contents --counterexample !

11. 11 relevant underconstraint Dir_0 (Acyclic) Root_0 Dir_0 (Acyclic) parent contents

12. 12 2 types of errors underconstraint – allow erroneous behaviors ›easy to identify ›easy to locate ›harmless if missed - stronger result! overconstraint - disallow important behaviors ›hard to identify ›hard to locate ›dangerous if missed - may mask errors! extreme: no behaviors violate property because no behaviors exist (simple liveness) dangerous case: missing only error-revealing behaviors

13. 13 overconstraint module FileSystem sig Object { parent: one Dir } sig Dir extends Object { contents: set Object } sig File extends Object {} one sig Root extends Dir {} fact DefineContents { contents = ~parent } fact Partioning { File + Dir = Object } pred WellFormed () { no Root.parent all o: Object | o in Root.*contents } assert Acyclic { WellFormed() => all o: Object | o !in o.^contents } check Acyclic for 5 --lone Dir } --no counterexample ! --all o: Object | o in Root.*contents

14. 14 what would help? when solutions exist ›an example explains why when no solutions exist ›a proof explains why

15. 15 what would help? when solutions exist ›an example explains why when no solutions exist ›a proof explains why problem with proofs ›long & hard (1,000's of resolutions) ›in terms of CNF clauses solution ›just what was was used in the proof ›do so in terms the user can understand (use the text of the model)

16. 16 extracted unsat core module FileSystem sig Object { parent: one Dir } sig Dir extends Object { contents: set Object } sig File extends Object {} one sig Root extends Dir {} fact DefineContents { contents = ~parent } fact Partioning { File + Dir = Object } pred WellFormed () { no Root.parent all o: Object | o in Root.*contents } assert Acyclic { WellFormed() => all o: Object | o !in o.^contents } check Acyclic for 5 --lone Dir } --no counterexample ! --all o: Object | o in Root.*contents

17. 17 core extraction extracted unsat core ›subset of model ›sufficient to rule out solutions ›changing rest of model leaves it unsat ›double checks user’s intuition guarantee: Altering non-core portions of an unsatisfiable model in a syntactically valid manner will leave the model unsatisfiable.

18. 18 algorithm constr aint langua ge user’s view SAT solver … (a - b) in b …

19. 19 algorithm constr aint langua ge user’s view SAT solver … (a - b) in b … a b b inin - AST of user’s model

20. 20 algorithm constr aint langua ge CNF clauses user’s view SAT solver … (a - b) in b … a b b in - (b3)  (b1 V ~b2 V b3)  (b1 V ~b3 V b4)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  … convert AST of user’s model

21. 21 algorithm constr aint langua ge CNF clauses SAT solver user’s view SAT solver … (a - b) in b … a b b inin - convert analyze AST of user’s model (b3)  (b1 V ~b2 V b3)  (b1 V ~b3 V b4)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

22. 22 algorithm constr aint langua ge CNF clauses SAT solver user’s view SAT solver … (a - b) in b … a b b inin - convert analyze AST of user’s model solve “unsatisfiabl e” (b3)  (b1 V ~b2 V b3)  (b1 V ~b3 V b4)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

23. 23 algorithm constr aint langua ge CNF clauses SAT solver CNF core user’s view core extraction SAT solver subset … (a - b) in b … a b b inin - convert analyze AST of user’s model solve extract core “unsatisfiabl e” (b3)  (b1 V ~b2 V b3)  (b1 V ~b3 V b4)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

24. 24 algorithm constr aint langua ge CNF clauses SAT solver CNF core user’s view core extraction SAT solver subset … (a - b) in b … a b b inin - convert analyze map back AST of user’s model solve extract core “unsatisfiabl e” Highlighte d AST (b3)  (b1 V ~b2 V b3)  (b1 V ~b3 V b4)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

25. 25 algorithm constr aint langua ge CNF clauses SAT solver CNF core user’s view core extraction SAT solver subset … (c - b) in b … c b b inin - convert analyze map back AST of user’s model solve extract core “unsatisfiabl e” alter AST Highlighte d AST Altered AST (b3)  (b1 V ~b2 V b3)  (b1 V ~b3 V b4)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

26. 26 algorithm constr aint langua ge CNF clauses SAT solver CNF core CNF clauses user’s view core extraction SAT solver subset superset … (c - b) in b … c b b inin - convert analyze map back AST of user’s model solve extract core “unsatisfiabl e” alter AST Highlighte d AST Altered AST (b3)  (b1 V ~b2 V b3)  (b5 V ~b6 V b7)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

27. 27 algorithm constr aint langua ge CNF clauses SAT solver CNF core CNF clauses SAT solver user’s view core extraction SAT solver subset superset … (c - b) in b … c b b inin - convert analyze map back AST of user’s model solve extract core “unsatisfiabl e” alter AST Highlighte d AST Altered AST (b3)  (b1 V ~b2 V b3)  (b5 V ~b6 V b7)  (~b1 V b2)  (b1 V ~b3)  (~b1 V ~b2)  …

28. 28 complications shared subformulae ›important optimization ›may cause larger core ›can often be trimmed core may not be minimal ›iterate to fixed point ›usually locally minimal core is not unique ›could exist different (smaller) core requires node-by-node translation ›one CNF variable per node ›clauses for different nodes are independent

29. 29 case studies logs of common errors ›‘dumb bugs’ ›all languages have them ›time consuming in practice ›core extraction often nails them major case studies ›known, subtle bugs ›iolus - secure multi-casting core extraction revealed bug’s location ›firewire - ‘tree identify’ protocol core extraction helped narrow down bug’s location

30. 30 key related work vacuity testing by model checkers ›(Beer '01, Chockler '01, Kupferman '99, Vardi '03) ›modal logic in particular form ›focus on property not model ›cannot pinpoint model subsets responsibility ›(Chockler) ›relative important of subformulae ›analogous to number of cores a formula is in procedure call abstraction ›uses unsat core in refinement step

31. 31 conclusions contributions ›filled a hole in declarative modeling tools ›mapped proof contents to something meaningful ›proof of correctness ›case studies Shlyakhter, Seater, Jackson, Sridharan, Taghdiri. Debugging Declarative Models Using Unsatisfiable Cores. Automated Software Engineering (ASE), 2003. (best paper award)