1. Properties Incompleteness Evaluation by Functional Verification IEEE TRANSACTIONS ON COMPUTERS, VOL. 56, NO. 4, APRIL 2007 1

2. Outline 2  Introduction  Background  Methodology  Generation of faulty implementations  Estimation of golden model incompleteness  Incremental property coverage computation  Experimental results  Conclusion

3. Introduction 3 Simulation-based techniques  Lack of exhaustiveness Formal verification  Overcome the exhaustiveness problem  Properties are derived from informal design specifications.  Model checking: prove the presence of bugs, but not their absence

4. Verification Flow Based on Model Checking 4

5. Vacuum Cleaning vs. Property Coverage Evaluation 5  Vacuum cleaning  Property coverage evaluation P = { p 1, p 2, …, p n } pipi pipi p n+1

6. Introduction – Model Checking 6 To increase the effectiveness of model checking Vacuity detection: look for properties that hold in a model and can be strengthened without causing them to fail Property coverage: address the question of whether enough properties have been defined How many properties should be defined to completely check the implementation? Coverage metric!

7. Coverage Metric 7  To measure property incompleteness  State coverage  Path coverage  Transition-based coverage

8. Introduction – Previous Work 8 Mutation-based ACTL, LTL, and CTL State coverage  path coverage  transition-based coverage Implementation-based State explosion problem Cannot precisely reflect the completeness of properties How about use mutation coverage jointly with dynamic verification to address the quality of the model checking process?

9. Background 9  Kripke structure K = {S, S 0, R, L}  FSM M = {I, O, S, s 0, R}  Product machine M P = M 1 X P M 2  Retroactive network Ι ε

10. Methodology Overview 10

11. Why Properties will be incomplete? 11 Functional test plan Design Verification System specifications  Informal to formal

12. Methodology Overview 12

13. Static vs. Dynamic 13  Static method  Formal verification  Time-consuming  Great effort in terms of memory resources  Exhaustive verification response  Dynamic method  ATPG & simulation  Lack of exhaustiveness  Rapider than static method

14. Generation of Faulty Implementations 14  The proposed methodology is independent of the adopted fault model  Different fault models can provide different estimations of the property completeness  Functional fault model  Bit coverage  has been proved to be related to design errors  Bit coverage fault model assumptions  Bit failure: stuck-at 0 or stuck-at 1  Condition failure: stuck-at true or stuck-at false  Single fault: A faulty implementation is generated for each fault

15. Generation of Faulty Implementations 15  Fault model and fault coverage for ATPG  Define functional fault model  RTL level  Bit coverage  Bit failure: stuck-at 0 or stuck-at 1  Condition failure: stuck-at true or stuck-at false  Single fault: A faulty implementation is generated for each fault  Has been proved to be related to design errors

16. Detectable Faults 16 fifi 0 1 000011 Environment

17. Generation of Faulty Implementations(cont.) 17  Detectable faults

18. Generation of Faulty Implementations(cont.) 18  A non-optimized algorithm  If fail then f is ε -detectable  Time-consuming and very likely state explosion  In this work: an approximation of the real set of ε -detectable

19. Methodology Overview 19

20. p-detectable and P-detectable 20 fifi 0 1 000011 Environment pipi SAT pipi UNSAT P = { p 1, p 2, …, p n }

21. Estimation of Golden Model Incompleteness 21  P-detectable and P-det  Property coverage

22. Property coverage 22  C P = 1  P is complete w.r.t. a specific fault model  Non-optimized algorithm

23. Estimation of Golden Model Incompleteness(cont.) 23  C P = 1  formal properties are complete w.r.t. a particular fault model  Non-optimized algorithm

24. Witnesses and Counterexamples 24  Witnesses  Existentially quantified CTL property  Counterexamples  Universally quantified CTL property

25. Estimation of Golden Model Incompleteness(cont.) 25  Witnesses and counterexamples  Tools can provide witnesses and counterexamples for CTL and LTL properties  Input witness and input counterexample

26. Witness Coverage 26  Property coverage can be estimated by using input witnesses  From formal verification to dynamic method  Under some conditions, C P = C w

27. Proof of C P = C w 27  Consider the safety and liveness properties separately  Proof of theorem 5.6 (safety property):

28. Proof of C P = C w (cont.) 28  w p -detectable and W P -detectable

29. Proof of C P = C w (cont.) 29

30. Incremental Property Coverage Computation 30

31. Coverage Accuracy Comparison 31  Combining static and dynamic verification makes this methodology can deal with real industrial circuits.  The methodology presented in this paper covers faults rather than states.  Can estimate coverage more accurate (compare with previous works)

32. Experimental Results 32 Test vector

33. Inspire to IC/CAD Contest 33  Functional fault model  Estimate coverage by fault instead of properties