Properties Incompleteness Evaluation by Functional Verification IEEE TRANSACTIONS ON COMPUTERS, VOL. 56, NO. 4, APRIL

Outline 2  Introduction  Background  Methodology  Generation of faulty implementations  Estimation of golden model incompleteness  Incremental property coverage computation  Experimental results  Conclusion

Introduction 3 Simulation-based techniques  Lack of exhaustiveness Formal verification  Overcome the exhaustiveness problem  Properties are derived from informal design specifications.  Model checking: prove the presence of bugs, but not their absence

Verification Flow Based on Model Checking 4

Vacuum Cleaning vs. Property Coverage Evaluation 5  Vacuum cleaning  Property coverage evaluation P = { p 1, p 2, …, p n } pipi pipi p n+1

Introduction – Model Checking 6 To increase the effectiveness of model checking Vacuity detection: look for properties that hold in a model and can be strengthened without causing them to fail Property coverage: address the question of whether enough properties have been defined How many properties should be defined to completely check the implementation? Coverage metric!

Coverage Metric 7  To measure property incompleteness  State coverage  Path coverage  Transition-based coverage

Introduction – Previous Work 8 Mutation-based ACTL, LTL, and CTL State coverage  path coverage  transition-based coverage Implementation-based State explosion problem Cannot precisely reflect the completeness of properties How about use mutation coverage jointly with dynamic verification to address the quality of the model checking process?

Background 9  Kripke structure K = {S, S 0, R, L}  FSM M = {I, O, S, s 0, R}  Product machine M P = M 1 X P M 2  Retroactive network Ι ε

Methodology Overview 10

Why Properties will be incomplete? 11 Functional test plan Design Verification System specifications  Informal to formal

Methodology Overview 12

Static vs. Dynamic 13  Static method  Formal verification  Time-consuming  Great effort in terms of memory resources  Exhaustive verification response  Dynamic method  ATPG & simulation  Lack of exhaustiveness  Rapider than static method

Generation of Faulty Implementations 14  The proposed methodology is independent of the adopted fault model  Different fault models can provide different estimations of the property completeness  Functional fault model  Bit coverage  has been proved to be related to design errors  Bit coverage fault model assumptions  Bit failure: stuck-at 0 or stuck-at 1  Condition failure: stuck-at true or stuck-at false  Single fault: A faulty implementation is generated for each fault

Generation of Faulty Implementations 15  Fault model and fault coverage for ATPG  Define functional fault model  RTL level  Bit coverage  Bit failure: stuck-at 0 or stuck-at 1  Condition failure: stuck-at true or stuck-at false  Single fault: A faulty implementation is generated for each fault  Has been proved to be related to design errors

Detectable Faults 16 fifi Environment

Generation of Faulty Implementations(cont.) 17  Detectable faults

Generation of Faulty Implementations(cont.) 18  A non-optimized algorithm  If fail then f is ε -detectable  Time-consuming and very likely state explosion  In this work: an approximation of the real set of ε -detectable

Methodology Overview 19

p-detectable and P-detectable 20 fifi Environment pipi SAT pipi UNSAT P = { p 1, p 2, …, p n }

Estimation of Golden Model Incompleteness 21  P-detectable and P-det  Property coverage

Property coverage 22  C P = 1  P is complete w.r.t. a specific fault model  Non-optimized algorithm

Estimation of Golden Model Incompleteness(cont.) 23  C P = 1  formal properties are complete w.r.t. a particular fault model  Non-optimized algorithm

Witnesses and Counterexamples 24  Witnesses  Existentially quantified CTL property  Counterexamples  Universally quantified CTL property

Estimation of Golden Model Incompleteness(cont.) 25  Witnesses and counterexamples  Tools can provide witnesses and counterexamples for CTL and LTL properties  Input witness and input counterexample

Witness Coverage 26  Property coverage can be estimated by using input witnesses  From formal verification to dynamic method  Under some conditions, C P = C w

Proof of C P = C w 27  Consider the safety and liveness properties separately  Proof of theorem 5.6 (safety property):

Proof of C P = C w (cont.) 28  w p -detectable and W P -detectable

Proof of C P = C w (cont.) 29

Incremental Property Coverage Computation 30

Coverage Accuracy Comparison 31  Combining static and dynamic verification makes this methodology can deal with real industrial circuits.  The methodology presented in this paper covers faults rather than states.  Can estimate coverage more accurate (compare with previous works)

Experimental Results 32 Test vector

Inspire to IC/CAD Contest 33  Functional fault model  Estimate coverage by fault instead of properties