1. A Coverage Analysis for Safety Property Lists
Koen Lindström Claessen
Chalmers University
Gothenburg, Sweden
FMCAD 2007, Austin, Texas

2. Property-based Verification
Properties
Design ?

3. Dynamic Verification Checkers/ Monitors PSL/SVA properties Properties
Design
stimuli ? ? ?

4. In Formal Verification:
Simulation Coverage
95%
Statement coverage
Path coverage
Properties
75%
Design
In Formal Verification:
100%
Gate coverage
98%
State space coverage
66%
Register coverage
83%
stimuli

5. How About the Properties?
2. B & C => next A
3. not (A and C)
4. next A => B
Design ?
Removing / adding one of these does not affect coverage…

6. Property Coverage We need coverage for properties
How much is the design constrained by properties?
Do the properties express what we want to say?
Are any properties missing?
(lots of work done on this)
The more the merrier!

7. Mutation Coverage
Properties
Design
Design ! ?
“Point not covered”

8. [Chockler, Kupferman,Vardi]
Mutation Coverage
Expensive
Many (failing) calls to model checker
Involves the whole design
Solutions
Symbolically; still expensive
Approximations
Look at proof
Subset of mutations
[Chockler, Kupferman,Vardi]

9. This Talk A property coverage analysis Finds “forgotten cases” …
Design may change after analysis
Design does not have to exist
A property coverage analysis
Independent of the design
Finds “forgotten cases” …
… which indicate forgotten properties
Relatively cheap
Only involves properties
Independent of design size
NaN
Not a quantitative analysis
There may exist multiple designs

10. Forgotten Case Analysis
1. prop1
2. prop2
3. prop3
4. prop4
Design

11. “Forgotten Case”? Properties specify a function : In  Out? Balance
Too strict
Means complete specification
Synthesizable
Balance
Strict enough: to catch forgotten cases
Loose enough: to be practically usable

12. Choice: Forgotten Case
In this trace:
out1 is not constrained by properties at time 6:
Forgotten case!
Choice: Forgotten Case
Properties OK
in …
in …
in …
out …
out …
Properties still OK

13. fst always outputs the first element
Example: a FIFO
fst always outputs the first element
put overrides get
FIFO
get
fst
num
put
err in
err signals for 1 clock cycle when something goes wrong – the FIFO does not break

14. No restriction on the logic…
A First Property List
Safety properties
always (put=1 & num=n  next err=1)
always (put=1 & num<n  next num=num+1)
always (put=1 & num=0  next fst=in)
always (put=1 & 0<num<n  next fst=fst)
always (get=1 & put=0 & num=0  next err=1)
always (get=1 & put=0 & 0<num  next num=num-1)
No restriction on the logic…
n = max. FIFO size

15. Not constrained at time 0
Analyzing err
get 0 …
put 0 …
in 0 …
num 0 …
fst 0 …
err ? …
Not constrained at time 0

16. Holds at initial point in time
Analyzing err: A Fix
Add:
err=0
Holds at initial point in time

17. Analyzing err get 0 0 … put 0 0 … in 0 0 … num 0 0 … fst 0 0 …
Nothing goes wrong…

18. Analyzing err: A fix
Add:
always (get=0 & put=0  next err=0)

19. Analyzing err get 0 0 … put 1 0 … in 1 0 … num 0 1 … fst 0 1 …
Nothing goes wrong…

20. Analysis does not complain about err anymore…
Analyzing err: A fix
always (put=1 & num<n  next num=num+1)
always (get=1 & put=0 & 0<num  next num=num-1)
Change to:
always (put=1 & num<n  next num=num & next err=0)
always (get=1 & put=0 & 0<num  next num=num & next err=0)
Analysis does not complain about err anymore…

21. Not constrained at time 0
Analyzing num
get 0 …
put 0 …
in 0 …
num ? …
fst 0 …
err 0 …
Not constrained at time 0

22. Holds at initial point in time
Analyzing num: A Fix
Add:
num=0
Holds at initial point in time

23. Analyzing num get 0 0 … put 0 0 … in 0 0 … num 0 ? … fst 0 0 …
err 0 0 …
Not constrained

24. Analyzing num: A fix
Add:
always (get=0 & put=0  next num=num)

25. Analyzing num get 1 0 … put 0 0 … in 0 0 … num 0 ? … fst 0 0 …
err 0 1 …
An error has occurred

26. Analyzing num: A fix always (get=0 & put=0  next num=num) Change to:
always ((get=0 & put=0) v next err=1  next num=num)
The analysis is now happy about num…
Dependencies between outputs…

27. But we don’t want to specify it!
Analyzing fst
get 0 …
put 0 …
in 0 …
num 0 …
fst ? …
err 0 …
Not restricted
But we don’t want to specify it!

28. Analyzing fst: A fix Logically vacuous: “fst=fst” Add:
always (num=0  free fst)
A new keyword:
free
For the analysis:
free x means that x is constrained: “don’t complain!”

29. FIFO behavior not specified in properties
Analyzing fst
get …
put …
in …
num …
fst ? …
err …
Not restricted:
FIFO behavior not specified in properties

30. Unconstrained Outputs
Cases we have to be explicit about
Three causes:
It is supposed to be unspecified
We decide not to specify it (complicated)
We have forgotten to specify it
Cases we want to catch

31. Analyzing fst: A fix
always (get=1 & put=0 & 0<num  next num=num & next err=0)
Change to:
always (get=1 & put=0 & 0<num  next num=num & next err= & next free fst)

32. Analyzing fst get 0 0 … put 0 0 … in 0 0 … num 0 0 … fst 0 ? …
err 0 0 …
Real forgotten case

33. The analysis is now happy about fst…
Analyzing fst: A fix
always ((get=0 & put=0) v next err=1  next num=num)
Change to:
always ((get=0 & put=0) v next err=1  next num=num & next fst = fst)
The analysis is now happy about fst…
Luckily, we used free…

34. New & changed properties
initial values
implicit behavior
constant behavior
err=0 & num=0
always (get=0 & put=0  next err=0)
always ((get=0 & put=0) v next err=1  next num=num & next fst = fst)
always (put=1 & num<n  next num=num & next err=0)
always (get=1 & put=0 & 0<num  next num=num & next err= & next free fst)

35. Implementation For output s, find a trace where Props(s) is OK
Exists exactly one t where s[t]≠s’[t]
free(s)[t] should be false
(this trace must be infinite)

36. Using a standard LTL model checker (NuSMV)
Implementation (I)
Using a standard LTL model checker (NuSMV)
Find a trace satisfying:
Props(s) & Props(s’) & ◊!(s≠s’)
With the use of free:
Props(s,free_s) & Props(s’,free_s) & ◊!(~free_s & s≠s’)

37. Implementation (II) Property observer for safety property phi:
□OK holds iff. phi holds
Analysis for property observers:
□OK(s) & □OK(s’) ◊!(s≠s’)

38. Discussion (I) Forgotten case Alternatives…
Given an output signal s and a time t, and given the values of all other signals at all points in time, and given all values of s not at time t, do the properties force the value of s at time t?
Alternatives…

39. Discussion (II) Freeness
“free s” does not indicate that s can take on any value
rather, it is an artifical way of constraining s for the sake of the analysis, without actually restricting it logically

40. Conclusion: This Analysis
Identifies forgotten cases
Which inspire forgotten properties
Forces to specify when outputs are free
Distinction between forgotten cases and underconstrainedness
Is design-independent
pre-design / multiple implementations
cost