1. Statistical Probabilistic Model Checking
Håkan L. S. Younes
Carnegie Mellon University
Yesterday, Scott Smolka presented a statistical approach for LTL model checking. This is also a talk about using statistical techniques for model checking, but unlike Scott I will talk about probabilistic model checking.

2. Introduction Model checking for stochastic processes
Stochastic discrete event systems
Probabilistic time-bounded properties
Model independent approach
Discrete event simulation
Statistical hypothesis testing
With probabilistic model checking I mean that the model is a stochastic process, or more precisely a stochastic discrete event system. The properties that we will be looking at are also probabilistic. We will, in particular, look at the problem of verifying probabilistic time-bounded properties. I will present a model independent algorithm for verifying such properties for stochastic discrete event systems. The algorithm is based on discrete event simulation and statistical hypothesis testing. We make no assumptions regarding the dynamics of the system, other than that we can generate simulation traces for the system. I will tell you shortly exactly what kind of properties we can verify using our approach, but first let us look at an example system.

3. Example: Tandem Queuing Network
arrive
route
depart q1 q2
q1 = 0 q2 = 0
q1 = 0 q2 = 0
q1 = 1 q2 = 0
q1 = 1 q2 = 0
q1 = 2 q2 = 0
q1 = 2 q2 = 0
q1 = 1 q2 = 1
q1 = 1 q2 = 1
q1 = 1 q2 = 0
q1 = 1 q2 = 0
Consider a simple system consisting of two serially connected queues, each of capacity two. Messages arrive at the first queue, get routed to the second queue after some time, and leave the second queue after some additional time. The time between arrivals and the time that a message remains in each of the queues are random variables.
Say that we start off with both queues being empty at time zero. The hour glass represents the fact that time is a continuous quantity. Say that after some time, in this case 1.2 seconds, a message arrives at the first queue. This causes a transition to a state where the first queue contains one message and the second queue is empty. After additional time, another message may arrive at the first queue, which then contains two messages. One of the messages in the first queue then gets routed to the second queue. After another 1.6 seconds, the message leaves the second queue, leaving the system in a state at time 5.5 seconds where the first queue contains one message and the second queue is empty. This sequence of states is a path or trace for the system.
The system is stochastic because the time we spend in a state is a random variable.
Given a stochastic system like this tandem queuing network, we can ask the follow question: with both queues empty, is the probability less than 0.5 that both queues become full within 5 seconds? For the given path, both queues did not get full within 5 seconds, but that is just for a single path. This question represents a probabilistic model checking problem.
t = 0
t = 1.2
t = 3.7
t = 3.9
t = 5.5
With both queues empty, is the probability less than 0.5 that both queues become full within 5 seconds?

4. Probabilistic Model Checking
Given a model M, a state s, and a property , does  hold in s for M?
Model: stochastic discrete event system
Property: probabilistic temporal logic formula
In general, a probabilistic model checking problem can be formulated as follows. Given a model M, a state s of this model, and a property phi, does phi hold in s for M. In our case, the model is a stochastic discrete event system and the property is expressed as a formula in some probabilistic temporal logic.

5. Continuous Stochastic Logic (CSL)
State formulas
Truth value is determined in a single state
Path formulas
Truth value is determined over a path
We use CSL, the continuous stochastic logic, for expressing properties of continuous-time stochastic systems.
In CSL there are state formulas and path formulas.
The truth value of a state formula is determined in a single state, while the truth value of a path formula is determined over a path.
An essentially analogous logic for discrete-time systems is known as PCTL.
Discrete-time analogue: PCTL

6. State Formulas Standard logic operators: , 1  2, …
Probabilistic operator: P≥ ()
Holds in state s iff probability is at least  that  holds over paths starting in s
P< ()  P≥1– ()
State formulas are regular logic expression, with logic operators such as negation and conjunction.
In addition, there is a probabilistic operator. Rho is a path formula, and the probabilistic formula holds in a state s if and only if the probability is at least theta that the path formula rho holds over paths starting in s.
It is sufficient to consider only greater-than-or-equal-to. Strictly-less-than can be expressed using negation and greater-than-or-equal-to. Also, since we use a statistical approach, we cannot really differentiate between strictly-less-than and less-than-or-equal-to, or strictly-greater-than and greater-than-or-equal-to.
In full CSL there is also a steady-state operator, but we do not have a model independent approach for dealing with it so I will not discuss it further in this talk.

7. Path Formulas Until: 1 U ≤T 2
Holds over path  iff 2 becomes true in some state along  before time T, and 1 is true in all prior states
The primary path formula of interest is the until formula.
An until formula holds over a path sigma if and only if phi two becomes true in some state along sigma before time T, and phi 1 is true in all prior states. While T can be infinite, corresponding to an unbounded until formula, this talk will only involve properties with finite time bound. We use the time-bound when generating sample paths using discrete event simulation.

8. CSL Example
With both queues empty, is the probability less than 0.5 that both queues become full within 5 seconds?
State: q1 = 0  q2 = 0
Property: P<0.5(true U ≤5 q1 = 2  q2 = 2)
The question we asked about the tandem queuing network can be formulated as a CSL model checking problem. Both queues being empty corresponds to a state with q1 and q2 being zero. The property that the probability is less than 0.5 that both queues become full within 5 seconds can be expressed in CSL with a probabilistic state formula, where the path formula is an until formula with time bound 5.
To answer the posed question, we check if the CSL formula holds in the given state.

9. Model Checking Probabilistic Time-Bounded Properties
Numerical Methods
Provide highly accurate results
Expensive for systems with many states
Statistical Methods
Low memory requirements
Adapt to difficulty of problem (sequential)
Expensive if high accuracy is required
So how do we solve problems like this? Two different approaches have been suggested in the literature: one based on the numerical computation of probabilities and one based on statistical hypothesis testing. Numerical methods are particularly well-suited when high accuracy in the verification result is required, but scale poorly with an increase in the size of the state space of the model. A statistical method, on the other hand, has very low memory requirements. It can also adapt to the difficulty of the problem at hand, and I will tell you exactly what I mean by that a little bit later. However, statistical hypothesis testing tends to be expensive if high accuracy is required, so there are pros and cons with both approaches.

10. Statistical Solution Method [Younes & Simmons 2002]
Use discrete event simulation to generate sample paths
Use acceptance sampling to verify probabilistic properties
Hypothesis: P≥ ()
Observation: verify  over a sample path
The focus of this talk is on a statistical solution method for CSL model checking, proposed by Younes and Simmons in a CAV paper from This approach takes advantage of the fact that CSL model checking is a hypothesis testing problem rather than an estimation problem. The approach uses discrete event simulation to generate sample paths, and a technique called acceptance sampling to verify probabilistic properties. The hypothesis we are testing is whether the path formula rho holds with at least probability theta. An observation in our case is the result of verifying rho over a sample path.
I want to stress once again that we are not estimating the probability of rho holding!
Not estimation!

11. Error Bounds Probability of false negative: ≤ 
We say that  is false when it is true
Probability of false positive: ≤ 
We say that  is true when it is false
Since we use a statistical approach, we have to accept some probability of error. However, we should at least bound the probability of error.
There are two types of error: false negatives and false positives. We want the probability of a false negative to be at most alpha, and the probability of a false positive to be at most beta.

12. Performance of Test 1 –  Probability of accepting P≥ () as true  
The performance of an acceptance sampling test can be viewed graphically as follows.
Along the x-axis we have the actual probability of rho holding, and there we have our threshold theta.
Along the y-axis we have the probability that we accept the hypothesis as true when using the test. There we have our error bounds alpha and beta.
We do not know the actual probability of rho holding, but if it is to the right of theta then the property we are verifying is true. If it is to the left of theta, then the property is false.  
Actual probability of  holding

13. Ideal Performance of Test
False negatives
False positives
1 – 
Unrealistic!
Probability of accepting P≥ () as true
Ideally we want a test that accepts the hypothesis with high probability (at least 1-alpha) if it is really true, and accepts the hypothesis with low probability (at most beta) if it is really false. This kind of performance is plotted here, but we soon realize that it is unrealistic to expect there to be an efficient test with a performance like this.  
Actual probability of  holding

14. Realistic Performance of Test2 p1 p0
Indifference region
False negatives
False positives
1 – 
Probability of accepting P≥ () as true
We therefore relax the problem by introducing an indifference region of width two delta centered around our threshold. We want a high probability of accepting the hypothesis if the true probability of rho holding is to the right of the indifference region and a low probability of accepting the hypothesis if the probability is to the left of the indifference region. If the probability of rho holding is in the indifference region, we are willing to accept a high error probability. This relaxation makes it possible to construct an efficient test.  
Actual probability of  holding

15. Sequential Acceptance Sampling [Wald 1945]
True, false, or another observation?
One particular way of doing acceptance sampling, is sequential acceptance sampling. This technique was developed by Wald in a truly amazing paper from If I got to take one thing with me to a deserted island, it would be this paper.
A sequential test works as follows. We make observations of the truth value of rho, and after each observation we need to decide whether to accept or reject the hypothesis, or if we need to make more observations. Each observation can be viewed as the outcome of a coin flip, and we want to determine if the probability of, say, heads coming up is at least theta. We keep flipping the coin, recording the number of heads we have seen as well as the total number of coin flips, and based on this information we decide if we have enough information to accept or reject the hypothesis so that we respect the given error bounds.

16. Graphical Representation of Sequential Test
Number of observations
Number of positive observations
OK, we have already seen the kind of performance we expect to get in terms of error probability from using acceptance sampling. But how do we decide whether to accept/reject the hypothesis or make more observations?
The best way to explain how this works is to give a graphical representation of the sequential test.
Along the x-axis we have the number of observations made so far.
Along the y-axis we have the number of positive observations seen, which in our case is the number of sample paths that have satisfied the path formula rho so far.
The blue area represents unreachable points, because we can never make more positive observations than the total number of observations.

17. Graphical Representation of Sequential Test
We can find an acceptance line and a rejection line given , , , and 
acceptance line
Continue until line is crossed
Number of observations
Number of positive observations
accept
Verify  over sample paths
continue
We can find an acceptance and a rejection line given the parameters theta, delta, alpha, and beta. We then plot a curve representing the outcome of our test starting with zero samples and zero positive samples. As long as the curve stays between the two lines, we continue sampling. If the curve crosses the rejection line, we reject the hypothesis, and if it crosses the acceptance line, we accept the hypothesis.
The two lines are, by the way, parallel.
rejection line
Start here
reject

18. Special Case p0 = 1 and p1 = 1 – 2 “Five nines”: p1 = 1 – 10–5
Reject at first negative observation
Accept at stage m if p1m ≤ 
Sample size at most dlog  / log p1e
“Five nines”: p1 = 1 – 10–5
Now, the number of observations we need to make in order to reach a decision is rather tricky to determine in the general case, but there is a special case of some interest, which we can give a clear upper bound for.
Consider the case when the upper boundary of the indifference region is 1. In that case we reject the hypothesis at the first negative observation we see and we accept the hypothesis at stage m if we have made only positive observations and p1 to the power m is at most beta. The maximum sample size is therefore the ceiling of log beta divided by log p1.
So why is this an interesting case? A popular concept in reliability engineering is “five nines”. We can model this by setting p0 to one and p1 to one minus 10^-5. The table shows the maximum sample size for different values of beta. For example, if we want a probability of at most 10^-8 of accepting a system as functional if the failure probability is at least 10^-5, then we need to make at most 1.8 million observations. This may sound like a lot, but think about the accuracy we get. 
Maximum sample size
10–2
460,515
10–4
921,030
10–8
1,842,059

19. Case Study: Tandem Queuing Network
M/Cox2/1 queue sequentially composed with M/M/1 queue
Each queue has capacity n
State space of size O(n2)
To give you an idea of how the statistical approach compares to a numerical approach, I present two case studies.
First consider a variation of the tandem queuing network from before consisting of two sequentially composed queues. Each queue has capacity n, and the size of the state space is quadratic in n. 1 2 a …
1 − a  

20. Tandem Queuing Network (results) [Younes et al. 2004]
P≥0.5(true U≤T full)
106
105
104
 = 10−6
 =  = 10−2
 = 0.5·10−2
103
Verification time (seconds)
102
101
100
10−1
10−2
101
102
103
104
105
106
107
108
109
1010
1011
Size of state space

21. Tandem Queuing Network (results) [Younes et al. 2004]
P≥0.5(true U≤T full)
106
105
104
 = 10−6
 =  = 10−2
 = 0.5·10−2
103
Verification time (seconds)
102
101
100
10−1
10−2
101
102
103
104 T

22. Case Study: Symmetric Polling System
Single server, n polling stations
Stations are attended in cyclic order
Each station can hold one message
State space of size O(n·2n)
Next consider a polling system with a single server and n polling stations. The stations are attended by the server in cyclic order, and each station can hold one message. The server starts by polling station 1. If there is a message, station 1 is served and then the server goes on to the next station. If there is no message, the server immediately starts polling the next station.    
Server …
Polling stations

23. Symmetric Polling System (results) [Younes et al. 2004]
serv1  P≥0.5(true U≤T poll1)
106
105
104
 = 10−6
 =  = 10−2
 = 0.5·10−2
103
Verification time (seconds)
102
101
100
10−1
10−2
102
104
106
108
1010
1012
1014
Size of state space

24. Symmetric Polling System (results) [Younes et al. 2004]
serv1  P≥0.5(true U≤T poll1)
106
105
104
 = 10−6
 =  = 10−2
 = 0.5·10−2
103
Verification time (seconds)
102
101
100
10−1
10−2
101
102
103 T

25. Symmetric Polling System (results) [Younes et al. 2004]
serv1  P≥0.5(true U≤T poll1)
102
n = 10
T = 40
101
Verification time (seconds)
==10−10
100
==10−8
==10−6
==10−4
10−1
==10−2
(=10−6)
10−4
10−3
10−2 

26. Tandem Queuing Network: Distributed Sampling
Use multiple machines to generate samples
m1: Pentium IV 3GHz
m2: Pentium III 733MHz
m3: Pentium III 500MHz
% samples
m1 only n m1 m2 m3
time 63 70 20 10
0.46 71 29
0.50
0.58
2047 60 26 14
1.28 30
1.46
1.93
65535 65 21
26.29 67 33
33.89
44.85
Each sample can be generated independently, so we can generate samples on multiple machines in parallel to speed up the statistical solution method further. This table shows the verification times for a tandem queuing system of varying size. We have used up to three machines, each machine running at a different speed. The speedup is essentially linear in the added CPU speed.

27. Summary
Acceptance sampling can be used to verify probabilistic properties of systems
Sequential acceptance sampling adapts to the difficulty of the problem
Statistical methods are easy to parallelize

28. Other Research Failure trace analysis Planning/Controller synthesis
“failure scenario” [Younes & Simmons 2004a]
Planning/Controller synthesis
CSL goals [Younes & Simmons 2004a]
Rewards (GSMDPs) [Younes & Simmons 2004b]

29. Tools Ymer Tempastic-DTP Statistical probabilistic model checking
Decision theoretic planning with asynchronous events

30. References
Wald, A Sequential tests of statistical hypotheses. Ann. Math. Statist. 16:
Younes, H. L. S., M. Kwiatkowska, G. Norman, and D. Parker Numerical vs. statistical probabilistic model checking: An empirical study. In Proc. TACAS-2004.
Younes, H. L. S., R. G. Simmons Probabilistic verification of discrete event systems using acceptance sampling. In Proc. CAV-2002.
Younes, H. L. S., R. G. Simmons a. Policy generation for continuous-time stochastic domains with concurrency. In Proc. ICAPS-2004.
Younes, H. L. S., R. G. Simmons b. Solving generalized semi-Markov decision processes using continuous phase-type distributions. In Proc. AAAI-2004.