Ymer: A Statistical Model Checker Håkan L. S. Younes Carnegie Mellon University

YounesYmer: A Statistical Model Checker2 Probabilistic Model Checking Given a model M, a state s, and a property , does  hold in s for M ? Model: stochastic discrete event system Property: probabilistic temporal logic formula Example:  ≥0.1 [   ≤5 full ]

YounesYmer: A Statistical Model Checker3 Statistical Solution Method Use acceptance sampling to verify probabilistic properties Hypothesis: P ≥  [  ] Observation: verify  over a sample path Bounds on probability of verification error Probability of false negative: ≤  Probability of false positive: ≤ 

YounesYmer: A Statistical Model Checker4 Error Bounds Actual probability of  holding Probability of error when verifying P ≥  [  ]    p1p1 p0p0 Indifference region 22

YounesYmer: A Statistical Model Checker5 Ymer at a Glance Supports time-homogeneous generalized semi-Markov processes Limited to time-bounded properties Distributed acceptance sampling (even with sequential acceptance sampling) Purely statistical approach for verifying nested probabilistic statements

YounesYmer: A Statistical Model Checker6 Distributed Acceptance Sampling Master Acceptance Sampling Slave simulation Slave simulation  SlaveMaster register model & property observation done 

YounesYmer: A Statistical Model Checker7 Avoiding Sample Bias Process observations as they come in? No, bias against observations that take a long time to generate (long sample paths) Process observations according to a predetermined schedule Schedule: Received: 

YounesYmer: A Statistical Model Checker8 Case Study: Symmetric Polling System Single server, n polling stations Stations are attended in cyclic order Each station can hold one message State space of size O(n·2 n ) Server … Polling stations

YounesYmer: A Statistical Model Checker9 Results Percent of single machine 100 Size of state space Machine 1: 733 MHz Pentium III Machine 2: 500 MHz Pentium III

YounesYmer: A Statistical Model Checker10 Nested Probabilistic Statements: Robot Grid World Probability is at least 0.9 that goal is reached within 100 seconds while periodically communicating  ≥0.9 [  ≥0.5 [   ≤9 comm]  ≤100 goal ]

YounesYmer: A Statistical Model Checker11 Statistical Verification of Nested Probabilistic Statements Cannot verify path formula without some probability of error Probability of false negative: ≤  ′ Probability of false positive: ≤  ′ Observation error

YounesYmer: A Statistical Model Checker12 Performance Considerations Verification error is independent of observation error Pick observation error to minimize effort The same state may be visited along multiple sample paths Memoize verification results to avoid repeated effort

YounesYmer: A Statistical Model Checker13 Robot Grid World (results) Verification time (seconds) 10 −2 10 − Size of state space  ≥0.9 [  ≥0.5 [   ≤9 comm]  ≤100 goal ]  =  = 0.05  =  = 10 −2

YounesYmer: A Statistical Model Checker14 Robot Grid World: Effect of Memoization Sample size Unique/visited states Size of state space

YounesYmer: A Statistical Model Checker15 Availability Source code is released under GPL