1. Run Time Monitoring of Reactive Systems Mikhail Auguston Naval Postgraduate School maugusto@nps.edu Mark Trakhtenbrot Holon Academic Institute of Technology ilmarktr@yahoo.com

2. Outline Problems in development of reactive systems Model-based development and analysis Our approach:  TL-based assertion language  Automatic generation of monitors for run-time verification Outline of translation scheme Further steps

3. Why development of reactive systems is difficult? Main problem: complex behavior  Intricate event-driven interaction with the environment  Concurrency  Timing factors Typically:  Critical applications (embedded RT controllers, …)  Must fulfill tough requirements (safety, timeliness, … )

4. Challenges  Capture requirements to express the expected behavior  Check that system meets the requirements  ideal: full verification  more realistic: testing * how to ensure a proper coverage? * how we know the test result is / isn’t correct? can this task be automated?

5. So, what to do? Apply formal methods! Precise spec of behavior:  Variants of TL [Pnueli, …] (future/past modalities; timing)  Patterns for typical properties [Avrunin, …] (response, precedence, existence, absence,…) Model-based development:  formal model of system (behavior, structure, …)  statecharts to capture behavior; a renown standard  executable model; a basis for automated tools  model-level analysis (closer to problem domain)

6. Expressing system properties Properties expressed using a special language, e.g. Temporal Logic Examples: ALWAYS (cross_empty or barrier_closed) ALWAYS (request  EVENTUALLY allocate_resource) Meaning: the property holds in any computation More complicated: property holds on part of execution paths (need quantifiers on path) Time-related properties: ALWAYS (p  EVENTUALLY (5) q) Q should happen within 5 time units after P.

7. Expressing system properties May be difficult to use without a strong logic background Combination of English with C-like expressions: Event A follows event B after less than 10 time units Patterns for typical behavioral properties E.g. Precedence patterns (S precedes P): Before R EVENTULLY P  (not P Until (S or R)) After Q Between Q and R After Q until R Pattern defines scope in which the property should hold

8. Statechart for Early Warning System

9. Model-based analysis with Statemate Simulation: check, record and replay test scenarios Verification: based on predefined assertion templates Code synthesis: executable C, for host or target OS Watchdog charts: - not part of the system model, but co-executed alongside it - used to drive and monitor system execution (in simulation or generated code run)

10. Model-based analysis Simulation: - Analysis of quality attributes (e.g. stability with respect to changes of the data) - Analysis of events occurring simultaneously (to resolve scheduling and priorities issues in the system) - "What if" checks (alternative decisions, "impossible" scenarios) - Check rarely occurring situations - Check behavior of the system under extreme values. - Influence of various uncontrolled and unpredictable factors - Time-related analysis (performance, very fast / very slow processes)

11. Model-based analysis Verification: Generic properties (e.g. existence of racing, or non- determinism) Model-specific properties, e.g. reachability of a state in which certain condition is fulfilled (formulated in terms of model elements) If violation found: a script is generated, driving simulation to the problematic state

12. Monitor charts Monitor (watchdog) chart: - Direct access (visibility) to all model elements - Support for box black-box and white-box testing Sample assertion: Processing of a request must be accomplished within 5 seconds, and before receiving of the next request Violation: entering into ERROR state WaitProcessError Request Processing Done Request or Timeout(5) /Violation Notification

13. Suggested approach: Statemate-based run-time verification Goal: support effective testing (finding errors rather than validation / showing correctness) Monitoring of code generated from system model Main features:  Assertion language to specify: - allowed and forbidden behavior - actions to be taken (collect and report diagnostic data)  Automatic creation of monitor charts from assertions Used Statemate capabilities:  Co-translation of model and monitor charts to C  Co-execution of generated code components

14. Advantages of the approach Flexible assertion language vs. predefined patterns Smooth transition: assertions  executable monitors Overcome limitations of simulation and verification

15.  Analyze models reflecting design decisions (tasks/events mapping,…), in realistic target environment  Real time analysis - rather than SIM time SIM synch time (clock-driven systems): all reactions assume same duration SIM asynch time (event-driven systems): all reactions assume zero duration  No "finite state" limitation; may connect to real input sources  No limitations on model size

16. Assertion language  Boolean expressions: system state properties in(S) and (x > 5) ; entered(S)  started(A) ; ch(d)  Regular expressions: describe sequences of states [SELECT (Open | Read | Write | Close)] SATISFY Open (Read | Write )* Close  Temporal formulas: order properties ALWAYS (request  EVENTUALLY response) ALWAYS (execute  SOMETIME_WAS do_set_up)  Timed temporal formulas: real-time restrictions ALWAYS (request  EVENTUALLY (10) response)  Actions: to trace interesting events (e.g. property violation) reports, profiling info (based on objects’ attributes)

17. Example - assertion for EWS ALWAYS (OUT_OF_RANGE  EVENTUALLY (15) (RESET or started(PRINT_ALARM))) ON_FAIL [printf(“Violation after occurrence of OUT_OF_RANGE at time %f “, OUT_OF_RANGE.occur_time)]

18. More on temporal properties Manna & Pnueli Classification (P – past formula) :  Safety: ALWAYS (P), Guarantee: EVENTUALLY (P)  Obligation: Boolean combination of safety & guarantee  Response: ALWAYS (EVENTUALLY(P))  Persistence: EVENTUALLY (ALWAYS(P))  Reactivity: Bool. combination of response & persistence  Any TL formula is equivalent to a Reactivity formula; others used for convenience & flexibility Limitations:  nesting depth <= 2 for unrestricted future operators  unrestricted operator not embedded in a restricted one

19. Sample monitor construction ALWAYS (OUT_OF_RANGE  EVENTUALLY (15) (RESET or started(PRINT_ALARM))) / ERROR_MESSAGE

20. Translation of restricted formula ALWAYS (N) P

21. Translation of unrestricted formula ALWAYS (P)

22. Translation of unrestricted formula EVENTUALLY (P)

23. Translation of unrestricted formula ALWAYS (EVENTUALLY (P) )

24. Semantic issues  Monitoring: deals with finite runs; semantics of TL – based on infinite runs  [Eisner, et.al.]: reasoning on truncated executions  Assume the run finishes 4 sec after the last p ALWAYS (p  EVENTUALLY(10) q) ; false if no q after the last p ALWAYS (p  ALWAYS(10) q) ; true if always q after the last p  In general: user responsible to provide proper actions

25. Further steps Implementation:  done - experiments with manually created monitors  next step – actual implementation A more friendly assertion language  e.g. combine English with formulas Optimized translation for certain types of assertions Apply to other design paradigms using statecharts  e.g. in OO: monitor systems with dynamically created objects (problematic for model checking)