1. Use of Verification for Testing and Debugging of Complex Reactive Systems Mark Trakhtenbrot Holon Academic Institute of Technology

2. Why development of reactive systems is difficult? Typically:  Critical applications (embedded RT controllers, …)  Must fulfill tough requirements (safety, timeliness, … ) Main problem: complex behavior  Intricate event-driven interaction with the environment; deep dependencies that are difficult to capture  Concurrency  Timing factors Result: design, analysis and validation are very difficult

3. Two approaches to system validation Testing - analyze behavior of the actual system, under selected test scenarios - system is examined in its real physical and operational environment Verification - “exhaustive testing” of system model under various assumptions on environment - ideally, fully certifies the system if property violated, generates the relevant test scenario

4. Testing - problems  Full testing is impossible  Proper coverage; popular criteria (all states, all branches, all data flows) are important but not sufficient  Good selection of tests is often based on tester’s experience and intuition  In real systems: huge amount of tests is needed; manual creation is not realistic  Tested property and test itself are often described in a non-formal, incomplete way - problem in interpretation of results - difficult to reproduce a failed run

5. Verification – challenges and limitations  Need to formalize system requirements Checked properties and environment assumptions are expressed in a formal assertion language; for example: - Temporal Logic to express order and timing constraints - templates for typical properties (response, precedence, …)  Need to create a formal model of the actual system  Problem: adequate abstraction from details that are not relevant to the checked property under-approximation: model violates the spec  system violates the spec over-approximation: model fulfils the spec  system fulfils the spec

6.  Verification limitations: - model need be final-state (restrict unbounded data, dynamically created objects,…) - exponential size of state space to be explored (challenge: find assumptions that are strong enough to narrow the search, while still covering interesting scenarios) - slow / unrealistic for big real-world models  Testing : - no "finite state" limitation (may connect to real input sources, create unlimited amount of objects, …) - no limitations on system size - compiled code runs fast

7. Combining T and V: how to achieve better results?  Typically: - testing first, to gain a good level of stability - verification for certification  Model checking used for: - test generation (e.g. to achieve the state coverage: systematic check of state reachability, with scenario generation) - generate scenario leading to test’s initial configuration (satisfying the test’s pre-condition)  Suggested: - reproduction of failed runs

8. Reproduction of failed runs  Typically, only partial info related to the failed run is available: - the remembered one, or thought to be relevant - even when a trace available, it may be not detailed enough (decision on what to trace is taken in advance)  A common problem, regardless the system type or reporter type (customer or tester)

9. Reproduction of failed runs  Especially serious for reactive real-time systems: - even small changes in order / time of input events may have significant impact on system reaction - time-related factors are especially hard to reproduce - concurrency causes non-determinism in the execution order  Challenge: how to utilize partial knowledge to reproduce and localize the error

10. Suggested process  Analyze the available knowledge about “irreproducible” system execution that violates property P - analysis of recorded trace - remembered observations  Use it to set up model checking of the system model that verifies whether P holds  Translate the found model-level counter-example into a code-level scenario  Run the code under this scenario and debug it

11. Considered framework: Model-based development  formal model of system (behavior, structure, …)  statecharts to capture behavior  executable model; a basis for automated tools (simulation, verification, code generation)  model-level analysis while code execution (backward traceability)  since code is derived from the model, they are (hopefully) always in synch

12. Model-based analysis with Statemate Simulation: check, record and replay test scenarios Verification: Model Checker / Model Certifier (developed by W.Damm & Co) - racing and non-determinism - reachability (drive-to-property) analysis - check of temporal properties (expressed with predefined templates), including timing constraints - when a property violated, generates a simulation script to reproduce the failure

13. Important abstractions Both use simulated time SIM synch time (clock-driven systems): every step takes the same time SIM asynch time (event-driven systems): when in stable state, advance clock, read inputs & run till stabilization; zero-time chain reaction Basic assumption: system is fast enough to complete its reaction before arrival of next stimuli

14. Important abstractions Maximum parallelism (full synchronization): At each step, all enabled transitions in all active statecharts are executed simultaneously

15. Analysis with generated code  Code synthesis: executable C (for host or target OS) derived from the model  What can we gain from testing of derived code if it so closely relates to the model?  Overcome limitations of simulation and verification - Real Time execution, get rid of SIM time assumptions - Models augmented with design attributes (tasks/events mapping,…) * generate highly optimized code for realistic target environment * tasks run asynchronously; full synchronization only within every separate task These can not be addressed by either simulation or verification

16. Suggested process  Analyze the available knowledge about “irreproducible” system execution that violates property P - analysis of recorded trace - remembered observations  Use it to set up model checking of the system model that verifies whether P holds (used to define environment assumptions)  Translate the found model-level counter-example into a task that runs in parallel with the code and feeds as needed: right inputs at right time points (use CodeGen API)  Debug and localize the problem in the model, based on code- to-model backward traceability

17. What can / can’t be achieved this way?  The approach improves the ability to reproduce bugs, but it doesn’t provide an ultimate solution  E.g. under the simulated time assumption (“the system is fast enough …”) MC may conclude that not P is never reached, even though P is violated in real-time execution  The opposite is also possible. Hence the approach requires that violation of P in model will be re-checked, under the found scenario, also in code.

18. Example: Rail Cross Control

19. Rail Cross Control (cntd.) Trains - speed: constant 10 … 160 km/h; length: 1,000 … 3,500 meters - distance between two trains on the same track: at least 4000 meters. Barriers - can work properly or be damaged (then need be repaired) - normally, opened / closed within 5 sec. - should be closed 40 sec before a trains enters the cross - should be opened when a train leaves the cross, unless another train is 55 sec or less before entering the cross Lights - are absolutely reliable - modes: off, blinking, hold on

20. Safe Cross Property If there is a train in any of the cross sectors, then for each of the tracks: - either the track's barrier is closed - or its light holds on

21. Functional view of the system

22. Cross control

23. Barrier control

24. Safe Cross Property Safe_Cross == IN_CROSS1 or IN_CROSS2  ( (BAR1_CLOSED or LIGHT1_ON) and (BAR2_CLOSED or LIGHT2_ON) )

25. Testing the generated code Code is generated directly from the model (no design attributes used) Tested scenarios: A single train on some of the tracks : OK Two trains on the same track, with required minimum distance between them : OK Two trains on different tracks : - 2 nd entered when the 1 st already exited from the cross: OK - 2 nd entered at some point before the 1 st reached the cross: FAILURE

26. Defining the Model Checker analysis Type of analysis: Drive-to-property (not Safe_Cross) To restrict the search, used the following techniques: - Freezing variable values - Simple assumptions (boolean formulae over model elements) - Statechart-modeling of general (independent of specific scenario) facts about environment behavior Freeze: - TRAINi_SPEED : 160, TRAINi_LENGTH : 3500 (if they were not known, then had to properly define the variables’ ranges) - Barriers were not damaged, hence input events OUT_OF_USE* and REPAIR* are frozen at value false

27. Defining the Model Checker analysis Simple assumptions: Violation observed with only one train on each of the tracks: ENTER_TRACKi  not (IN_STARTi or IN_CROSSi or IN_FINISHi) (i=1,2) Train on the 2 nd track entered when there was a train on the 1 st track, and it didn’t reach yet the cross segment: ENTER_TRACK2  IN_START1 and not (IN_CROSS1 or IN_FINISH1)

28. Defining the Model Checker analysis Environment modeling

29. Model checking result and its use for code testing Model Checker: - found reachable configuration satisfying not Safe_Cross - created a script that drives simulation to that configuration Testing of generated code: - used CodeGen API to translate the script into a driver-task that runs aside the system code and feeds it according to the found scenario - instrumented the code for creation of execution traces (used the relevant CodeGen’s option) - run black-box test (traced in/out events); Safe_Cross failed - run white-box test (traced some internals), to localize the problem

30. Black-box testing -- at 0:00:08.020 : event ENTER_TRACK1 generated -- at 0:00:24.985 : event ENTER_TRACK2 generated -- at 0:00:27.198 : condition BAR1_CLOSED changed value to true -- at 0:00:27.248 : condition BAR2_CLOSED changed value to true -- at 0:00:27.287 : condition LIGHT1_FLASHING changed value to true -- at 0:00:27.331 : condition LIGHT2_FLASHING changed value to true -- at 0:01:03.801 : event ENTER_CROSS1 generated -- at 0:01:04.732 : event ENTER_FINISH1 generated -- at 0:01:20.765 : event ENTER_CROSS2 generated -- at 0:01:21.646 : event ENTER_FINISH2 generated -- at 0:02:22.544 : event EXIT_START1 generated -- at 0:02:23.455 : event EXIT_CROSS1 generated -- at 0:02:27.572 : condition BAR1_CLOSED changed value to false -- at 0:02:27.611 : condition BAR2_CLOSED changed value to false -- at 0:02:27.654 : condition LIGHT1_OFF changed value to true -- at 0:02:27.696 : condition LIGHT2_OFF changed value to true

31. Conclusions  The approach improves the ability to reproduce bugs that are otherwise hardly reproducible  Applicable for more complex properties, such as ALWAYS (BAR_DOWN  EVENTUALLY (10) in[CLOSED] or LIGHT_ON) Based on support by Model Certifier  Limitations due to potential semantic differences between model and generated code; model checking results must be re-tested with generated code  Test execution for derived code can be run-time verified using a monitor automatically created from the formal system assertion (MT & M.Auguston, 2004)