1. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 1 Getting Results From Testing Laura K. Dillon Software Engineering and Network Systems Lab Michigan State University www.cse.msu.edu/sens

2. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 2 Overview l Background: Testing l The Problem: Temporal Behavior Validation l A Solution: Graphical Interval Logic- Based Oracles

3. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 3 PREMISES l Software intensive systems must be tested extensively l Testing should be done early and often

4. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 4 Testing is more than running tests! Test results Reqs/ Design/ Code Formal Specifications Evaluate adequacy of tests Evaluate correctness of tests Create test plans Test plans Reqs/ Design/ Code Reqs/ Design/ Code Run tests

5. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 5 A specification-based test oracle Test results Formal Specifications Evaluate correctness of tests Checks that behaviors exhibited during testing satisfy the specifications

6. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 6 Overview l Background: Testing l The Problem: Temporal Behavior Validation l A Solution: Graphical Interval Logic- Based Oracles

7. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 7 The Oracle Problem l Answers the question: Were any of the test executions erroneous? l Concurrency adds a new dimension of complexity to this problem

8. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 8 A concurrent system … l Consists of asynchronous communicating threads of control: Tasks / Processes / Threads l Threads synchronize/communicate by sending messages or through sharing data l Temporal ordering matters

9. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 9 GAS STATION EXAMPLE [Helmbold, Luckham: 1985] Cus(1) Change Cus(N) Change Operator Prepay Charge Pump StartFinish Act...

10. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 10 Example Temporal Properties l Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas l If Cus(1) prepays before Cus(2), then Cus(1) gets to pump gas before Cus(2) gets to pump gas

11. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 11 l Large number of behaviors can be produced l Behaviors can be long l Behaviors that violate necessary temporal constraints are easily overlooked When validating behaviors of concurrent software systems...

12. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 12 Overview l Background: Testing l The Oracle Problem: Behavior Validation l A Solution: Graphical Interval Logic- Based Oracles

13. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 13 Overview of GIL Oracles l Desired temporal properties of execution traces are expressed formally in Graphical Interval Logic (GIL) l Execution traces are generated in testing l Oracle checks that execution traces satisfy GIL specifications l Displays anomalies to help user identify faults

14. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 14 Programmer defines “conditions” to trace Operator Prepay Charge Pump StartFinish Cus(1) Change Act Cus(N) Change Pay1 PayN Pump1

15. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 15 Formal comments define Pay1,..., PayN --% pay$n, for $n in ID_RANGE: Condition is --% triggered by --% begin accept Cus($n):Operator.prepay --% cancelled by --% begin activation Gas_Simulation --% end accept Operator:Cus($n).change Similarly: Pump1,..., PumpN

16. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 16 An execution trace induces a state sequence init acc C(1):Op.pre acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start end C(1):Pp.fini acc C(2):Pp.start acc Op:C(1).chan end C(2):Pp.fin acc C(1):Op.pre acc C(3):Pp.start acc Op:C(2).chan end C(3):Pp.fin acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start Pay1 Pump1 Pay2 Pump2...

17. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 17 DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas

18. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 18 DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) Pay1 || | [ ) Pay1 Pump1

19. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 19 DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) Pay1 || | [ ) Pay1 Pump1

20. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 20 DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) Pay1 || | [ ) Pay1 Pump1

21. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 21 | DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) Pay1 || [ ) Pay1 Pump1

22. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 22 | DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) Pay1 || [ ) Pay1 Pump1

23. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 23 Fair1.2: If Cus(1) prepays before Cus(2), then Cus(1) gets to pump before Cus(2) gets to pump

24. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 24 Fair1.2: | [ ) Pump1 Pump2 [ ) Pay1 || Pay2 Pay1 ||

25. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 25 Fair1.2: [ ) Pay1 || Pay2 || | [ ) Pump1 Pump2 Pay1

26. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 26 Fair1.2: [ ) Pay1 || Pay2 || | [ ) Pump1 Pump2 Pay1

27. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 27 Fair1.2: [ ) Pay1 || Pay2 || | [ ) Pump1 Pump2 Pay1

28. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 28 Fair1.2: [ ) Pay1 || Pay2 || | [ ) Pump1 Pump2 Pay1

29. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 29 init acc C(1):Op.pre acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start end C(1):Pp.fin acc C(2):Pp.start acc Op:C(1).chan end C(2):Pp.fin acc C(1):Op.pre acc C(3):Pp.start acc Op:C(2).chan end C(3):Pp.fin acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start Pay1 Pump1 Pay2 Pump2 [ ) Pay1 || [ ) Pay1 Violates DoesPump1

30. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 30 A GIL formula, f, determines a deterministic finite state automaton, D f, such that, for every finite state sequence, s,... s f if and only if Accept(s, D f )

31. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 31 5 DFA for DoesPump1 01 2 43 P1 M1 P1M1 P1M1 P1M1 P1 P1 = Pay1 M1 = Pump1 true P1

32. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 32 Checking an execution trace init acc C(1):Op.pre acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start end C(1):Pp.fini acc C(2):Pp.start acc Op:C(1).chan end C(2):Pp.fin acc C(1):Op.pre acc C(3):Pp.start acc Op:C(2).chan end C(3):Pp.fin acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start Pay1 Pump1 Pay2 Pump2... 0233444422333333

33. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 33 Summary l Testing concurrent systems is complicated by the need to check temporal properties l Temporal oracle can be produced from GIL specifications  Checks that execution traces satisfy GIL specifications  Displays anomalies to help user identify faults

34. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 34 Yet to do l Refine heuristics for displaying failures l Conduct case studies/user studies l Integrate oracles into a proactive debugger

35. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 35 Acknowledgements Thanks! T his work was supported in part by l NSF grants EIA-0000433, CDA-9617310, and CCR-9896190 l Department of the Navy, Office of Naval Research under grant N00014-01-1-0744

36. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 36 Meridian: An Integrated Toolkit for Developing Interactive Distributed Applications l NSF Experimental Partnership Program l Betty Cheng, Laura Dillon, Phillip McKinley, Kurt Stirewalt

37. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 37 Interactive Distributed Applications (IDAs) l Examples:  On-board driver/pilot navigation systems.  Computer-supported collaborative work environments.  Distributed interactive simulation. Interact with users. Processing/data distributed across a network.

38. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 38 Characteristics of IDAs l Interactivity:  Must interact with one or more human users.  Design requires prototyping and experimentation. l Concurrency:  Comprise levels of communicating, concurrent components.  Analysis requires formal reasoning. l Reuse:  IDAs built primarily from reusable components.  E.g., comm. protocols, resource managers, data displays.  Design involves selecting/specializing components.

39. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 39 Meridian goals l Advance development technology for IDAs.  Formalize development artifacts and relationships between them  Libraries of reusable components l Provide end-to-end automation techniques. l Impact practice:  Must complement existing design methods and notations, e.g. UML

40. L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 40 Meridian Vision Model Editing Specification Analysis Design Processing Testing/ Simulation IDA ModelsIDA Constraints IDA Interface Requirements IDA Reuse Repository IDA External Parameters Specifications Refined Specifications Code Feedback User Requirements Test Cases, Oracles