1. Reachability Testing of Concurrent Programs1 Reachability Testing of Concurrent Programs Richard Carver, GMU Yu Lei, UTA

2. Reachability Testing of Concurrent Programs2 Outline  Introduction  The Reachability Testing (RT) Process  Computing Race Variants  An Example Scenario  Conclusion and Future work

3. Reachability Testing of Concurrent Programs3 A concurrent program...  Contains a set of cooperative threads/processes  multithreaded programs - threads synchronize by accessing shared memory  distributed programs – threads synchronize by exchanging messages  Is pervasive in modern software development  better resource utilization  increases computing efficiency  provides simple solutions to certain problems  Is notoriously difficult to test due to their non- deterministic behavior

4. Reachability Testing of Concurrent Programs4 Testing Strategies  Non-deterministic testing - execute the same program with the same input for multiple times:  easy, but inefficient, and some errors cannot be detected  Deterministic testing - select a set of test sequences and then force them to be exercised:  can detect more subtle errors, but requires additional effort for test sequence selection and runtime control  Prefix-based testing – a combination of non- deterministic and deterministic testing:  a test run executes deterministically at the beginning and then proceeds non-deterministically

5. Reachability Testing of Concurrent Programs5 Reachability testing... is based on prefix-based testing, and has the following main features:  Dynamic generation of test sequences - avoids constructing any static model  Dealing with partial orders directly – avoids generating redundant interleavings  Complete coverage on synchronization behavior – has important applications on program verification  Can be implemented in a platform-independent manner

6. Reachability Testing of Concurrent Programs6 Program Execution Model In our paper, a general representation of concurrent executions is provided, which allows RT to be applied to different types of programs, including semaphore- based, monitor-based, and message-passing programs.

7. Reachability Testing of Concurrent Programs7 Outline  Introduction  The RT Process  Computing Race Variants  An Example Scenario  Conclusion and Future work

8. Reachability Testing of Concurrent Programs8 SYN-sequence  Records the relative ordering of events that occur on a synchronization object  e.g., P/V events on a semaphore, entry/reentry events on a monitor, lock/unlock events on a lock, send/receive events on a channel  Used to characterize the synchronization behavior of an execution  the result of an execution is determined by the program text, the input data, and the SYN-sequence exercised by the execution

9. Reachability Testing of Concurrent Programs9 Concurrent Path Port M; T1T2T3 (1) M.send(10)(2) M.send(4)(3) x = M.receive (); (4) y = M.receive (); (5) z = x - y Assume that an execution of the above program exercises the sequence of statements: (1, 2, 3, 4, 5). Can we determine the value of z?

10. Reachability Testing of Concurrent Programs10 Space-Time Diagram T1 T2T3 s1 r1 s2 r2 T1 T2T3 s1 r1 s2 r2 z = 6 z = -6

11. Reachability Testing of Concurrent Programs11 The RT Framework 1. Execute a program P with input I non- deterministically to collect a SYN-sequence Q 2. Compute the race variants of Q, each of which is a prefix of one or more other SYN-sequences 3. For each variant, conduct a prefix-based test run to collect a new SYN-sequence Q’ 4. Repeat 2, 3, and 4 for each newly collected SYN- sequence Q’.

12. Reachability Testing of Concurrent Programs12 Tracing and Replay Additional runtime control is needed to conduct a prefix-based test run:  Synchronization library with semaphores, monitors, and message passing  Library has built-in tracing and replay capabilities  No modifications to the thread scheduler

13. Reachability Testing of Concurrent Programs13 Outline  Introduction  The RT Process  Computing Race Variants  An Example Scenario  Conclusion and Future work

14. Reachability Testing of Concurrent Programs14 Race Condition  Refers to the phenomenon where a receiving event can be synchronized with two or more sending events (in different executions)  e.g., the messages sent by two send events can be received by the same receive event  The race set of a receive event r consists of all the send events that r could possibly be synchronized with (in different executions).

15. Reachability Testing of Concurrent Programs15 Example T1 T2T3T4 s1 r1 s2 r2 s5 r3 s3 r5 T5 s4 r4 {s2} {s4, s5} {s5} {} Note that s1 is not in the race set of r2, because r2 may not even exist if r1 receives a message other than s1.

16. Reachability Testing of Concurrent Programs16 Identifying Races  Assign each event a vector timestamp to indicate the happens-before relation  Whether there is a race between two events can be determined by comparing their timestamps  Two timestamp assignment schemes described in paper:  Thread-centric: keeps a logic clock for each thread  Object-centric: keeps a version number for each synchronization object

17. Reachability Testing of Concurrent Programs17 Race Variant  Represents the beginning portion of one or more SYN-sequences that could have happened but didn’t  Derived from a SYN-sequence by changing one or more race outcomes in the sequence  If we change a race outcome, then all the events that could potentially be affected by the new outcome need to be removed

18. Reachability Testing of Concurrent Programs18 Race Table rjrj i j v Value v indicates how receiving event r in Q is changed to create variant V: v = -1: r is removed from V; v = 0: the send partner of r is left unchanged in V; and v > 0: in V, the send partner of r is changed to the v-th event in race_set(r, Q), Consists of one column for each receiving event with a non-empty race set Each row can be used to derive a unique, partially- ordered race variant of Q.

19. Reachability Testing of Concurrent Programs19 Example T1 T2T3T4 s1 r1 s2 r2 s5 r3 s3 r5 T5 s4 r4 {s2} {s4, s5} {s5} {} r1r3r4 001 01 02 100 11 T1 T2T3T4 s1 r1 s2 r2 s5 r3 s3 T5 s4 r4 v1 T1 T2T3T4 s1s2 r1 r3 s3 T5 s4 v5

20. Reachability Testing of Concurrent Programs20 Outline  Introduction  The RT Process  Computing Race Variants  An Example Scenario  Conclusion and Future work

21. Reachability Testing of Concurrent Programs21 An Example Scenario T1 T2 T3 T4 T5 T6 s1: send (T2, a); r1: x = recv (); r2: y = recv (); s3: send (T3, c); r4: u = recv (); s5: send (T5, e) s2: send (T3, b); r3: z = recv (); s4: send(T5, d); r5: v = recv (); if (u == e) { s6: send (T2, f); } (a) T1T2T3T4T5T6 Q0 s1 r1 s2 r2 s3 r3 s4 r4 s5 r5 T1T2T3T4T5T6 Q1 s1 r1 s2 r2 s3 r3 s4 r4 s5 r5 T1T2T3T4T5T6 Q3 s1 r1 s2 r2 s3 r3 s4 r4 s5 r5 s6 T1T2T3T4T5T6 Q4 s1 r1 s2’ r2 s3 r3’ s4 r4 s5 r5 s6 r3 T1T2T3T4T5T6 Q5 s1 r1 s2 r2 s3 r3’ s4 r4 s5 r5 s6 r3 T1T2T3T4T5T6 Q2 s1 r1 s2 r2 s3 r3 s4 r4 s5 r5 s6 (b)

22. Reachability Testing of Concurrent Programs22 Outline  Introduction  The RT Process  Computing Race Variants  An Example Scenario  Conclusion and Future Work

23. Reachability Testing of Concurrent Programs23 Conclusion  RT generates test sequences dynamically, which is more scalable than many model-based approaches. Our latest tool has almost constant memory requirements.  Compared to JPF and VeriSoft, which try to reduce the chances of generating redundant interleavings, RT deals with partial orders directly.  While most tools for concurrent programs testing rely on access to the thread scheduler and are thus platform dependent, our RT tool is platform independent.

24. Reachability Testing of Concurrent Programs24 Future Work  Test oracle – automatically evaluate test runs against properties specified in some logic formalism  Coverage-based RT - use coverage measures to selectively exercise a subset of the SYN- sequences  Model extraction – use RT to extract a complete model of the synchronization/communication behavior of a program  Real-time testing – add explicit time into the RT framework