1. Testing Concurrent Programs COMP 402 - Production Programming Mathias Ricken Rice University Spring 2009

2. Moore’s Law *

3. *

4. Timeliness CPU clock frequencies stagnate Multi-Core CPUs provide additional processing power –Multiple threads needed to use multiple cores Writing concurrent programs is difficult!

5. Programming Examples

6. Unit Testing Unit tests… –Test a part, not the whole program –Occur earlier –Automate testing –Serve as documentation –Prevent bugs from reoccurring –Help keep the shared repository clean Effective with a single thread of control

7. Foundation of Unit Testing Unit tests depend on deterministic behavior Known input, expected output… Success  correct behavior Failure  flawed code Outcome of test is meaningful

8. Problems Due to Concurrency Thread scheduling is nondeterministic and machine-dependent –Code may be executed under different schedules –Different schedules may produce different results Known input, expected output… Success  correct behavior in this schedule, may be flawed in other schedule Failure  flawed code Success of unit test is meaningless

9. Possible Solutions Programming Language Features –Ensuring that bad things cannot happen –May restrict programmers Lock-Free Algorithms –Ensuring that if bad things happen, it’s ok –May limit data structures available Comprehensive Testing –Testing if bad things happen in any schedule –Does not prevent problems, but does not limit solutions either

10. Contributions Improvements to JUnit –Detect exceptions and failed assertions in threads other than the main thread Annotations for Concurrency Invariants –Express complicated requirements about locks and threads Tools for Schedule-Based Execution –Record, deadlock monitor –Random delays, random yields

11. Improvements to JUnit Uncaught exceptions and failed assertions –Not caught in child threads

12. Sample JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } public void testAssertion() { public void testAssertion() { assertEquals(0, 1); assertEquals(0, 1); }} if (0!=1) throw new AssertionFailedError(); } Both tests fail.

13. Problematic JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Main thread Child thread Main thread Child thread spawns uncaught! end of test success!

14. Problematic JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Main thread Child thread Uncaught exception, test should fail but does not!

15. Improvements to JUnit Uncaught exceptions and failed assertions –Not caught in child threads Thread group with exception handler –JUnit test runs in a separate thread, not main thread –Child threads are created in same thread group –When test ends, check if handler was invoked

16. Thread Group for JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread invokes checks TestGroup’s Uncaught Exception Handler

17. Thread Group for JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread Test thread Child thread spawns uncaught! end of test failure! invokes group’s handler Main thread spawns and waitsresumes check group’s handler

18. Improvements to JUnit Uncaught exceptions and failed assertions –Not caught in child threads Thread group with exception handler –JUnit test runs in a separate thread, not main thread –Child threads are created in same thread group –When test ends, check if handler was invoked Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end Past tense: occurred!

19. Child Thread Outlives Parent public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread Test thread Child thread spawns uncaught! end of test failure! invokes group’s handler Main thread spawns and waitsresumes check group’s handler

20. Child Thread Outlives Parent public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread Test thread Child thread spawns uncaught! end of test success! invokes group’s handler Main thread spawns and waitsresumes check group’s handler Too late!

21. Improvements to JUnit Child threads are not required to terminate –A test may pass before an error is reached Detect if any child threads are still alive –Declare failure if test thread has not waited –Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.) Previous schedule is a test failure –Should be prevented by using Thread.join()

22. Enforced Join public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }); }); t.start(); … t.join(); t.start(); … t.join(); }} Thread t = new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}); t.start(); … t.join(); … throw new RuntimeException("booh!"); Test thread Child thread

23. Improvements to JUnit Child threads are not required to terminate –A test may pass before an error is reached Detect if any child threads are still alive –Declare failure if test thread has not waited –Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.) Previous schedule is a test failure –Should be prevented by using Thread.join()

24. Testing ConcJUnit Replacement for junit.jar or as plugin JAR for JUnit 4.2 –Available as binary and source at http://www.concutest.org/ http://www.concutest.org/ Results from DrJava’s unit tests –Child thread for communication with slave VM still alive in test –Several reader and writer threads still alive in low level test (calls to join() missing) DrJava currently does not use ConcJUnit –Custom-made TestCase class –Does not check if join() calls are missing

25. Conclusion Improved JUnit now detects problems in other threads –Only in chosen schedule –Needs schedule-based execution Annotations ease documentation and checking of concurrency invariants –Open-source library of Java API invariants Support programs for schedule-based execution

26. Future Work Schedule-Based Execution –Replay given schedule –Generate possible schedules –Dynamic race detection –Probabilities/durations for random yields/sleeps Extend annotations to Floyd-Hoare logic –Preconditions, postconditions –Representation invariants

27. Extra Slides

28. Test all possible schedules –Concurrent unit tests meaningful again Number of schedules (N) –t: # of threads, s: # of slices per thread detail Tractability of Comprehensive Testing

29. Extra: Number of Schedules back Product of s-combinations For thread 1: choose s out of ts time slices For thread 2: choose s out of ts-s time slices … For thread t-1: choose s out of 2s time slices For thread t-1: choose s out of s time slices Writing s-combinations using factorial Cancel out terms in denominator and next numerator Left with (ts)! in numerator and t numerators with s!

30. If program is race-free, we do not have to simulate all thread switches –Threads interfere only at “critical points”: lock operations, shared or volatile variables, etc. –Code between critical points cannot affect outcome –Simulate all possible arrangements of blocks delimited by critical points Run dynamic race detection in parallel –Lockset algorithm (e.g. Eraser by Savage et al) Tractability of Comprehensive Testing

31. Critical Points Example Thread 1 Thread 2 Local Var 1 Shared Var Lock lock access unlock All accesses protected by lock Local variables don’t need locking All accesses protected by lock

32. Fewer critical points than thread switches –Reduces number of schedules –Example:Two threads, but no communication  N = 1 Unit tests are small –Reduces number of schedules Hopefully comprehensive simulation is tractable –If not, heuristics are still better than nothing Fewer Schedules

33. Limitations Improvements only check chosen schedule –A different schedule may still fail –Requires comprehensive testing to be meaningful May still miss uncaught exceptions –Specify absolute parent thread group, not relative –Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) details

34. Extra: Limitations May still miss uncaught exceptions –Specify absolute parent thread group, not relative (rare) Koders.com: 913 matches ThreadGroup vs. 49,329 matches for Thread –Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) Koders.com: 32 method definitions for uncaughtException method back

35. Extra: DrJava Statistics 200473661036905116416196518.83%1071 Unit tests passedfailed not run Invariantsmetfailed % failed KLOC “event thread” 2006881881003441230616379611.03%12999 back