1. ConcJUnit: Unit Testing for Concurrent Programs COMP 600 Mathias Ricken Rice University August 24, 2009

2. 2 Brian Goetz, Java Concurrency in Practice, Addison-Wesley, 2006

3. 3 Concurrency Practiced Badly [1]

4. 4 Unit Tests… Occur early Automate testing Keep the shared repository clean Serve as documentation Prevent bugs from reoccurring Allow safe refactoring

5. 5 Existing Unit Testing Frameworks JUnit, TestNG Don’t detect test failures in child threads Don’t ensure that child threads terminate Tests that should fail may succeed

6. 6 ConcJUnit Replacement for JUnit  Backward compatible  Just replace junit.jar file Detects failures in all threads Warns if child threads outlive main thread Available at www.concutest.orgwww.concutest.org

7. 7 Sample JUnit Tests public class Test extends TestCase { public void testException() { throw new RuntimeException("booh!"); } public void testAssertion() { assertEquals(0, 1); } } if (0!=1) throw new AssertionFailedError(); } Both tests fail.

8. 8 JUnit Test with Child Thread public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); } new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); throw new RuntimeException("booh!"); Main thread Child thread Main thread Child thread spawns uncaught! end of test success!

9. 9 JUnit Test with Child Thread public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); } new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); throw new RuntimeException("booh!"); Uncaught exception, test should fail but does not! By default, no uncaught exception handler installed for child threads

10. 10 Changes to JUnit Thread group with exception handler  JUnit test runs in a separate thread, not main thread  Child threads are created in same thread group  When test ends, check if handler was invoked

11. 11 JUnit Test with Child Thread public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); } new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); throw new RuntimeException("booh!"); invokes checks TestGroup’s Uncaught Exception Handler

12. 12 JUnit Test with Child Thread public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); } new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); throw new RuntimeException("booh!"); Test thread Child thread uncaught! end of test Main thread spawns and joinsresumes check group’s handler invokes group’s handler failure!

13. 13 Child Thread Outlives Parent public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); } new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); throw new RuntimeException("booh!"); Test thread Child thread uncaught! end of test success! invokes group’s handler Main thread check group’s handler Too late!

14. 14 Well-Formed Tests Uncaught exceptions and failed assertions in all threads cause failure If the test is declared a success before all child threads have ended, failures may go unnoticed Therefore, all child threads must terminate before test ends  Check for living child threads after test ends

15. 15 Check for Living Child Threads public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); } new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); throw new RuntimeException("booh!"); Test thread Child thread uncaught! end of test failure! invokes group’s handler Main thread check for living child threads check group’s handler

16. 16 Correctly Written Test public class Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { public void run() { /* child thread */ } }); t.start(); t.join(); } Thread t = new Thread(new Runnable() { public void run() { /* child thread */ } }); t.start(); t.join(); // wait until child thread has ended /* child thread */ Test thread Child thread end of test success! Main thread check for living child threads check group’s handler [4]

17. 17 Well-Formed Tests Use Join Uncaught exceptions and failed assertions in all threads cause failure Therefore, all child threads must terminate before test ends Without join() operation, a test may get “lucky”  Require all child threads to be joined

18. 18 Fork/Join Model Parent thread joins with each of its child threads May be too limited for a general-purpose programming language Child thread 1 Child thread 2 Main thread

19. 19 Join with All Offspring Threads Main thread joins with all offspring threads, regardless of what thread spawned them Child thread 1 Child thread 2 Main thread

20. 20 Join with Last Thread of Chain Chain of child threads guaranteed to outlive parent Main thread joins with last thread of chain Child thread 1 Child thread 2 Main thread Child thread 3

21. 21 Generalize to Join Graph Threads as nodes; edges to joined thread Test is well-formed as long as all threads are reachable from main thread Child thread 1 Child thread 2 Main thread Child thread 3 MT CT1 CT2 CT3

22. 22 Child thread 1 Child thread 2 Main thread MT CT1 CT2 Child thread 1 Child thread 2 Main thread MT CT1 CT2 Join Graph Examples

23. 23 Child thread 1 Child thread 2 Main thread MT CT1 CT2 Unreachable Nodes An unreachable node has not been joined  Child thread may outlive the test

24. 24 this current thread MT CT Constructing the Graph In Thread.start() – add node for this End of Thread.join() – add edge from Thread.currentThread() to this [3]

25. 25 Modifying the Java Runtime Changing Thread.start() and join()  Need to modify Java Runtime Library  Utility to process user’s rt.jar file  Put new jar file on boot classpath: -Xbootclasspath/p:newrt.jar

26. 26 Modifying the Java Runtime May not always be possible  ~30 MB hard drive space needed  Access to boot classpath option required ConcJUnit written to work without it  Just no join graph and “lucky” warnings [5]

27. 27 Evaluation JFreeChart  All tests passed; tests are not concurrent DrJava: 900 unit tests  Passed:880  No join: 1  Lucky: 18  Timeout: 1  Runtime overhead: ~1 percent

28. 28 Limitations Only checks chosen schedule  A different schedule may still fail Example: Thread t = new Thread(…); if (nondeterministic()) t.join(); [2]

29. 29 Future Work Randomly insert sleeps or yields  Example:If a notify() is delayed, a wait() may time out. Can detect a number of sample problems Record schedule, replay if test fails  Makes failures reproducible if found [6]

30. 30 Thank you! www.concutest.org

31. 31 Extra Slides

32. 32 Notes 1. Probably not caused by concurrency problems; just a metaphor. ←← 2. Also cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) ←← 3. Only add edge if joined thread is really dead; do not add if join ended spuriously. ←←

33. 33 public class Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } Thread t = new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } throw new RuntimeException("booh!"); Loop since join() may end spuriously 4. Spurious Wakeup ←

34. 34 Thread Creation Context In Thread.start() – also record stack trace of Thread.currentThread(  Easy to find source code of a child thread that is not joined  Also available for uncaught exception stack traces 5.

35. 35 Creation Context Example class Main { void foo() { // which one? new Helper().start(); //... } AssertionError: at Helper.m(Helper.java:2) at Helper.run(Helper.java:3) Started at: at Main.foo(Main.java:4) at Main.bar(Main.java:15) at Main.main(Main.java:25) class Helper extends Thread { void m() { Assert.fail(); } public void run() { m(); } } ←

36. 36 Notes 6. Have not studied probabilities or durations for sleeps/yields: One inserted delay may negatively impact a second inserted delay Example: If both notify() and wait() are delayed. ←←

37. 37 Many Thanks To… My advisor  Corky Cartwright My committee members  Walid Taha  David Scott  Bill Scherer NSF and Texas ATP  For providing partial funding

38. 38 Concurrency Problems Not Found During Testing [1]