1. A Framework for Testing Concurrent Programs COMP 600 Mathias Ricken Rice University August 27, 2007

2. Unit Testing Program ? Difficult Sub- program Sub- program Sub- program ??? Less difficultEven less difficult ?

3. Unit Testing Unit tests… –Test a part, not the whole program –Occur earlier –Automate testing –Keep the shared repository clean –Serve as documentation –Prevent bugs from reoccurring Effective with a single thread of control

4. Foundation of Unit Testing Unit tests depend on deterministic behavior Known input, expected output… Success  correct behavior Failure  flawed code Outcome of test is meaningful

5. Problems Due to Concurrency Thread scheduling is nondeterministic and machine-dependent –Code may be executed under different schedules –Different schedules may produce different results Known input, expected output… Success  correct behavior in this schedule, may be flawed in other schedule Failure  flawed code Success of unit test is meaningless

6. Timeliness of the Problem Many programs already use concurrency –Often hidden, as part of GUI Most speed increases are due to multiple cores on one chip –Clock speeds have not increased much Increased use of concurrency in the future

7. Possible Solutions Programming Language Features –Ensuring that bad things cannot happen –May restrict programmers Lock-Free Algorithms –Ensuring that if bad things happen, it’s ok –May limit data structures available Comprehensive Testing –Testing if bad things happen in any schedule –Does not prevent problems, but does not limit solutions either details

8. Tractability of Comprehensive Testing Deciding whether any given program contains an error is undecidable –Reduction to the halting problem Does not imply undecidable for all programs

9. Number of Schedules Test all possible schedules –Concurrent unit tests meaningful again Number of schedules (N) –t: # of threads, s: # of slices per thread Exponential in both s and t details

10. Critical Points If program is race-free, we do not have to simulate all thread switches –Threads interfere only at “critical points”: lock operations, volatile variables, etc. –Code between critical points cannot affect outcome –Simulate all possible arrangements of blocks delimited by critical points

11. Critical Points Example Thread 1 Thread 2 Local Var 1 Shared Var Lock lock access unlock All accesses protected by lock Local variables don’t need locking All accesses protected by lock

12. Fewer Schedules Fewer critical points than thread switches –Reduces number of schedules –Example:Two threads, but no communication  only one schedule required Unit tests are small –Reduces number of schedules Hopefully comprehensive simulation is tractable –If not, heuristics are still better than nothing

13. Parts of the Framework Improvements to JUnit –Detect exceptions and failed assertions in threads other than the main thread Annotations for Concurrency Invariants –Express complicated requirements about locks and threads Tools for Schedule-Based Execution –Record, deadlock monitor –Random delays, random yields

14. Concurrency Invariants Has to be called in event thread –TableModel, TreeModel May not be called in event thread –invokeAndWait() Have to acquire readers/writers lock –AbstractDocument –DrJava’s documents

15. Invariants Difficult to Determine May be found in –Javadoc comments –Only in internal comments –Whitepapers Often not documented at all Errors not immediately evident Impossible to check automatically

16. Java Annotations Add invariants as annotations @NotEventThread public static void invokeAndWait( Runnable r) { … } Process class files –Find uses of annotations –Insert bytecode to check invariants at method beginning

17. Advantages of Annotations Java Language constructs –Syntax checked by compiler Easy to apply to part of the program –e.g. when compared to a type system change Light-weight –Negligible runtime impact if not debugging (slightly bigger class files) Automated Checking

18. Predicate Annotations In annotation definition, specify static boolean Java method –Method must be callable from every context  completely static and public Data in annotation, method arguments and value of this passed when method invoked

19. Predicate Annotation Example public class TestCode { @TestAllowed(allowed=true) @TestAllowed(allowed=true) public void test(String param){…} public void test(String param){…}}… TestCode t = new TestCode(); t.test("xxx"); @PredicateLink( value=Predicates.class, value=Predicates.class, method="example", method="example", arguments=true) arguments=true) public @interface TestAllowed { boolean allowed; boolean allowed;} public class Predicates { public static boolean example( public static boolean example( Object this0, Object this0, String param, String param, boolean allowed) { boolean allowed) { return (allowed)? // this0==t return (allowed)? // this0==t (param.equals("test")): // param=="xxx" (param.equals("test")): // param=="xxx" (!param.equals("test")); // allowed==true (!param.equals("test")); // allowed==true} zoom

20. Provided Annotations @NotEventThread@NotThreadWithName@NotSynchronizedThis@NotSynchronizedArgument@NotNullArgument@DistinctArguments Inverses, conjunctions, disjunctions

21. Invariant Inheritance Invariants apply to the method and all overriding methods in subclasses  Methods can have invariants defined elsewhere All annotations describe requirements for the client (and, due to subclassing, for subclasses) –Allows frameworks to describe requirements

22. Invariant Subtyping To maintain substitutability, subclasses may not strengthen invariants Invariants can be modeled as special input parameter –Tuple of invariants (“record” in λ calculus [Pierce]) –Subtyping rules for records declare the “wider” record as subtype –In function types, parameter types are contravariant

23. Invariant Subtyping Analyze methods with invariants Invariants subtyping: A <@ B <@ C I A = {}, I B = {inv 1 }, I C = {inv 1,inv 2 }; I C <: I B <: I A F A = I A → ·, F B = I B → ·, F C = I C → ·; F A <: F B <: F C Java subtyping: C <: B <: A class A { void f() void f() { … }; { … };} class B extends A { extends A { @Inv1 @Inv1 void f() void f() { … }; { … };} class C extends B { extends B { @Inv2 @Inv2 void f() void f() { … }; { … };}

24. Detection of Subtyping Problems If Java subtyping and invariant subtyping disagree (A <: B but B <@ A) –Substitutability not maintained –Statically emit warning Detect if client subclasses do not use framework classes as prescribed –Safer multithreaded frameworks

25. Java API Annotations Started to annotate methods in Java API Community project at http://community.concutest.org/ http://community.concutest.org/ –Browse and suggest annotations Annotations can be extracted into XML –Share annotations –Add checks without needing source code

26. Testing Invariant Checker Annotated two DrJava versions –3/26/2004 –9/2/2006 Ran test suite, logged invariant violations –2004: 18.83% failed –2006: 11.03% failed 2006 version easier to annotate –Better documentation of invariants details

27. Conclusion Improved JUnit now detects problems in other threads Annotations ease documentation and checking of concurrency invariants Support programs for schedule-based execution

28. Future Work Schedule-Based Execution –Replay given schedule –Generate possible schedules –Dynamic race detection –Probabilities/durations for random yields/sleeps Extend annotations to Floyd-Hoare logic –Preconditions, postconditions –Representation invariants

29. Many Thanks To… My advisor –Corky Cartwright My committee members –Walid Taha –Bill Scherer My friends –JavaPLT, CS and Rice NFS and Texas ATP –For partially providing funding

30. Extra Slides

31. Possible Solutions Programming Language Features –Race Freedom –Deadlock Freedom, Safe Locking –Atomicity, Transactions –Usually require changes in type system –Fundamental change C++ standards: 1998, 2003, 200x Java major changes: 1997 (1.0), 2002 (1.4), 2004 (1.5) back

32. Possible Solutions Lock-Free Algorithms –Work on copy and assume no concurrency is present (or current thread will finish first) –If there was interference, threads that don’t finish first redo their work –Require some system support (e.g. compare- and-swap), then done in libraries –Not all common data structures are practical and efficient to implement as lock-free back

33. Extra: Number of Schedules back Product of s-combinations For thread 1: choose s out of ts time slices For thread 2: choose s out of ts-s time slices … For thread t-1: choose s out of 2s time slices For thread t-1: choose s out of s time slices Writing s-combinations using factorial Cancel out terms in denominator and next numerator Left with (ts)! in numerator and t numerators with s!

34. Improvements to JUnit Uncaught exceptions and failed assertions –Not caught in child threads

35. Sample JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } public void testAssertion() { public void testAssertion() { assertEquals(0, 1); assertEquals(0, 1); }} if (0!=1) throw new AssertionFailedError(); } Both tests fail.

36. Problematic JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Main thread Child thread Main thread Child thread spawns uncaught! end of test success!

37. Problematic JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Main thread Child thread Uncaught exception, test should fail but does not!

38. Improvements to JUnit Uncaught exceptions and failed assertions –Not caught in child threads Thread group with exception handler –JUnit test runs in a separate thread, not main thread –Child threads are created in same thread group –When test ends, check if handler was invoked

39. Thread Group for JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread invokes checks TestGroup’s Uncaught Exception Handler

40. Thread Group for JUnit Tests public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread Test thread Child thread spawns uncaught! end of test failure! invokes group’s handler Main thread spawns and waitsresumes check group’s handler

41. Improvements to JUnit Uncaught exceptions and failed assertions –Not caught in child threads Thread group with exception handler –JUnit test runs in a separate thread, not main thread –Child threads are created in same thread group –When test ends, check if handler was invoked Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end

42. Child Thread Outlives Parent public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread Test thread Child thread spawns uncaught! end of test failure! invokes group’s handler Main thread spawns and waitsresumes check group’s handler

43. Child Thread Outlives Parent public class Test extends TestCase { public void testException() { public void testException() { new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } }).start(); }).start(); }} new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); }}).start(); throw new RuntimeException("booh!"); Test thread Child thread Test thread Child thread spawns uncaught! end of test success! invokes group’s handler Main thread spawns and waitsresumes check group’s handler Too late!

44. Improvements to JUnit Child threads are not required to terminate –A test may pass before an error is reached Detect if any child threads are still alive –Declare failure if test thread has not waited –Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.) Previous schedule is a test failure –Should be prevented by using Thread.join()

45. Testing ConcJUnit Replacement for junit.jar or as plugin JAR for JUnit 4.2 Available as binary and source at http://www.concutest.org/ http://www.concutest.org/ Results from DrJava’s unit tests –Child thread for communication with slave VM still alive in test –Several reader and writer threads still alive in low level test (calls to join() missing)

46. Limitations Improvements only check chosen schedule –A different schedule may still fail –Requires comprehensive testing to be meaningful May still miss uncaught exceptions –Specify absolute parent thread group, not relative Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) details

47. Extra: Limitations May still miss uncaught exceptions –Specify absolute parent thread group, not relative (rare) Koders.com: 913 matches ThreadGroup vs. 49,329 matches for Thread –Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) Koders.com: 32 method definitions for uncaughtException method back

48. Generic Annotations? Write @And as generic annotation? public @interface And { T[] terms(); T[] terms();} public @interface OnlyThreadWithName { String name(); String name();} Generics not allowed in annotations

49. Extra: DrJava Statistics 200473661036905116416196518.83%1071 Unit tests passedfailed not run Invariantsmetfailed % failed KLOC “event thread” 2006881881003441230616379611.0312999 back

50. Predicate Annotation Example @PredicateLink(value=Predicates.class, method="example", method="example", arguments=true) arguments=true) public @interface TestAllowed { boolean allowed; boolean allowed;} Definition Refers to Predicates.example back

51. Predicate Annotation Example public class TestCode { @TestAllowed(allowed=true) @TestAllowed(allowed=true) public void test(String param) { … } public void test(String param) { … }}… TestCode t = new TestCode(); t.test("xxx"); Usage Call back

52. Predicate Annotation Example public class Predicates { public static boolean example( public static boolean example( Object this0, Object this0, String param, String param, boolean allowed) { boolean allowed) { return (allowed)? (param.equals("test")): return (allowed)? (param.equals("test")): (!param.equals("test")); (!param.equals("test"));} back Predicate Method

53. back Predicate Annotation Example

54. Problem: Multiple Annotations Java does not allow the same annotation class multiple times @NotThreadWithName("foo") @NotThreadWithName("bar") // error void testMethod() { … } Conjunctions, disjunctions and negations?

55. Annotation Subclasses? Let annotation extend a supertype? public @interface Invariant { } public @interface OnlyThreadWithName extends Invariant { String name(); } extends Invariant { String name(); } public @interface And extends Invariant { Invariant[] terms(); Invariant[] terms();} Subtyping not allowed for annotations

56. Work-Around Different meta-annotation, @Combine @Combine(Combine.Mode.AND) public @interface SeveralNames { OnlyThreadWithName[] value(); OnlyThreadWithName[] value();}@SeveralNames({@NotThreadWithName("foo"), @NotThreadWithName("bar")}) @NotThreadWithName("bar")}) void testMethod() { … }

57. Combine Annotations May only contain invariant annotations –Predicate annotations –Combine annotations –Arrays of the above Predicate method automatically generated –Calls member predicate methods –Performs AND, OR, XOR, NOT or IMPLIES

58. Invariants As Class Annotation Short-hand for annotating all methods What about methods already introduced in a super class, e.g. Object.notify() ? What about methods introduced in subclasses? @Invariant class A { void foo() { … } void foo() { … } void bar() { … } void bar() { … }} class A { @Invariant @Invariant void foo() { … } void foo() { … } @Invariant @Invariant void bar() { … } void bar() { … }}

59. Invariants As Class Annotation Apply invariants on a class only to methods introduced in the class or subclasses class A { public void foo() { … } public void foo() { … }} @Invariant // only applies to bar() @Invariant // only applies to bar() class B extends A { class B extends A { public void bar() { … } public void bar() { … }}

60. Bytecode Rewriting Except for JUnit, all tools use bytecode rewriting –Class files easier to parse than source –Works for classes without source (Java API) –Can perform changes on-the-fly Adding code to detect properties –“instrumentation” Offline and on-the-fly –Offline:Class file  class file tool –On-the-fly: Custom class loader

61. Local vs. Global Changes can be done by –modifying one method (local) –all code that calls the method (global) Local instrumentation usually better –Fewer changes, less bytecode –Harder to make a partial instrumentation if global Not all instrumentations can be done locally –If method is native, no class file exists

62. Other Contributions Recording schedules –thread ID/type –thread ID/type/object ID/class/method/PC Deadlock detector –thread ID/type/object ID recording required –Creates wait graph for each thread and lock –Cycle in graph implies deadlock

63. Random Sleeps/Yields Randomly insert sleeps or yield before or after critical points Example: If a notify() is delayed, a wait() may time out. Can detect a number of sample problems Have not studied probabilities or durations for sleeps/yields –One inserted delay may negatively impact a second inserted delay Example: If both notify() and wait() are delayed.