1. CAPP: Change-Aware Preemption Prioritization Vilas Jagannath, Qingzhou Luo, Darko Marinov Sep 6 th 2011

2. Motivation Parallel code required for performance – Dominant paradigm is shared-memory, multithreaded code Difficult to develop (correct) multithreaded code – Different behavior under different schedules – Afflicted by bugs like data races, deadlocks, atomicity violations… Difficult to test multithreaded code – Exploration required, time consuming – Current exploration techniques focus on one code version – Code evolves, need efficient regression testing CAPP: Change-Aware Preemption Prioritization 2

3. @Test public void testFilterWriteThreadSafety() throws InterruptedException { … FilterWriteThread fwThread1 = new FilterWriteThread(); FilterWriteThread fwThread2 = new FilterWriteThread(); fwThread1.start(); fwThread2.start(); fwThread1.join(); fwThread2.join(); assertEquals(0, fwThread1.result); assertEquals(0, fwThread2.result); } class FilterWriteThread extends Thread { int result = 0; ProtocolCodecFilter pc; … public void run() { try { pc.filterWrite(nextFilter, session, writeRequest); … } catch (Exception e) { e.printStackTrace(); result = 1; } } } Apache Mina Test for filterWrite() 3

4. Exploration 4 S0 S1 T1 S2 T1 S3 T1 S.. T2 S.. T1 S.. T2 Schedule Preemption S.. T2 Transition (S0, T1)(S0, T2)

5. Multithreaded Testing 5 Code Test Test execution explores many schedules

6. Work Pool (S1, T2) Exploration Order 6 (S0, T1) (S0, T2) S1 T1 (S1, T1) S2 T1 S0 Eg. DFS: Stack BFS: Queue …

7. Current Work Pool Exploration Prioritization 7 S1 T1 S2 T1 S0 Next Work Pool (S0, T1) (S0, T2) (S1, T2)(S1, T1) S.. T2 S.. T2 (S.., T2) (S.., T1) ….. Eg. CHESS Gambit Active Testing ConTest AVIO CTrigger …

8. Regression Multithreaded Testing 8 Code Test Code`

9. Change-Unaware Prioritization 9 Code Test Code`

10. Change-Unaware Prioritization 10 Code Test Code`

11. CAPP: Change-Aware Preemption Prioritization 11 Code Test Code` Use evolution information to prioritize exploration of schedules that perform preemptions at change impacted code. CAPP is a general framework. We instantiate with 14 heuristics.

12. Apache Mina r912148 to r912149 public void filterWrite(NextFilter nextFilter, IoSession session, WriteRequest writeRequest) throws Exception { … flushWithoutFuture(); … } public void flushWithoutFuture() { Queue bufferQueue = getMessageQueue(); for (;;) { Object encodedMessage = bufferQueue.poll(); encodedMessage == null) { break; } if (// Flush only when the buffer has remaining. if (!(encodedMessage instanceof IoBuffer) || ((IoBuffer) encodedMessage).hasRemaining()) { … } } } 12 public void filterWrite(NextFilter nextFilter, IoSession session, WriteRequest writeRequest) throws Exception { … Queue bufferQueue = getMessageQueue(); while (!bufferQueue.isEmpty()) { Object encodedMessage = bufferQueue.poll(); // Flush only when the buffer has remaining. if (!(encodedMessage instanceof IoBuffer) || ((IoBuffer) encodedMessage).hasRemaining()) { … } } … DIRMINA-803 58 schedules to detect failure using CHESS 5 schedules using one of the CAPP heuristics

13. CAPP Parts Impacted Code Points (ICP) Collection (Static) Change-Aware Exploration Prioritization (Dynamic) 13

14. Impacted Code Points (ICP) Collection 14 CodeCode` Inspect structure diff to collect impacted: – Classes, Methods, Lines, Fields Focus on changes to thread communication: – Expand synchronized blocks, methods – Fields accessed by changed lines – Fields accessed in methods called by changed lines

15. Current Work Pool Change-Aware Exploration Prioritization 15 S1 T2 S2 T1 S0 Next Work Pool (S0, T1) (S0, T2) (S1, T2) (S1, T1) Use ICPs to partition

16. Family of Heuristics Defined by ICP Match Mode and Prioritization Mode ICP Match Mode : Is transition impacted by changes? – Class (on Stack), Method (on Stack), Line (on Stack), Field Prioritization Mode : Which work pool to add transitions to? – All, Some 14 heuristics – CA (Class All) – … – MOS (Method on Stack Some)… – … 16

17. ICP Match Mode ICP Match Mode: Is transition impacted by changes? – Class (on Stack), Method (on Stack), Line (on Stack), Field 17 S.. ICPs: {………, class A,………….....} (S.., T1) (S.., T2) T1: Executing class A, method m, line 55 T2: Executing class B, method k, line 32 Class (S.., T1) (S.., T2)

18. Prioritization Mode 18 Current Work Pool 18 S.. Next Work Pool (S.., T1) Prioritization Mode: Which work pool to add transitions to? – All : Only prioritize preemptions among impacted transitions – Some : Prioritize preemptions among impacted and not-impacted (S.., T2)(S.., T3)

19. Prioritization Mode 19 Current Work Pool 19 S.. Next Work Pool Prioritization Mode: Which work pool to add transitions to? – All : Only prioritize preemptions among impacted transitions – Some : Prioritize preemptions among impacted and not-impacted All (S.., T1)(S.., T2)(S.., T3)

20. Prioritization Mode 20 Current Work Pool 20 S.. Next Work Pool Prioritization Mode: Which work pool to add transitions to? – All : Only prioritize preemptions among impacted transitions – Some : Prioritize preemptions among impacted and not-impacted All (S.., T1)(S.., T2)(S.., T3)

21. Prioritization Mode 21 Current Work Pool 21 S.. Next Work Pool Prioritization Mode: Which work pool to add transitions to? – All : Only prioritize preemptions among impacted transitions – Some : Prioritize preemptions among impacted and not-impacted Some (S.., T1)(S.., T2)(S.., T3)

22. Prioritization Mode 22 Current Work Pool 22 S.. Next Work Pool Prioritization Mode: Which work pool to add transitions to? – All : Only prioritize preemptions among impacted transitions – Some : Prioritize preemptions among impacted and not-impacted Some (S.., T1)(S.., T2)(S.., T3) T1

23. Implementations Two implementations Java PathFinder (JPF) – Stateful exploration using checkpoints – Depth first, unbounded – CAPP implemented using a custom SimplePrioritySearch ReEx – Stateless exploration using re-execution – CHESS (iterative context bound 2) – CAPP implemented using a custom SchedulingStrategy 23

24. Evaluation Subjects 15 multithreaded programs: – Ranging from 52-54,000 loc, with 2-607 classes, running 2-9 threads – Evolution from non-buggy version to buggy version 7 deadlock faults, 8 data race and atomicity violation faults 24 SubjectSLOCThreadsICPs Airline13666 Allocation20935 BoundedBuffer11092 BubbleSort8944 Deadlock5233 ReadersWriters15452 ReplWorkers43232 RAXextended16662 Groovy54,872360 Lang48,36933 Mina34,804336 Pool110,0423148 Pool24,4733201 Pool310,802229 Pool410,783347

25. Experiments Independent variables: – 14 heuristics – Stateful vs. Stateless exploration – Default exploration order vs. 50 Random exploration orders 19,890 explorations, around a month of computing time Measured savings in exploration cost compared to change- unaware exploration: – Transitions for JPF implementation – Schedules for ReEx implementation 25

26. Default Non-Randomized Exploration 26 5.3x 2.7x 0.8x

27. CAPP Current multithreaded testing techniques focus on one code version Code evolves, need techniques that leverage evolution info CAPP uses evolution info to prioritize exploration: – Up to 5.3x average reduction in exploration cost Ongoing work: – Regression Schedule Selection [ICST 10, STVR in revision] – Schedule Expression [ESEC/FSE 11] – Multithreaded Tests Generation 27

28. Possible Project Ideas 28 Better impact analysis Combine selection and prioritization (RMC) Prioritize across tests Multiple faults Improve ReEx, revisit stateless randomized search (PRSS) Combine IMUnit with CAPP – How to find/generate/simplify buggy IMUnit schedules? – How to decide whether some IMUnit schedules is valid or not after code evolves (a.k.a., identify bad concurrency bug fixes)?

29. UPCRC 29 Universal Parallel Computing Research Center – Funded by Intel and Microsoft – Goal: improve parallelism for client computation – Two centers: Berkeley and Illinois UPCRC Illinois – About 20 faculty + 40 grad students – Research on entire computation stack: architecture, runtime systems, compilers, programming languages, design patterns, applications… and testing

30. Results Summary Does CAPP help? – Up to 5.3x reduction on average – Only one heuristic (out of 28) increased cost (0.8x) How do heuristics compare? – Field ICP Match Mode heuristics perform best – ALL Prioritization Mode heuristics perform best Stateful vs. Stateless? – Greater benefit for stateless exploration Default vs. Random? – Conclusions from random results similar to default results – Except ICP Match Mode Detailed results in the paper 30

31. Multicore World 31 Performance! Shared Memory Multithreaded

32. Difficult to Develop Multithreaded Code Scheduling non-determinism Data races Deadlocks Atomicity violations … 32 Correct Shared Memory Multithreaded

33. Difficult to Test Multithreaded Code Exploration required Time consuming Current techniques focus on one code version Code evolves Need efficient regression testing 33 Code Test