1. StealthTest: Low Overhead Online Software Testing Using Transactional Memory Jayaram Bobba, Weiwei Xiong*, Luke Yen †, Mark D. Hill, and David A. Wood Multifacet Project (www.cs.wisc.edu/multifacet)www.cs.wisc.edu/multifacet Dept. of Computer Sciences University of Wisconsin-Madison * Dept. of Computer Sciences University of Illinois at Urbana- Champaign † Currently at Advanced Micro Devices

2. Executive Summary (1/2) The Problem Software testing hard – Multithreading makes harder Online software testing can help – Run tests on deployed software E.g., Delta Execution for patch testing [ Tucek et al., ASPLOS 2009 ] – Non-intrusive mechanisms fork (existing) Functionally Hidden Low Overhead Good Scaling © 2009 Multifacet Project University of Wisconsin-Madison 2

3. Executive Summary (2/2) StealthTest Leverage Transactional Memory for online testing Non-Intrusive? – transaction { test(); abort} – Fast TM mechanisms © 2009 Multifacet Project University of Wisconsin-Madison Demonstrate two uses Delta Execution In vivo Testing Functionally Hidden Low Overhead Good Scaling 3

4. Outline Online Software Testing – E.g., Patch Validation StealthTest: TM for online testing Delta Execution using StealthTest In vivo Testing using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison 4

5. Online Patch Validation Bug fixes can introduce more bugs – Patches must be validated Online Validation [ Nagaraja et al., OSDI 2004 ] – Increased resource usage – Lockstep execution © 2009 Multifacet Project University of Wisconsin-Madison Testing Production Input Output Diff 5

6. Delta Execution [Tucek et al., ASPLOS 2009] Online Patch Validation Most patches are small Patched and Un-patched executions similar Delta Execution – Run together except when they differ © 2009 Multifacet Project University of Wisconsin-Madison 6

7. Delta Execution using fork © 2009 Multifacet Project University of Wisconsin-Madison Production Testing fork Install  data Unpatched execution Patched execution Compute  data Isolate  data Merged execution Time 7

8. Multi-threading and fork © 2009 Multifacet Project University of Wisconsin-Madison Production Testing fork Install  data Unpatched execution Patched execution Compute  data Isolate  data Merged execution Time 8 Stop all threads to get a consistent memory snapshot ‘Park’ all other threads

9. fork Poor Performance ~9.8ms for split/~106ms for merge [Tucek et al, ASPLOS 2009] Poor Scalability Web-server response rate reduced by 43% Want an alternate mechanism © 2009 Multifacet Project University of Wisconsin-Madison 9

10. Outline Online Software Testing – E.g., Patch Validation StealthTest: TM for online testing Delta Execution using StealthTest In vivo Testing using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison 10

11. Delta Execution using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison Isolate patched execution Isolate patched execution Introspect patched execution Introspect patched execution Monitor delta data access Monitor delta data access Delta Execution Transactional Memory transaction{…} Version Management Tracks new/old values Version Management Tracks new/old values Conflict Detection Monitor accesses Conflict Detection Monitor accesses StealthTest 11 fork Execute on child process Page diffing mprotect

12. StealthTest Interface © 2009 Multifacet Project University of Wisconsin-Madison Isolate patched execution Isolate patched execution Introspect patched execution Introspect patched execution Monitor delta data access Monitor delta data access Delta Execution transaction{…} Version Management Tracks new/old values Version Management Tracks new/old values Conflict Detection Monitor accesses Conflict Detection Monitor accesses StealthTest ST_begin_transaction ST_abort_transaction ST_begin_transaction ST_abort_transaction ST_get_old ST_get_new ST_get_old ST_get_new ST_protect_set ST_protect_clear ST_protect_set ST_protect_clear 12 Transactional Memory

13. Requirements from TM Strong Atomicity [Martin et al., CAL 2006] Transactions isolated from non-transactions => Test transactions isolated from application code Flexible Conflict Resolution Can abort transactions if necessary => Abort tests if they block application Communication from within transactions => Expose result of a test © 2009 Multifacet Project University of Wisconsin-Madison 13

14. Outline Online Software Testing – E.g., Patch Validation StealthTest: TM for online testing Delta Execution using StealthTest In vivo Testing using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison 14

15. Delta Execution using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison Production Testing fork Install  data Unpatched execution Patched execution Compute and Isolate  data Merged execution fork StealthTest Production Install  data Unpatched execution Patched execution Compute and Isolate  data Merged execution ST_abort_tran… ST_begin_tran… ST_protect_set transaction 15 ST_get_newST_get_old Introspect and rollback

16. Multi-threaded Delta Execution © 2009 Multifacet Project University of Wisconsin-Madison Production Testing fork Install  data Unpatched execution Patched execution Compute and Isolate  data Merged execution fork StealthTest Original Install  data Unpatched execution Patched execution Compute and Isolate  data Merged execution ST_abort_tran… ST_begin_tran… ST_protect_set transaction 16 ST_get_newST_get_old Introspect and rollback

17. Evaluation (1) Effective? (2) Non-intrusive? Workloads – Collection of multi-threaded server apps – Same as Tucek et al., ASPLOS 2009 Pin-based TM Emulation 2-way SMP with 2.4GHz Pentium 4 CPUs and 2.5GB RAM © 2009 Multifacet Project University of Wisconsin-Madison 17

18. (1) Effective? © 2009 Multifacet Project University of Wisconsin-Madison Memory allocation Works 18 sockets

19. (2) Non-intrusive? © 2009 Multifacet Project University of Wisconsin-Madison 19

20. Outline Online Software Testing – E.g., Patch Validation StealthTest: TM for online testing Delta Execution using StealthTest In vivo Testing using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison 20

21. In vivo Testing [Murphy et al. TR 2007, Chu et al. ICST 2008] Run unit tests on deployed software + More testing + More realistic Catch bugs early © 2009 Multifacet Project University of Wisconsin-Madison 21

22. In vivo Testing using StealthTest © 2009 Multifacet Project University of Wisconsin-Madison ST_begin_transaction(); try { test(); ST_begin_escape(); fprintf(log, “…”, success); ST_end_escape(); } catch/except() { ST_begin_escape(); fprintf(log, “…”, fail); ST_end_escape(); } ST_abort_transaction(NO_RETRY); 22

23. Evaluation Workloads – Bugbench Server Workloads – STAMP Transactional Memory benchmarks Implementation – Intel STM Language-Based TM – TL2 STM Library-Based TM Quad-core workstation with RHEL5 © 2009 Multifacet Project University of Wisconsin-Madison 23

24. (1) Effective? © 2009 Multifacet Project University of Wisconsin-Madison ProgramDescriptionSize (LOC) Bug TypeError Detected? NCOMfile compress1.9KStack SmashYes POLYfile “unixier”0.7KStack SmashYes GZIPfile compress8.2KBuffer OverflowYes MANdocumentation4.7KBuffer OverflowYes BCcalculator17.0KBuffer OverflowYes HTPD1web server224KAtomicityYes SQUDproxy cache93.5KBuffer OverflowPossible CVSversion control114.5KDouble FreePossible MSQL2DBMS514KAtomicityPossible MSQL3DBMS1028KAtomicityPossible Unsupported Library Calls Works Built on Intel STM. Run tests on Bugbench applications 24

25. (2) Non-intrusive? © 2009 Multifacet Project University of Wisconsin-Madison Built on TL2 STM. Run tests on STAMP applications (1000 tests per min) 25

26. Conclusions and Future Work StealthTest can provide non-intrusive online testing Synergistic benefit for TM and testing Future More testing frameworks – Band-aid patch testing [Sidiroglou et al., 2007] Multi-threaded tests © 2009 Multifacet Project University of Wisconsin-Madison 26

27. Executive Summary Software testing hard Online software testing can help – Existing mechanisms inadequate StealthTest leverages TM for non-intrusive online testing Demonstrate two uses – Delta Execution – In vivo Testing © 2009 Multifacet Project University of Wisconsin-Madison Functionally Hidden Low Overhead Good Scaling 27

28. © 2009 Multifacet Project University of Wisconsin-Madison 28

29. © 2009 Multifacet Project University of Wisconsin-Madison 29

30. Atomicity Violation Bugs? © 2009 Multifacet Project University of Wisconsin-Madison 30

31. Degree-2 Transactions Isolate only writes. Implementation – Reads in escape action – Early Release – Add new type of transaction to TM © 2009 Multifacet Project University of Wisconsin-Madison 31

32. StealthTest Wish List Hardware Support System Calls within Transactions Interaction between Locks and Transactions © 2009 Multifacet Project University of Wisconsin-Madison 32

33. © 2009 Multifacet Project University of Wisconsin-Madison 33

34. Related Work Binary Translation (SPROCKETS) Code Emulation (STEM) TLS (Oplinger&LAM, PathExpander) © 2009 Multifacet Project University of Wisconsin-Madison 34

35. In vivo Testing Motivation Ordering Bug in MySQL In vivo Test: Data Consistency Checks © 2009 Multifacet Project University of Wisconsin-Madison 35 Buggy Code (In mysys/thr_lock.c): void thr_lock_delete(THR_LOCK *lock) { … pthread_mutex_destroy(&lock->mutex); … list_delete(thr_lock_thread_list, &lock->list); … }