Guoliang Jin, Linhai Song, Wei Zhang, Shan Lu, and Ben Liblit University of Wisconsin–Madison Automated Atomicity- Violation Fixing

Focus on Single-Variable Atomicity Bugs  Multicore era is coming already here  Programmers still struggle to reason about concurrency  Many ways to find and replay concurrency bugs  Atomizer, CTrigger, CCI, Kivati, Pacer, Falcon, …  But that’s still not a fix!  Automated fixer for single-variable atomicity bugs  Leverage variety of bug-finding tools  Static analyses & optimizations to patch one bug  Heuristics to merge & simplify groups of patches  Run-time validation during testing or post-deployment

Thread 2Thread 1 Collect Atomicity-Violation Triples read x write x p: read x c: read x r: write x pipi cici riri

Static Analyses to Construct One Patch p c

Lock Selection and Optimization  Any potentially-blocking operations in critical region?  No: wait forever when acquiring lock  Yes: time out when acquiring lock  Conservative over-approximation finds ad hoc spin loops  Is recursion possible within critical region?  No: use non-reentrant locks  Yes: use reentrant locks  Can we reach r while in the p–c critical region?  No: retain lock operations before/after r  Yes: remove lock operations before/after r  Try to reuse existing global lock (in practice, never)

Selective Fusion of Multiple Patches  Can improve performance and readability  Depends on costs of lock/unlock, thread count, contention, …  No statically-optimal answer  But some redundant operations are always good to remove  May be mandatory to avoid deadlocks  Merge if p 1, c 1, or r 1 are in any of patch 2’s critical regions  Heuristic, but works well in practice p1p1 p2p2 c2c2 c1c1 p1p1 p2p2 c1c1 c2c2 p1p1 p2p2 c1c1 c2c2

Run-Time Monitoring and Feedback  Never adds new interleavings, but may cause deadlock  Choice of two run-time deadlock detectors 1. High-overhead, immediate notification  Must monitor all synchronization operations at all times  Always knows complete resource-allocation graph 2. Low-overhead, delayed notification  Does nothing until after a lock timeout  Eventually infers resource-allocation graph, then checks for cycles  If bug detector is incomplete, reapply to patched program  May report additional (p, c, r) pairs, requiring further patching  Done fixing when bug detector can no longer find problems

Evaluation: Overall Patch Quality BugNaïveUnmergedMergedManual Apache-- MySQL 1-  MySQL 2-  Mozilla 1-  Mozilla 2- Cherokee- FFT--  PBZIP2---  Patched failure rates: 0% (except PBZIP2 and FFT)  Patched overheads: <0.6% (except PBZIP2)  Readily understandable, with few mutexes after merging

Evaluation: Failure Rate Under Testing BugOriginalNaïveUnmergedMerged Apache85%deadlock83%0% MySQL 141%deadlock0% MySQL 253%deadlock0% Mozilla 141%deadlock0% Mozilla 248%deadlock0% Cherokee81%deadlock0% FFT74%73%87%30% PBZIP294%deadlock66%20%  Patched failure rates: 0% (except PBZIP2 and FFT)  Patched overheads: <0.6% (except PBZIP2)  Readily understandable, with few mutexes after merging

Evaluation: Performance Overhead BugNaïveUnmergedMergedManual Apache-0.45%-0.97%-0.26% MySQL % 0.45% MySQL % 1.02% Mozilla %0.55%0.12% Mozilla % -0.20% Cherokee--1.02%-1.04%0.39% FFT-0.02%-0.07%-0.02%0.19% PBZIP2-89,132%181.82%-0.20%  Patched failure rates: 0% (except PBZIP2 and FFT)  Patched overheads: <0.6% (except PBZIP2)  Readily understandable, with few mutexes after merging

Conclusions  Patient says, “Doctor, doctor, it hurts when I do this.”  Doctor replies, “Then don’t do that!” ☺  Natural to apply this to concurrency  But must be exceedingly careful in the details  Makes strong guarantees, but does not promise the world  Never allows interleavings that were not already possible  But may cause deadlocks by over-constraining execution space  Uses some heuristics, but excellent results in practice  Overheads too low to reliably measure  Produces small, simple, understandable patches  Completely eliminates detected bugs in targeted class

Abstract Fixing software bugs has always been an important and time-consuming process in software development. Fixing concurrency bugs has become especially critical in the multicore era. However, fixing concurrency bugs is challenging, in part due to non-deterministic failures and tricky parallel reasoning. Beyond correctly fixing the original problem in the software, a good patch should also avoid introducing new bugs, degrading performance unnecessarily, or damaging software readability. Existing tools cannot automate the whole fixing process and provide good-quality patches. I will present AFix, a tool that automates the whole process of fixing one common type of concurrency bug: single-variable atomicity violations. AFix starts from the bug reports of existing bug-detection tools. It augments these with static analysis to construct a suitable patch for each bug report. It further tries to combine the patches of multiple bugs for better performance and code readability. Finally, AFix's run-time component provides testing customized for each patch. Experimental evaluation shows that patches automatically generated by AFix correctly eliminate six out of eight real-world bugs and significantly decrease the failure probability in the other two cases. AFix patches never introduce new bugs and have similar performance to manually-designed patches.