Lazy Diagnosis of In-Production Concurrency Bugs Baris Kasikci, Weidong Cui, Xinyang Ge, Ben Niu

Why Does In-Production Bug Diagnosis Matter? Potential to fix bugs that impact users Short release cycles make in-house testing challenging Release cycles can be as frequent as a few times a day1 [1] https://code.facebook.com/posts/270314900139291/rapid-release-at-massive-scale

Concurrency Bug Diagnosis W Atomicity Violation Thread 1 Thread 2 Time Thread 1 Thread 2 if (*x) { y = *x; } free(x); x = NULL; Concurrency bug diagnosis requires knowing the order of key events (e.g., memory accesses)

Challenges of Concurrency Bug Diagnosis Diagnosis requires reproducing bugs [PBI, ASPLOS’13] [Gist, SOSP’15] Practitioners report that they can fix reproducible bugs [PLATEAU’14] It may not be possible to reproduce in-production concurrency bugs Inputs for reproducing bugs may not be available Exposing bugs in production may incur high overhead [RaceMob, SOSP’13]

In theory, ΔT can be on the order of a nanosecond Record/Replay Tracing fine-grained interleavings incurs high overhead State-of-the-art record/replay has 28% overhead [DoublePlay, ASPLOS’11] R W Atomicity Violation Time Thread 1 Thread 2 ΔT1 ΔT2 In theory, ΔT can be on the order of a nanosecond

Coarse Interleaving Hypothesis Study with 54 bugs in 13 systems Smallest ΔT is 91 microseconds R W Atomicity Violation Time Thread 1 Thread 2 ΔT1 ΔT2 91 us ~1ns ~ 10^5 A lightweight, coarse-grained time tracking mechanism can help infer ordering

Lazy Diagnosis Snorlax Leverages the coarse interleaving hypothesis Hybrid dynamic/static root cause diagnosis technique Snorlax Lazy Diagnosis Prototype Fully Accurate Concurrency Bug Diagnosis (11 bugs in 7 systems) Low overhead (always below < 2%)

Outline Usage model Design Evaluation

Current Bug Diagnosis Model Root cause diagnosis

Lazy Diagnosis Usage Model Root cause + Control- flow trace & Timing Info Root cause diagnosis Control flow trace speeds up static analysis Coarse-grained timing information helps determine ordering

Outline Usage model Design Evaluation

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Hybrid Points-to Analysis FAILURE (CRASH) I1 store i32* %21, %bufSize store %Queue* %1, %q IF I2 load %Queue*, %fifo Finds instructions with operands pointing to the same location as the failing instruction’s operand

Hybrid Points-To Analysis Uses the control flow traces to limit the scope of static analysis Runs fast, scales to large programs (e.g., httpd, MySQL) Lazy Control flow traces trigger the analysis Interprocedural Bug patterns may span multiple functions Flow-insensitive Discards execution order of instructions for scalability

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Type-Based Ranking load %Queue*, %fifo FAILURE (CRASH) 1 2 store i32* %21, %bufSize store i32* %21, %bufSize Type-based Ranking store %Queue* %1, %q store %Queue* %1, %q Highly ranks instructions operating on types that match the failing instruction's operand type

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Bug Pattern Computation Thread 1 Thread 2 Bug Pattern I FAILURE load %Queue*, %fifo load %Queue*, %fifo load %Queue*, %fifo Bug Pattern Computation Bug Pattern Computation store %Queue* %1, %q store %Queue* %1, %q store i32* %21, %bufSize store i32* %21, %bufSize Thread 1 Thread 2 Bug Pattern II

Bug Pattern Computation Our implementation uses timing packets in Intel Processor Trace Granularity of a few 10s of microseconds We measured the smallest ΔT between key events as 91 microseconds Leverages the coarse interleaving hypothesis to establish instruction orders

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Lazy Diagnosis Hybrid Bug Pattern Statistical Type-based Points-to Analysis Type-based Ranking Bug Pattern Computation Statistical Diagnosis

Statistical identification of failure predicting patterns store %Queue* %1, %q load %Queue*, %fifo Thread 1 Thread 2 FAILURE (CRASH) store %Queue* %1, %q load %Queue*, %fifo Thread 1 Thread 2 SUCCESS store %Queue* %1, %q load %Queue*, %fifo Thread 1 Thread 2 SUCCESS store %Queue* %1, %q load %Queue*, %fifo Thread 1 Thread 2 SUCCESS store %Queue* %1, %q load %Queue*, %fifo Thread 1 Thread 2 SUCCESS store %Queue* %1, %q load %Queue*, %fifo Thread 1 Thread 2 FAILURE (CRASH) Statistical identification of failure predicting patterns

Outline Usage model Design Evaluation

Evaluation of Snorlax Is Snorlax effective? Is Snorlax accurate? Is Snorlax efficient? How does Snorlax compare to its competition?

Experimental Setup Real-world C/C++ programs 11 concurrency bugs Workloads from program’s test cases and test cases by other researchers

Snorlax’s Effectiveness Snorlax correctly identified the root causes of 11 bugs Determined after manual investigation of developer fixes A single failure recurrence is enough for root cause diagnosis In practice, for concurrency bugs, “event orders” = “root cause” Snorlax can effectively diagnose concurrency bugs

All stages of Lazy Diagnosis are necessary for full accuracy Snorlax’s Accuracy Contribution Accuracy All stages of Lazy Diagnosis are necessary for full accuracy

Snorlax has low runtime performance overhead (always below 2%) Snorlax’s Efficiency Percentage Overhead 0.97% Snorlax has low runtime performance overhead (always below 2%)

Snorlax vs. Gist 39% Percentage Overhead 3% 0.9% 1.9% Snorlax scales better than Gist with the increasing number of application threads

Lazy Diagnosis Snorlax Leverages the coarse interleaving hypothesis Hybrid dynamic/static root cause diagnosis technique Snorlax Lazy Diagnosis Prototype Fully Accurate Concurrency Bug Diagnosis (11 bugs in 7 systems) Low overhead (always below < 2%) Scales well with the increasing number of threads Michigan is hiring!