About Me I'm a software engineer @Cloudera Committer on HDFS Previously I worked on the Ceph distributed filesystem and the Linux kernel, among other things.

Project Motivation Parallel programming is becoming more and more important. New CPUs are expanding “out, not up”... more cores, same MHz Solid-state drives are removing the I/O bottlenecks to parallelism

Motivation Parallel programming is hard. Race conditions often don't show up in testing. Mutex locks and unlocks can be buried deep inside library code, which makes inspection more difficult.

C/C++ Challenges For performance reasons, many behaviors are left undefined in pthreads. Attempting to unlock a mutex you have not locked. Destroying a mutex that is held by another thread. Using a condition variable with two distinct mutexes from two different threads.

C/C++ Challenges The pthreads library offers no introspection features Can't assert(mutex is locked)

Design Challenges Performance Can benchmark Correctness Can test... but tests may not reveal all problems Not deterministic

Deadlock Deadlock occurs when two or more threads both require resources that the other thread(s) hold. Thread 1 Lock A Lock B Thread 2 Lock B Lock A

Visualizing Deadlock One way to think of deadlock is as a dependency cycle. Thread 1 Thread 2 Thread 3

Preventing Deadlock, Idea #1 Use Static analysis Problem: determining whether an arbitrary program will deadlock is equivalent to the halting problem. It is undecidable.

Preventing Deadlock, Idea #1 Despite their limitations, static analysis tools can still be helpful in some cases. Example: the Linux kernel's “sparse” tool allows programmers to make annotations. __must_hold: the specified lock is held on function entry and exit.

Preventing Deadlock, Idea #2 Use pthread_mutex_trylock cleverly Wait with a timeout. Release some resources if the timeout expires. Problem: requires clever programming Another Problem: timeouts slow down our application. This solution is useful in some cases, but it's not very general.

Preventing Deadlock, Idea #3 Don't prevent deadlock. Just detect it when it happens and restart the system. Some distributed databases take this approach, using a deadlock detector thread Not very general

Preventing Deadlock, Idea #4 Absolute ordering If you ever take mutex B after A, never take mutex A after B.

Preventing Deadlock, Idea #4 Absolute ordering This is easier than using pthread_mutex_trylock, and it doesn't involve timeouts. Formalized as CERT recommendation CON35-C: Avoid deadlock by locking in predefined order.

Problems It's difficult to maintain correct mutex ordering. There are a lot of mutexes in most programs. Even one cycle could cause a deadlock. You need to analyze all mutexes to be safe... even those found in library code

Locksmith Locksmith detects mutex ordering violations at runtime. It also detects many cases of undefined behavior. Previous implementations Ceph Linux kernel

Locksmith Not all possible deadlocks will occur most of the time. Locksmith makes the potential for deadlock visible, even in cases where the potential is rare. Complains, gives stack traces for mutex ordering issues or bad behavior.

Locksmith Locksmith is implemented as a shared library which overrides the pthreads functions. Application Using LD_PRELOAD Locksmith Pthreads

Locksmith Being implemented on top of pthreads gives Locksmith very good portability Programs don't need to be recompiled to make use of Locksmith. User-defined mutexes are possible

Enforcing Mutex Ordering For every pair of mutexes, we should never take B before A, if we once take A before B. Essentially, each “lock” operation that a thread does creates an edge in a graph from each mutex it holds to the mutex it is taking.

Deadlock, revisited A cycle means a possible deadlock. Thread 1 Lock A Lock B Thread 2 Lock B Lock A A cycle means a possible deadlock. A B

Features: logging Can log to many different sinks via LKSMITH_LOG stderr or stdout syslog File Program callbacks

Features: ignore patterns Can ignore mutexes locked inside certain frames LKSMITH_IGNORED_FRAMES LKSMITH_IGNORED_FRAME_PATTERNS Useful for working around known bugs

Error checking In addition to doing its own checks, Locksmith enables error-checking mutexes whenever possible Checks for taking sleeping locks (mutexes) while holding a spin lock. And many other checks...

Usable in Many Environments Initialized on first use of pthreads Can be used with pthreads static initializers PTHREAD_COND_INITIALIZER PTHREAD_MUTEX_INITIALIZER Usable within C++ global constructors

Alternatives jcarder A great tool-- for Java. Not available for C/C++ Userspace version of linux kernel lockdep http://lwn.net/Articles/536363 Requires an init call GPL2 (not LGPL)

Alternatives Use PTHREAD_MUTEX_ERRORCHECK This is always available, since it's part of pthreads. This detects some undefined behavior, but has basically no race detection. Locksmith enables this.

Alternatives: Race Detectors These tools build a graph of “happens- before” relationships based on sychronization operations. They warn when they notice both a read and a write operation to the same memory location from more than one thread without such a “happens-before” relationship Popular versions Google Thread Sanitizer (TSAN) Helgrind DRD

Race Detectors Advantages Can find races that don't involve locks at all-- such as “double checked locking” en.wikipedia.org/wiki/Double- checked_locking‎ TSAN is integrated with clang

Race Detectors Disadvantages DRD and TSAN don't seem to detect locking ordering violations (at least in the versions I looked at) Can use tremendous amounts of memory (although TSAN has a “fast mode” which may help)

Race Detectors Disadvantages DRD, TSAN, and Helgrind are implemented as valgrind tools – but you can't use valgrind in some environments, like JNI

Future Directions Speed up implementation Hash commonly occuring stack traces (to avoid symbol lookup, etc) Support rwlocks and some other constructs Integrate with race detector?

References https://github.com/cmccabe/lksmith http://data-race-test.googlecode.com/files/ThreadSanitizer.pdf http://www.cs.cmu.edu/~nbeckman/papers/race_detection_su rvey.pdf