How not to do Java Concurrency And how to find if you did it wrong Mark Winterrowd

Introductions Coverity Me You! Bob Copyright Coverity, Inc., 20142

Meet Bob Bob's development philosophy: Speed is everything Clever tweaks are their own reward Trying it out once or twice is plenty testing Your manager has assigned him to help you out! Copyright Coverity, Inc., 20143

Your Mission: Keep your code correct! Bob will write code mirroring mistakes seen in the wild For novices: preparation for handling your own Bob For experts: Examples occurring in real code Modeled off of code seen in Tomcat, Eclipse and Android Copyright Coverity, Inc., 20144

Patterns discussed here are low level Prefer to use higher level structures e.g., java.util.concurrent Hide uses behind higher level calls or objects Sometimes unavoidable Copyright Coverity, Inc., 20145

Multithreaded Lazy Initialization Or, sacrificing correctness for speed

Task: Lazily initialize a singleton In a single-threaded program: private static MyObj inst = null; public static MyObj getInst() { if(inst == null) { inst = new MyObj(); } return inst; } This won't work in a multithreaded program. Copyright Coverity, Inc., 20147

Simple Fix: private static MyObj inst = null; static final Object initLock = new Object(); public static MyObj getInst() { synchronized(initLock) { if(inst == null) { inst = new MyObj(); } return inst; } Bob objects: "This requires you to wait to get the lock every time!" Copyright Coverity, Inc., 20148

Bob's Performance Optimization public static MyObj getInst() { inst = new MyObj(); } return inst; } "Since we only need to set inst the first time, synchronizing to check for null is unnecessary!" Copyright Coverity, Inc., if(inst == null) { synchronized(initLock) { Is Bob right? Is his code correct?

Bob's optimized code in action Copyright Coverity, Inc., Thread 1Thread 2 if(inst == null) synchronized(initLock) inst = new MyObj(); return inst;synchronized(initLock) inst = new MyObj(); return inst; No longer a singleton!

The JVM can start and stop threads at will Holding a lock prevents another thread from executing code guarded by the same lock object Bob says "In that case, I have an idea!" Copyright Coverity, Inc.,

"Also check in the synchronized block!" private static MyObj inst = null; static final Object initLock = new Object(); public static MyObj getInst() { if(inst == null) synchronized(initLock) { Copyright Coverity, Inc., "Now, we can't have inst be initialized multiple times!" } return inst; } if(inst == null) inst = new MyObj(); Is Bob right? Is his code correct?

Let's take a look at this line Copyright Coverity, Inc., inst = new MyObj(); MyObj() { this.field1 = someVal1; this.field2 = someVal2; } A raw MyObj is created before running the constructor (raw MyObj).field1 = someVal1; (raw MyObj).field2 = someVal2; inst = (raw MyObj);

JVM re-orders code at runtime! Must have same result in current thread Cannot move code out of or through a synchronized block Reordering inside or into a synchronized is fine! Copyright Coverity, Inc., public static MyObj getInst() { if(inst == null) synchronized(initLock) { if(inst == null) { inst = new MyObj(); } return inst; } inst = (raw MyObj) inst.field1 = someVal inst.field2 = someVal2

This instruction ordering in action: Copyright Coverity, Inc., Thread 1Thread 2 if(inst == null) synchronized(initLock) inst = (raw MyObj); return inst; Returned an uninitialized MyObj! What is the name of this bad pattern? if(inst == null)

Preventing reordering bugs Hold the lock while checking the field for null public static MyObj getInst() { Copyright Coverity, Inc., synchronized(initLock) { if(inst == null) inst = new MyObj(); } return inst; } if(inst == null) Back to our original solution! "This problem seems unlikely. Is fixing it worth the slowdown?"

What slowdown? How unlikely? Uncontested lock acquisition is fast in modern JVMs In computing, unlikely can happen often In 2004, a developer opened Eclipse bug with this comment: Copyright Coverity, Inc., "I know I'm picky, but [...] getManifest has an example of what is known as "Double-Checked Locking problem [...] BundleLoader.createClassLoader has the same problem"

In the wild: Eclipse bug Copyright Coverity, Inc., Reply in 2006: "Synchronizing the places where we lazily create ClassLoader/BundleLoaderProxy objects will cause measurable slowdown. To fix this properly would require at least three additional syncs [...] I added todo comments in the code where double checks are currently being done." A month later: "We are currently having a very hard to reproduce bug when loading preference pages of our RCP application [...] I traced the problem to getBundleLoader [...] When looking at [two methods], I saw they both contained the TODO for this bug. Isn't it better to have slightly less performance instead of possible threading bugs?"

Bug as reported in Coverity Connect Copyright Coverity, Inc.,

Premature Optimization... Especially dangerous in concurrency Rare bugs may be likely in a user's workflow Slow locking may be unnecessary contention Is the lock protecting too much data? Is the lock protecting unrelated critical sections? Can some expensive operations be safely moved outside of a locked region? Copyright Coverity, Inc.,

Choosing your locks Or rather, what not to choose

Make this code thread safe static Object[] items; void update(Object newItem) { Copyright Coverity, Inc., "Synchronize on items so another thread can't change items." Object[] oldArr = items; items = new Object[items.length + 1] items[items.length - 1] = newItem; for(int i=0; i < oldArr.length; i++){ items[i] = oldArr[i]; } } synchronized(items) {

What does synchronizing on a field do? Does not block accesses from other threads to that field Acquires a lock on the field's contents Copyright Coverity, Inc., Thread 1 synchronized(items) items = new Object[...] synchronized(items) items = new Object[...] Different contents! Object[ ] oldArr = itemsitems[...] = newItem Thread 2

Don't lock on mutable fields Copyright Coverity, Inc., static Object[] items; void update(Object newItem) { synchronized( Object[] newAr = new Object[items.length+1]; newAr[newAr.length - 1] = newItem; for(int i=0; i < items.length; i++){ newArr[i] = items[i] } items = newAr; } ) { private static final Object lock = new Object(); itemslock

Tomcat bug 46990: Locking a mutable field Copyright Coverity, Inc., "Whilst there aren't any explicit bugs caused by this, it may be behind some of the harder to reproduce bugs."

Bob sends you the following code for review Copyright Coverity, Inc., class AppCtx { private static SysCtx sCtx; public synchronized SysCtx getSysCtx() { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } " synchronized ensures this code is thread-safe!"

What does the synchronized modifier do? Copyright Coverity, Inc., class AppCtx { private static SysCtx sCtx; public synchronized SysCtx getSysCtx() { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } } synchronized(this) {

Two AppCtx, each in their own thread Copyright Coverity, Inc., this AppCtx 1 AppCtx 2 this sCtx null unlocked synchronized(this) Static AppCtx Members AppCtx1AppCtx2 if(sCtx == null) synchronized(this) if(sCtx == null) sCtx = new SysCtx() Initialized return sCtx; sCtx = new SysCtx()return sCtx; No longer a singleton!

Guard static members with static locks Copyright Coverity, Inc., class AppCtx { private static SysCtx sCtx; public SysCtx getSysCtx() { synchronized( ) { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } private static final Object lock=new Object(); thislock

In the wild: Android bug Copyright Coverity, Inc., Caused loss of display info, user name, and permissions

Wait and Notify and threads communicating poorly

Thread 1 consumes results from Thread 2 LinkedList jobQueue; // Called in Thread 1 void processItemFromQueue() { QueueItem item = null; synchronized(qLock) { item = jobQueue.remove(); } processItem(item); } // Called in Thread 2 void putItemInQueue(QueueItem item) { synchronized(qLock) { jobQueue.add(item); } Copyright Coverity, Inc., What if jobQueue is empty?

The wait / notify pattern lock.wait( ) : current thread stops running, releases lock, and is placed on lock 's wait set lock.notifyAll( ) : Starts up all threads in the wait set, which attempt to re-acquire lock. Call wait when your thread needs another thread to satisfy some condition Call notifyAll when your thread has satisfied a condition for another thread Copyright Coverity, Inc.,

wait and notify in action Copyright Coverity, Inc., Executing Instructionslock Locked By Unlocked Wait Set synchronized(lock)lock.wait( ) Blocked on lock synchronized(lock)lock.wait( )synchronized(lock)lock.wait( )lock.notifyAll( )release locksynchronized(lock)release lock

You see this in a review of Bob's code: void processItemFromQueue() { QueueItem item = null; // Hold the lock as briefly as possible! if(jobQueue.empty()) synchronized(qLock) { qLock.wait(); } item = jobQueue.remove(); processItem(item); } void addItemToQueue(QueueItem item) { synchronized(qLock) { jobQueue.add(item); qLock.notifyAll(); } Copyright Coverity, Inc., Is Bob's code correct?

Bob's code in action: Copyright Coverity, Inc., Executing Instructions qLock Locked By Unlocked Wait Set if(jobQueue.empty()) Consumer qLock.wait() C Blocked on qLock synchronized(qLock) ConsumerProducer synchronized(qLock) Producer jobQueue.add(item)release qLockqLock.notifyAll() No waiting threads to notify returns true Consumer is waiting while items are in the queue!

Waiting on a stale condition is a bug Another thread can change the state between the check and the wait At best, causes unnecessary delays What if future notifications were blocked on this task? Always check wait condition while holding lock Copyright Coverity, Inc., Bob comes back with a second version...

item = jobQueue.remove(); Code review round #2 void processItemFromQueue() { QueueItem item; Copyright Coverity, Inc., Bob: "Now the wait will only occur when the queue is empty!" if(jobQueue.empty()) { processItem(item); qLock.wait(); synchronized(qLock) { Is Bob right? Is his code correct? " A thread can also wake up without being notified, interrupted, or timing out, a so- called spurious wakeup." -Object.wait() documentation } } }

Bob's code in action: Copyright Coverity, Inc., Executing Instructions qLock Locked By Unlocked Wait Set if(jobQueue.empty()) Consumer qLock.wait() C Blocked on qLock synchronized(qLock) ConsumerProducer A spurious wakeup occurs! jobQueue.remove() jobQueue empty!

Check your wait condition inside a loop! Copyright Coverity, Inc., void processItemFromQueue() { QueueItem item; synchronized(qLock) { qLock.wait(); item = jobQueue.remove(); } processItem(item); } whileif (jobQueue.empty())

In the wild: Eclipse bug Copyright Coverity, Inc., Deadlock when ctrl-x was used to cut text. Diagnosed as a wait occurring and never waking up. Remains unfixed, timeout to wait was set to 30s.

The bug (in 2004 code) In code from 2005 Copyright Coverity, Inc., This bug was "fixed" with a 30 second wait timeout in 2013.

Preventing new concurrency bugs and fixing the ones you have

Don't get overly clever! Do you need a multithreaded program? Do you need lazy initialization? Initialize using static{} Is there a higher level structure to replace wait / notify ? Our example: java.util.concurrent.BlockingQueue Sometimes it's unavoidable, or hard to remove. Copyright Coverity, Inc.,

Standard methods help somewhat Education Code reviews Testing Copyright Coverity, Inc., Nondeterminism limits effectiveness Is there anything else we can do? Don’t fix pre-existing problems

Manual Inspection Responsible for finding many of our sampled fixed defects One tip we found in Bugzilla: Copyright Coverity, Inc., No guarantee anyone will notice right away Most of the fixed bugs found by inspection had been present for over 3 years. Why not have a machine inspect your code instead? is Slow "Search[ing] for the ' synchronized ' keyword goes a long way"

Static Analysis: Automatic code inspection Open source static analysis FindBugs™ finds many issues But Coverity's static analysis finds more real bugs with better explanations and a lower rate of false reports Also, we integrate with FindBugs Copyright Coverity, Inc.,

Analyze your code for free! Coverity's SCAN program provides free defect reports for open source programs. Coverity's CodeSpotter: free-to-use SaaS analysis Also a commercial enterprise solution for proprietary software Copyright Coverity, Inc.,

More about Java Concurrency Java Language Specification "Java Concurrency in Practice" by Goetz et al "Java theory and practice: Are all stateful Web applications broken?" by Goetz "Double checked locking is broken", many signers. edLocking.html edLocking.html Copyright Coverity, Inc.,

Copyright 2014 Coverity, Inc.