COLT: Effective Testing of Concurrent Collection Usage Alex Aiken Nathan Bronson Stanford University Martin Vechev Eran Yahav IBM Research Mooly Sagiv Ohad Shacham Tel Aviv University

Concurrent Data Structures Writing highly concurrent data structures is complicated Efficient concurrent collections with atomic operations provided by libraries Atomic and efficient composition of several library operations often needed by the client –Efficiency and correctness are client responsibility atomic V put(K,V) {…} atomic V get(K) {…} atomic V putIfAbsent(K,V) {…} atomic V remove(K) {…} atomic boolean remove (K,V) {…} atomic V replace(K,V) {…} atomic boolean replace (K,V, V’) {…} atomic boolean contains(V) {…} atomic boolean containsKey(K) {…} V foo (key K) { Object val = m.get(K) ; if (val != null) { val = m.remove(K); } return val; }

Question How can we write an atomic operation composed from a few libraries operations?

TOMCAT Motivating Example Attribute removeAttribute(String name) { Attribute val = null; synchronized(attr) { found = attr.containsKey(name) ; if (found) { val = attr.get(name); attr.remove(name); } return val; }  Coarse-Grain (TOMCAT 5.*) Fine-Grain (TOMCAT 6.*) Attribute removeAttribute(String name) { Attribute val = null; /* synchronized(attr) { */ found = attr.containsKey(name) ; if (found) { val = attr.get(name); attr.remove(name); } /* } */ return val; } Invariant: removeAttribute(name) returns the value it removes from attr or null

Attribute removeAttribute(String name) { Attribute val = null; found = attr.containsKey(name) ; if (found) { val = attr.get(name); attr.remove(name); } return val; } removeAttribute(“A”) { Attribute val = null; found = attr.containsKey(“A”) ; if (found) { val = attr.get(“A”); attr.remove(“A”); } return val; removeAttribute(“A”) { Attribute val = null; found = attr.containsKey(“A”) ; if (found) { val = attr.get(“A”); attr.remove(“A”); T1 T2 attr Φ { } Breaking the invariant: removeAttribute(name) returns the value it removes from attr or null T1.val o null

 TOMCAT Fix Attribute removeAttribute(String name) { Attribute val = null; synchronized(attr) { found = attr.containsKey(name) ; if (found) { val = attr.get(name); attr.remove(name); } return val; } Coarse-Grain (Before)Fine-Grain (After) Attribute removeAttribute(String name) { Attribute val = attr.get(name) ; if (val != null) { val = attr.remove(name); } return val; }

Very hard to reconstruct in TOMCAT –Specific thread interleaving –Same input key removeAttribute(“A”) { Attribute val = null; found = attr.containsKey(“A”) ; if (found) { val = attr.get(“A”); attr.remove(“A”); } return val; removeAttribute(“A”) { Attribute val = null; found = attr.containsKey(“A”) ; if (found) { val = attr.get(“A”); attr.remove(“A”); T1 T2 attr Φ { } T1.val o null

TOMCAT Motivating Example Attribute removeAttribute(String name) { Attribute val = null; found = attr.containsKey(name) ; if (found) { val = attr.get(name); attr.remove(name); } return val; }

TOMCAT Motivating Example removeAttribute is a new operation of attr composed by attr’s operations Should be atomic in any client environment Resilient for future client changes Backed up by our experimental results Attribute removeAttribute(String name) { Attribute val = null; found = attr.containsKey(name) ; if (found) { val = attr.get(name); attr.remove(name); } return val; } No operation on attr attr atomic V put(K,V) {…} atomic V get(K) {…} atomic V putIfAbsent(K,V) {…} atomic V remove(K) {…} atomic boolean remove (K,V) {…} atomic V replace(K,V) {…} atomic boolean replace (K,V, V’) {…} atomic boolean contains(V) {…} atomic boolean containsKey(K) {…}.

Challenge Checking atomicity of composed operations in any client environment

Memoization Example V compute(K) { synchronized(m) { val = m.get(K); if (val == null) { val = calculateVal(K); m.put (K, val); } return val; } Coarse-Grain

Fine-Grained Concurrent Clients Concurrent collections provide basic operations for fine- grained concurrency ( read-check-modify, CAS like) V putIfAbsent(K, V) –Store and return null or return the current value if K already exists boolean replace(K, V, V’) –Replace by if already exists –Return true on success and false on failure boolean remove(K, V) –Remove if its exists –Return true on success and false on failure

Memoization Example V compute(K) { val = m.get(K); if (val == null) { val = calculateVal(K); m.putIfAbsent(K, val); } return m.get(K); } V compute(K) { synchronized(m) { val = m.get(K); if (val == null) { val = calculateVal(K); m.put (K, val); } return val; }  m.remove(K ) Coarse-Grain (Before)Fine-Grain (After)

Memoization Fix What do we mean by atomic? // atomic V compute(K) { val = m.get(K); if (val == null) { val = calculateVal(K); tmpVal = m.putIfAbsent(K, val); if (tmpVal != null) val = tmpVal; } return val; } atomic implementation of compute

Linearizability [Herlihy and Wing, TOPLAS'90] V compute(K) { val = m.get(K); val != null if (val == null) { val = calculateVal(K); tmpVal = m.putIfAbsent(K, val); if (tmpVal != null) val = tmpVal; } return val; } val=m.get(7)compute(7) {…tmpVal=m.putIfAbsent(7,8) if (tmpVal != null) return val; { } val=m.get(7)compute(7) { if (tmpVal != null) return val;

Thread Modular Linearizability ConcurrentHashMap MUT atomic V put(K,V) {…} atomic V get(K) {…} atomic V putIfAbsent(K,V) {…} atomic V remove(K) {…} atomic boolean remove (K,V) {…} atomic V replace(K,V) {…} atomic boolean replace (K,V, V’) {…} atomic boolean contains(V) {…} atomic boolean containsKey(K) {…}. V compute(K) { val = m.get(K); if (val == null) { val = calculateVal(K); m.putIfAbsent(K, val); } return m.get(K); }

Thread Modular Linearizability atomic V put(K,V) {…} atomic V get(K) {…} atomic V putIfAbsent(K,V) {…} atomic V remove(K) {…} atomic boolean remove (K,V) {…} atomic V replace(K,V) {…} atomic boolean replace (K,V, V’) {…} atomic boolean contains(V) {…} atomic boolean containsKey(K) {…}. V compute(K) { val = m.get(K); if (val == null) { val = calculateVal(K); m.putIfAbsent(K, val); } return m.get(K); }

val=m.get(7)compute(7) {…tmpVal=m.putIfAbsent(7,8)return m.get(7) val=m.get(8)compute(8) {…tmpVal=m.putIfAbsent(8,8)return m.get(8)m.put(9,10) val=m.get(12)compute(12) {…tmpVal=m.putIfAbsent(12,8)return m.get(12)m.put(19,12) val=m.get(5)compute(5) {…tmpVal=m.putIfAbsent(5,13)return m.get(5) m.put(5,12) val=m.get(20)compute(20) {…return m.get(20)m.put(20,10) val=m.get(2)compute(2) {…tmpVal=m.putIfAbsent(2,8)return m.get(2) m.put(30,12) m.remove(20) val=m.get(14)compute(14) {…tmpVal=m.putIfAbsent(14,8)return m.get(14) m.remove(14) val=m.get(20)compute(20) {…return m.get(20) m.remove(14) m.put(9,10)... Thread Modular Linearizability

val=m.get(7)compute(7) {…tmpVal=m.putIfAbsent(7,8)return m.get(7) val=m.get(8)compute(8) {…tmpVal=m.putIfAbsent(8,8)return m.get(8)m.put(9,10) val=m.get(12)compute(12) {…tmpVal=m.putIfAbsent(12,8)return m.get(12)m.put(19,12) val=m.get(5)compute(5) {…tmpVal=m.putIfAbsent(5,13)return m.get(5) m.put(5,12) val=m.get(20)compute(20) {…return m.get(20)m.put(20,10) val=m.get(2)compute(2) {…tmpVal=m.putIfAbsent(2,8)return m.get(2) m.put(30,12) m.remove(20) val=m.get(14)compute(14) {…tmpVal=m.putIfAbsent(14,8)return m.get(14) m.remove(14) val=m.get(20)compute(20) {…return m.get(20) m.remove(14) m.put(9,10)... Hard to test even when modular  Large number of interleavings Bug exhibited very infrequently

val=m.get(7)compute(7) {…tmpVal=m.putIfAbsent(7,8)return m.get(7) val=m.get(8)compute(8) {…tmpVal=m.putIfAbsent(8,8)return m.get(8)m.put(9,10) val=m.get(12)compute(12) {…tmpVal=m.putIfAbsent(12,8)return m.get(12)m.put(19,12) val=m.get(5)compute(5) {…tmpVal=m.putIfAbsent(5,13)return m.get(5) m.put(5,12) val=m.get(20)compute(20) {…return m.get(20)m.put(20,10) val=m.get(2)compute(2) {…tmpVal=m.putIfAbsent(2,8)return m.get(2) m.put(30,12) m.remove(20) val=m.get(14)compute(14) { … tmpVal=m.putIfAbsent(14,8)return m.get(14) m.remove(14) val=m.get(20)compute(20) {…return m.get(20) m.remove(14) m.put(9,10)... Hard to test even when modular

Leveraging Commutativity Operations on different keys are guaranteed to commute Cannot lead to linearizability violation No need to try such interleavings Use commutatively for partial order reduction at the library level val=m.get(8)compute(8) {…tmpVal=m.putIfAbsent(8,8)return m.get(8)m.put(9,10) m.put(9,10) val=m.get(8)… tmpVal=m.putIfAbsent(8,8)return m.get(8) Compute (8) { m.put(9,10) val=m.get(8) … tmpVal=m.putIfAbsent(8,8)return m.get(8) compute(8) { val=m.get(8)compute(8) {…tmpVal=m.putIfAbsent(8,8)return m.get(8)m.put(9,10) val=m.get(8)compute(8) {…tmpVal=m.putIfAbsent(8,8) return m.get(8) m.put(9,10)

Leveraging Commutativity To expose bugs, try operations that do not commute Execute thread in adversarial environment –Adversarial environment picks non-commutative operations –Adversarial scheduler uses non-commutative No bugs are lost due to the reduction MUT OperationConditionAdversarial Action get(k)get(k) == nullput(k,*) get(k)get(k) != nullremove(k), put(k,v) | v!=get(k) putIfAbsent(k,v)get(k) == nullput(k,*) putIfAbsent(k,v)get(k) != nullremove(k) replace(k,oldV,v)get(k) == oldVremove(k) put(k,v’)| v’!= oldV,v replace(k,oldV,v)get(k) != oldVput(k,oldV)

Implementing Dynamic Modular Analyzer Compare concurrent execution to a specific sequential execution Concurrent execution emulated using an adversary Check that every concurrent operation returns the same result as its sequential counterpart Potential linearization point operation MUT Reference Method compare results... Adversary Potential linearization point compare results operation

Running Example V compute(K) { val = m.get(K); val != null if (val == null) { val = calculateVal(K); m.putIfAbsent(K, val); } return m.get(K); }

MUT Adversary Reference Method Map (m) op op Map (refM) Φ compute(7) { Φ val = m.get(7) if (val == null) val = calculateVal(7) {(7,12)} m.put(7,12) != refM.put(7,12) {(7,12)} m.putIfAbsent(7, 14) compute(7) { val = refM.get(7) if (val == null) return refM.get(7) Φ m.remove(7) != refM.remove(7) Φ return m.get(7) MUT OperationConditionAdversarial Action get(k)get(k) == nullput(k,*) get(k)get(k) != nullremove(k), put(k,v) | v!=get(k) putIfAbsent(k,v)get(k) == nullput(k,*) putIfAbsent(k,v)get(k) != nullremove(k) replace(k,oldV,v)get(k) == oldVremove(k), put(k,v’)| v’!= oldV,v replace(k,oldV,v)get(k) != oldVput(k,oldV) V compute(K) { val = m.get(K); val != null if (val == null) { val = calculateVal(K); m.putIfAbsent(K, val); } return m.get(K); } != 12 null

Benchmark Tested 89 MUTs from 48 applications –Apache’s Tomcat, Derby, Cassandra, My Faces – Trinidad, etc…

Results Tested 89 MUTs from 48 applications –Apache’s Tomcat, Derby, Cassandra, My Faces – Trinidad, etc… Discovered 56 linearizability violations 6 reports were proved as linearizable and the adversary refined accordingly Other MUTs were manually proved linearizable Filed bug reports; many already fixed Random adversary failed to find even a single violation –in 10 hours

Reasons For Success Deterministic MUTs For most MUTS –A bug exists for a key iff a bug exists for every key –Control influences by collection’s operation result Does not value dependant V compute(K) { val = m.get(K); if (val == null) { val = calculateVal(K); tmpVal = m.putIfAbsent(K, val); if (tmpVal != null) val = tmpVal; } return val; }

Future Work Verify thread modular linearizability –Small MUT –Use ADT semantics to remove implementation –Small model verification

Summary Fine-grained concurrency is hard Employing atomic library operations is error prune Leverage commutativy Sweet spot –Identify important bugs –Hard to find –Simple and efficient technique