1. E81 CSE 532S: Advanced Multi-Paradigm Software Development Chris Gill and Venkita Subramonian Department of Computer Science and Engineering Washington University, St. Louis cdgill@cse.wustl.edu A Concurrency and Synchronization Pattern Language

2. Main Themes of this Talk Review of concurrency and synchronization Review of patterns and pattern languages Design scenario case study –A peer-to-peer client-server application –Increasing liveness with concurrency patterns –Maintaining safety with synchronization patterns

3. Concurrency Forms of concurrency –Logical (single processor) parallelism –Physical (multi-processor) parallelism Goal: Increase Liveness –Progress is made: deadlock is avoided Also, avoid long blocking intervals if possible –Progress is also optimized where possible Independent activities logically parallel –Ideally, physically parallel if you can afford additional resources Full utilization of resources: something is always running –E.g., both I/O bound and compute bound threads in a process The most eligible task is always running –E.g., the one with the highest priority among those enabled

4. Concurrency, Continued Benefits –Performance Still make progress if one thread blocks (e.g., for I/O) –Preemption Higher priority threads preempt lower-priority ones Drawbacks –Object state corruption due to race conditions –Resource contention Need isolation of inter-dependent operations –For concurrency, synchronization patterns do this At a cost of reducing concurrency somewhat And at a greater risk of deadlock

5. Synchronization Forms of synchronization –Thread (lightweight) vs. process (heavy) locks Why is process locking more expensive? –Binary (mutex) vs. counted (semaphore) locks –What is being locked Scopes: scoped locking Objects: thread-safe interface Role-based: readers-writer locks “It depends”: strategized locking Goal: Safety –Threads do not corrupt objects or resources –More generally, bad inter-leavings are avoided Atomic: runs to completion without being preempted Granularity at which operations are atomic matters

6. Synchronization, Continued Benefits –Safety Prevent threads from interfering in bad ways –Control over (and reproducibility of) concurrency Restrict the set of possible inter-leavings of threads Drawbacks –Reduces concurrency (extreme form: deadlock) –Adds overhead, increases complexity of code Need to apply locking carefully, with a purpose –Only lock where necessary Based on design forces of the specific application Allow locks to be changed, or even removed

7. Concurrency & Synchronization Concurrency hazard: race conditions –Two or more threads access an object/resource –The interleaving of their statements matters –Some inter-leavings have bad consequences Synchronization hazard: deadlock –One or more threads access an object/resource –Access to the resource is serialized –Chain of accesses leads to mutual blocking Clearly, there is a design tension –Must be resolved for each distinct application –Applying patterns helps if guided by design forces

8. Review: What is a Pattern Language? A narrative that composes patterns –Not just a catalog or listing of the patterns –Reconciles design forces between patterns –Provides an outline for design steps A generator for a complete design –Patterns may leave consequences –Other patterns can resolve them –Generative designs resolve all forces Internal tensions don’t “pull design apart”

9. Towards a Pattern Language Key insights –Each pattern solves a particular set of issues –When applied, it may leave others open Depends on the scale of the context E.g., ACT does not itself address synchronization –Although it may help in combination with say the TSS pattern –It may also raise new issues For example, synchronization raises deadlock issues None of this matters without a context –Patterns are only useful when applied to a design –A design stems from the context (problem+forces) of something we’re trying to build So, let’s imagine we’re building something

10. Motivating Scenario Peer-to-peer client server applications –Each process sends and receives requests –Communicates with many other endsystems at once Distributed data and analysis components –Spread more or less evenly throughout network –Each has a particular function or kind of information –Endsystems can look up locations of specific components Directory, naming, and trading services are available Heterogeneous requests –Long-running independent file retrieval requests “Send me the news data feed from 04/21/03” “Send me the stock market ticker for 04/21/03” –Short-duration interdependent analysis requests “Compute the change (+ or -) for each stock sector” “Then find the largest change in a stock sector” “Then find 5 news events with highest correlation to that change”

11. Motivating Scenario, Continued Given –A high-quality LAN-scale analysis server … –designed to manage tens of components … –and interact with tens of other similar servers … –all connected via one company’s private network Assume –Written in C++, leverages C++11/STL features extensively –Design uses Wrapper Façade, other basic patterns –Reactive implementation (initially single-threaded) We’ll discuss other alternatives in the next part of the course E.g., proactive approaches using interceptor to mark/notify progress Goal –Scale this to a business-to-business VPN environment … –… with hundreds instead of tens of nodes locally –… each with hundreds instead of tens of collaborators

12. Review of Concurrency Patterns Active Object –Methods hand off request to internal worker thread Monitor Object –Threads allowed to access object state one-at-a-time Half-Sync/Half-Async –Process asynchronously arriving work in synchronous thread Leader/Followers –HS/HA optimization: leader delegates until its work arrives Thread Specific Storage –Separate storage for each thread, to avoid contention These complement the synchronization patterns

13. Review of Synchronization Patterns Scoped Locking –Ensures a lock is acquired/released in a scope Strategized Locking –Customize locks for safety, liveness, optimization Thread-Safe Interface –Reduce internal locking overhead –Avoid self-deadlock These complement the concurrency patterns

14. How Do We Get Started? Simply apply patterns in order listed? –Sounds ad hoc, no reason to think this would work A better idea is to let the context guide us –Compare current design forces to a pattern context Let’s start with the most obvious issue –Long-running tasks block other tasks –Long-running tasks need to be made concurrent Leads to selection of our first pattern –Concurrent handlers for long-running requests –I.e., apply the Active Object pattern

15. Active Object Context Pattern’s use in this design –Worker thread in handler –Input thread deposits requests for it –Queue decouples request/execution between threads LRRH::handle_event –An enqueue method for AO that puts a command object into the queue –Allows AOs to be registered with an asynchronously triggered dispatcher, e.g., an input (or reactor) thread Long Running Request Handlers (LRRH) enqueue requests worker thread input thread

16. Active Object Consequences Clear benefits for LRRH –Method invocation & execution are decoupled –Helps prevent deadlock –Reduces blocking Drawbacks –Queuing overhead –Context-switch overhead –Implementation complexity Costs well amortized so far –But only for LRRH –So, we can’t apply as is to short- running requests Next design refinement –Apply Half-Sync/Half-Async Long Running Request Handlers (LRRH) enqueue requests worker thread input thread

17. Half-Sync/Half-Async Context Still want long-running activities –Less overhead when related data are processed synchronously Also want short-lived enqueue and dequeue operations –Requests arrive asynchronously –Keep reactor and handlers responsive How to get both? –Make queueing layer smarter –Chain requests by their dependencies –E.g., HS/HA (via pipes & filters) How this is done is crucial –Blocking input (reactor) thread is bad –So, worker thread chains them before it works on a chain! Mixed Duration Request Handlers (MDRH) enqueue requests worker thread input thread worker chains related requests chain requests

18. HS/HA Consequences, part I Problem: race condition –Reactor thread enqueues while active object thread dequeues BTW, this has been an issue –Since we applied Active Object –But we pushed first on concurrency –Could have taken the other branch in our design process first, instead –Emphasizes iterative/rapid design! Need a mutex –Serialize access to the queue data structure itself But then need to avoid deadlock –What if an exception is thrown? (E.g., a call to new fails) Solution: apply scoped locking enqueue requests worker thread input thread worker chains related requests Mixed Duration Request Handlers (MDRH)

19. Scoped Locking Context Define/identify scopes around critical sections –I.e., the active object enqueue and dequeue methods Provide a local guard object on the stack –as the first element within the scope of a critical section Guard ensures a thread automatically acquires the lock –Guard constructor locks upon entering a critical section Guard ensures thread automatically releases the lock –Guard destructor unlocks upon exiting the local scope Guarded locks will thus always be released –even if exceptions are thrown from within the critical section

20. Scoped Locking Consequences Benefit: increased robustness with exceptions Liability: deadlock potential when used recursively –Self deadlock could occur if the lock is not recursive –E.g., if worker thread generates requests for itself Pattern language solution –Apply another pattern! –I.e., strategized locking

21. Strategized Locking Context & Consequences Parameterizes synchronization mechanisms –That protect critical sections from concurrent access Different handlers may need different kinds of synchronization –Null locks (i.e., if we’d ever add passive handlers) –Recursive and non-recursive forms of mutex locks –Readers/writer locks Decouple handlers and locks –Can change one without changing the other –Enhancements, bug fixes should be straight forward Flexibility vs. complexity, time vs. space

22. HSHA Consequences, part II Pop back up to HS/HA –Now that we’ve addressed queue locking issues A new concurrency issue –Chains that are ready, may wait –While thread works on others Could increase # of MDRH’s –Also increase context switches A better idea –Notice the chaining of related requests? –Looks like “sorting the mail” –I.e., apply Leader/Followers enqueue requests worker thread input thread worker chains related requests Mixed Duration Request Handlers (MDRH)

23. Leader when elected picks an incomplete chain Appends requests from reactor to growing chains When a chain is complete –If its for the leader Leader takes request chain A new leader is elected Old leader becomes follower Old leader works on the chain –If it’s for someone else Leader hands off to others Continues to build chains Leader/Followers Context enqueue requests leader thread input thread hand off chains follower threads Mixed Duration Request Handlers (MDRH)

24. Key benefits –Greater cache affinity: thread locality of reference –Context switch only at some events Liabilities –Some increase in implementation complexity I.e., to elect leaders, identify whose chain is whose We’ll just live with this for the purposes here –Some additional overhead But we already separated reactor from handlers So impact on lowest asynchronous level is reduced We’ll go a couple steps further in reducing this Overall, benefits of applying LF here outweigh costs Leader/Followers Consequences

25. TSS Context & Consequences Must lock thread hand-off –But access w/in a chain is single-threaded and thus ok Can save results locally –Put request result in TSS –Retrieve as input for next request in that chain Benefits –Avoids locking overhead –Increases concurrency Liabilities –Slightly more complex to store and fetch results –Portability of TSS (may need to emulate in code) enqueue requests leader thread input thread hand off chains follower threads Mixed Duration Request Handlers (MDRH)

26. A Few Remaining Issues Shared services across threads (naming, lookup) –Threads need safe access Design trade-off –Between simplicity and potential concurrency improvement Alternative patterns –Monitor Object E.g., for new services –Thread Safe Interface E.g., for legacy services enqueue requests leader thread input thread hand off chains follower threads Mixed Duration Request Handlers (MDRH)

27. Monitor Object Context & Consequences Threads use common services –E.g., Naming, Directory –Need thread-safe service access Single copy of each service –Inefficient to duplicate services –Threads access them sporadically Could just make services active –More complex if even possible Solution: Monitor Object –Synchronize access to the service Consequences –If worried about performance, can yield on condition variables –Assume that’s not an issue here (reasonable for Naming, etc.) follower threads common service

28. Thread-Safe Interface Context and Consequences Since synchronous service performance is fine –Go for simplicity, low-overhead –Avoids self-deadlock from intra- component method calls –Avoid complexity of condition based thread yield –Minimizes locking overhead Separates Concerns –Interface methods deal with synchronization –Implementation methods deal with functionality Only option with some single- threaded legacy services –Can’t modify, so wrap instead follower threads service implementation thread-safe service interface

29. Pattern Language Review We first applied Active Object –Made long-running handlers concurrent Then we applied Half-Sync/Half-Async –For chains of short-running requests Next were scoped & strategized locking –To make thread hand-off safe, efficient Then we applied Leader/Followers –To reduce thread context switches

30. Pattern Language Review, Continued We applied TSS next –Efficient access to separate information for each thread can help reduce contention Finally we applied Monitor Object and/or Thread-Safe Interface –For synchronized access to common (possibly legacy) services

31. Concluding Remarks Design mastery is often not based on novelty –Though occasionally it can help –It is mostly a matter of careful observation of the current context as the design unfolds … and how consequences of each decision shape later ones Study patterns and pattern languages –Benefit from prior experience with patterns Including published experience of other “design masters” –Practice working through design examples like this –Build intuition about how patterns fit together Mastery is a journey not a destination –Learn from every design, yours and others