Theory: Asleep at the Switch to Many-Core Phillip B. Gibbons Intel Research Pittsburgh Workshop on Theory and Many-Core May 29, 2009 Slides are © Phillip B. Gibbons

Theory: Asleep at the Switch to Many-Core2 Abstract Two decades after the peak of Theory’s interest in parallel computing, the age of many-core is finally underway. Moore’s Law is driving a steady doubling of cores per chip, forcing everyone to jump aboard the parallelism train (or be left behind). To overcome the challenges in realizing the potential of many-core, research is needed in all aspects of parallel computing. Yet, judging by the lack of parallel computing papers in FOCS, SODA, and STOC, Theory is asleep at the switch. This talk provides a wake- up call for Theory to regain a leadership role in parallel computing, before it’s too late. I will advocate five areas in which Theory can (and should) have an important impact, by playing to its strengths. Illustrative examples of initial progress in several of these areas will be drawn from our work on the Hierarchy-savvy Parallel Algorithm Design (Hi-Spade) project.

Theory: Asleep at the Switch to Many-Core3 Two Decades after the peak of Theory’s interest in parallel computing… The Age of Many-Core is finally underway Fueled by Moore’s Law: 2X cores per chip every 18 months All aboard the parallelism train! (Almost) The only way to faster apps

Theory: Asleep at the Switch to Many-Core4 All Aboard the Parallelism Train? Switch to Many-Core…Many Challenges Interest waned long ago Yet problems were NOT solved Research needed in all aspects of Many-Core Computer Architecture Programming Languages & Compilers Operating & Runtime Systems Theory YES! Who has answered the call?

Theory: Asleep at the Switch to Many-Core5 Theory: Asleep at the Switch Theory needs to wake-up & regain a leadership role in parallel computing “Engineer driving derailed Staten Island train may have fallen asleep at the switch.” (12/26/08)

Theory: Asleep at the Switch to Many-Core6 Theory’s Strengths Conceptual Models –Abstract models of computation New Algorithmic Paradigms –New algorithms, new protocols Provable Correctness –Safety, liveness, security, privacy,… Provable Performance Guarantees –Approximation, probabilistic, new metrics Inherent Power/Limitations –Of primitives, features,… …among others

Theory: Asleep at the Switch to Many-Core7 Parallel Thinking Memory Hierarchy Asymmetry/Heterogeniety Concurrency Primitives Power Montparnasse 1895 Five Areas in Which Theory Can (Should) Have an Important Impact

Theory: Asleep at the Switch to Many-Core8 Impact Area: Parallel Thinking Key: Good Model of Parallel Computation Express Parallelism Good parallel programmer’s model Good for teaching, teaching “how to think” Can be engineered to good performance

Theory: Asleep at the Switch to Many-Core9 Impact Area: Memory Hierarchy Deep cache/storage hierarchy Need conceptual model Need smart thread schedulers

Theory: Asleep at the Switch to Many-Core10 Impact Area: Asymmetry/Heterogeniety Fat/Thin cores SIMD extensions Multiple coherence domains Mixed-mode parallelism Virtual Machines...

Theory: Asleep at the Switch to Many-Core11 Impact Area: Concurrency Primitives Parallel prefix Hash map [Herlihy08] Map reduce [Karloff09] Transactional memory Memory block transactions [Blelloch08] Graphics primitives [Ha08] Make the case Many-Core should (not) support Improve the algorithm Recommend new primitives (prescriptive)

Theory: Asleep at the Switch to Many-Core12 Impact Area: Power Many-cores provide features for reducing power Voltage scaling [Albers07] Dynamically run on fewer cores, fewer banks Fertile area for Theory help

Theory: Asleep at the Switch to Many-Core13 Deep Dive: Memory Hierarchy Deep cache/storage hierarchy Need conceptual model Need smart thread schedulers

Theory: Asleep at the Switch to Many-Core14 Good Performance Requires Effective Use of the Memory Hierarchy CPU L1 L2 Cache Main Memory Magnetic Disks Performance: Running/response time Throughput Power Two new trends: Pervasive Multicore & Pervasive Flash bring new challenges and opportunities

Theory: Asleep at the Switch to Many-Core15 L2 Cache New Trend 1: Pervasive Multicore Shared L2 Cache Main Memory Magnetic Disks CPU L1 CPU L1 CPU L1 Makes Effective Use of Hierarchy Much Harder Challenges Cores compete for hierarchy Hard to reason about parallel performance Hundred cores coming soon Cache hierarchy design in flux Hierarchies differ across platforms Opportunity Rethink apps & systems to take advantage of more CPUs on chip

Theory: Asleep at the Switch to Many-Core16 Shared L2 Cache New Trend 2: Pervasive Flash Main Memory Magnetic Disks CPU L1 CPU L1 CPU L1 New Type of Storage in the Hierarchy Flash Devices Challenges Performance quirks of Flash Technology in flux, e.g., Flash Translation Layer (FTL) Opportunity Rethink apps & systems to take advantage

Theory: Asleep at the Switch to Many-Core17 How Hierarchy is Treated Today Ignorant (Pain)-Fully Aware Hand-tuned to platform [Effort high, Not portable, Limited sharing scenarios] API view: Memory + I/O; Parallelism often ignored [Performance iffy] Algorithm Designers & Application/System Developers often tend towards one of two extremes Or they focus on one or a few aspects, but without a comprehensive view of the whole

Theory: Asleep at the Switch to Many-Core18 Hide what can be hid Expose what must be exposed for good performance Robust: many platforms, many resource sharing scenarios Sweet-spot between ignorant and (pain)fully aware “Hierarchy-Savvy” Hierarchy-Savvy Parallel Algorithm Design (Hi-Spade) project …seeks to enable: A hierarchy-savvy approach to algorithm design & systems for emerging parallel hierarchies

Theory: Asleep at the Switch to Many-Core19 Hi-Spade Research Scope A hierarchy-savvy approach to algorithm design & systems for emerging parallel hierarchies Research agenda includes Theory:conceptual models, algorithms, analytical guarantees Systems:runtime support, performance tools, architectural features Applications:databases, operating systems, application kernels

Theory: Asleep at the Switch to Many-Core20 Cache Hierarchies: Sequential External Memory (EM) Algorithms Hi-Spade: Hierarchy-savvy Parallel Algorithm Design [See Vitter’s ACM Surveys article] + Simple model + Minimize I/Os – Only 2 levels Main Memory (size M) External Memory Block size B External Memory Model

Theory: Asleep at the Switch to Many-Core21 Alternative: Cache-Oblivious Algorithms Hi-Spade: Hierarchy-savvy Parallel Algorithm Design Main Memory (size M) External Memory Block size B Cache-Oblivious Model Twist on EM Model: M & B unknown to Algorithm + simple model Key Goal: Good performance for any M & B + Key Goal Guaranteed good cache performance at all levels of hierarchy – Single CPU only [Frigo99] Cache Hierarchies: Sequential

Theory: Asleep at the Switch to Many-Core22 Cache Hierarchies: Parallel Hi-Spade: Hierarchy-savvy Parallel Algorithm Design Explicit Multi-level Hierarchy: Multi-BSP Model [Valiant08] Goal: Approach simplicity of cache-oblivious model Hierarchy-Savvy sweet spot

Theory: Asleep at the Switch to Many-Core23 Challenge: –Theory of cache-oblivious algorithms falls apart once introduce parallelism: Hi-Spade: Hierarchy-savvy Parallel Algorithm Design Good performance for any M & B on 2 levels DOES NOT imply good performance at all levels of hierarchy Key reason: Caches not fully shared L2 CacheShared L2 Cache CPU2 L1 CPU1 L1 CPU3 L1 What’s good for CPU1 is often bad for CPU2 & CPU3 e.g., all want to write B at ≈ the same time B – Parallel cache-obliviousness too strict a goal

Theory: Asleep at the Switch to Many-Core24 Hi-Spade: Hierarchy-savvy Parallel Algorithm Design Key new dimension: Scheduling of parallel threads Has LARGE impact on cache performance L2 CacheShared L2 Cache CPU2 L1 CPU1 L1 CPU3 L1 Can mitigate (but not solve) if can schedule the writes to be far apart in time Recall our problem scenario: all CPUs want to write B at ≈ the same time B

Theory: Asleep at the Switch to Many-Core25 Existing Parallel Cache Models main memory block transfer (size = B) private cache (size = C) CPU block transfer (size = B) private cache (size = C) CPU block transfer (size = B) private cache (size = C) CPU12p ParallelPrivate-Cache Model: main memory shared cache (size = C) CPU block transfer (size = B) CPU 12p ParallelShared-Cache Model: Slide from Rezaul Chowdhury

Theory: Asleep at the Switch to Many-Core26 Competing Demands of Private and Shared Caches private cache CPU private cache CPU private cache CPU 1 2p main memory shared cache  Shared cache: cores work on the same set of cache blocks  Private cache: cores work on disjoint sets of cache blocks  Experimental results have shown that on CMP architectures  work-stealing, i.e., the state-of-art scheduler for private-cache model, can suffer from excessive shared-cache misses  parallel depth first, i.e., the best scheduler for shared-cache model, can incur excessive private-cache misses Slide from Rezaul Chowdhury

Theory: Asleep at the Switch to Many-Core27 Private vs. Shared Caches Parallel all-shared hierarchy: + Provably good cache performance for cache-oblivious algs 3-level multi-core model: insights on private vs. shared + Designed new scheduler with provably good cache performance for class of divide-and-conquer algorithms [Blelloch08] Hi-Spade: Hierarchy-savvy Parallel Algorithm Design L2 CacheShared L2 Cache CPU2 L1 CPU1 L1 CPU3 L1 – Results require exposing working set size for each recursive subproblem

Theory: Asleep at the Switch to Many-Core28 Parallel Tree of Caches Hi-Spade: Hierarchy-savvy Parallel Algorithm Design … … … … … … … … … … … … … … … … … … … … … … … Approach: [Blelloch09] Design low-depth cache-oblivious algorithm Thrm: for each level i, only O(M P D/ B ) misses more than the sequential schedule i i Low depth D Good miss bound

Theory: Asleep at the Switch to Many-Core29 Five Areas in Which Theory Can (Should) Have an Important Impact Parallel Thinking Memory Hierarchy Asymmetry/Heterogeniety Concurrency Primitives Power