University of Michigan, Ann Arbor OuterSPACE An Outer Product based Sparse Matrix Multiplication Accelerator 28 February, 2018 Subhankar Pal, Jonathan Beaumont, Dong-Hyeon Park, Aporva Amarnath, Siying Feng, Chaitali Chakrabarti†, Hun-Seok Kim, David Blaauw, Trevor Mudge and Ronald Dreslinski University of Michigan, Ann Arbor †Arizona State University HPCA 2018

Overview Introduction Algorithm Architecture Evaluation Conclusion HPCA 2018

Overview Introduction Algorithm Architecture Evaluation Conclusion HPCA 2018

The Big Data Problem Big data collected from various sources Sensor feed, social media, scientific experiments Challenge: the nature of data is sparse Architecture research previously focused on improving compute Sparse matrix computation: a key example of memory bound workloads GPUs achieve ~100 GFLOPS for dense matrix mult. vs. ~100 MFLOPS for sparse Two dominant kernels Sparse matrix-matrix multiplication (SpGEMM) Breadth-first search, algebraic multigrid methods Sparse matrix-vector multiplication (SpMV) PageRank, support vector machines, ML based text analytics Talk about importance of the algorithms rather than describe how they work - Talk about how memory-bound workloads such as SpGEMM have gained importance as opposed to before Graphs become adjacency matrices, for example; many fundamental problems get converted to SPMM Animate the points Point to sparse matrix image HPCA 2018

Inner Product based Matrix Multiplication 𝑐 𝑖,𝑗 = 𝑘=0 𝑁−1 𝑎 𝑖,𝑘 × 𝑏 𝑘,𝑗 HPCA 2018

Outer Product based Matrix Multiplication 𝐂= 𝑖=0 𝑁−1 𝐂 𝑖 = 𝑖=0 𝑁−1 𝐚 𝑖 𝐛 𝑖 HPCA 2018

Outer Product based Matrix Multiplication + No index matching + Low reuse distance 𝐂= 𝑖=0 𝑁−1 𝐂 𝑖 = 𝑖=0 𝑁−1 𝐚 𝑖 𝐛 𝑖 HPCA 2018

Comparison of the Approaches Inner Product Outer Product Entries need to be index-matched before multiplication Reuse distance is high; each column of B is reloaded after multiplication of a row of A with all columns of B All non-zero pairs belonging to a column of A and row of B produce meaningful outputs Reuse distance is small; a pair of column of A and row of B are multiplied and never used again ✔ ✔ ∙ ∙ ∙ ∙ ∙ ∙ ∙ Pause at the end of slide ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ✔ ∙ ∙ ∙ ✔ ✔ ∙ ∙ ∙ HPCA 2018

Overview Introduction Algorithm Architecture Evaluation Conclusion HPCA 2018

Outer Product Implementation Figure: Implementation of outer product algorithm in a system with parallel Processing Units HPCA 2018

Outer Product Implementation Figure: The outer product algorithm implementation Multiply phase Processing units multiply an element of a column of A with a row of B Issued as a single “multiply task”: A[i][j] x Brow[j] Merge phase A processing unit merges all partial products pertaining to a row of C Uses a modified mergesort based approach to minimize memory traffic Talk about the mergesort approach; just mention that there is no data sharing in merge; reduce the red text; blow up figures; (1,2 ) and (3,4) go in diff lines Fix caption animations HPCA 2018

Performance on Traditional Hardware: CPU Each matrix has 1 million uniformly distributed non-zeros Matrix dimension Figure: Performance of the outer product algorithm against Intel MKL on the CPU. Evaluated outer product against MKL on Broadwell CPU The outer product algorithm puts high pressure on the memory system N N×N matrices streamed out to main memory during multiply Loaded back in during merge Merge phase implementation involves no sharing, leading to cache thrashing HPCA 2018

Performance on Traditional Hardware: CPU Each matrix has 1 million uniformly distributed non-zeros Performance bottlenecked by the restricted cache hierarchy and compute parallelism Matrix dimension Figure: Performance of the outer product algorithm against Intel MKL on the CPU. Evaluated outer product against MKL on Broadwell CPU The outer product algorithm puts high pressure on the memory system N N×N matrices streamed out to main memory during multiply Loaded back in during merge Merge phase implementation involves no sharing, leading to cache thrashing HPCA 2018

Performance on Traditional Hardware: GPU Each matrix has 1 million uniformly distributed non-zeros Matrix dimension Figure: Performance of the outer product algorithm against CUSP on the GPU Evaluated outer product against CUSP on K40 GPU The multiply phase streams and processes data much faster than the CPU implementation, scaling roughly linearly with decreasing density The merge phase suffers from a much lower total throughput Resulted by control divergence between threads within a warp while executing the conditional branches in the merge phase HPCA 2018

Performance on Traditional Hardware: GPU Each matrix has 1 million uniformly distributed non-zeros Performance bottlenecked by the SIMD nature of warps Matrix dimension Figure: Performance of the outer product algorithm against CUSP on the GPU. Matrices Evaluated outer product against CUSP on K40 GPU The multiply phase streams and processes data much faster than the CPU implementation, scaling roughly linearly with decreasing density The merge phase suffers from a much lower total throughput Resulted by control divergence between threads within a warp while executing the conditional branches in the merge phase Solution: SPMD architecture! HPCA 2018

Overview Introduction Algorithm Architecture Evaluation Conclusion HPCA 2018

OuterSPACE: Multiply Phase Figure: OuterSPACE architecture for the multiply phase. SPMD-style Processing Elements (PEs), high-speed crossbars and non- coherent caches with request coalescing, HBM interface Local Control Processor (LCP): streaming instructions in to the PEs Central Control Processor (CCP): work scheduling and memory management Talk about the PE architecture itself! Animate data movement through the architecture HPCA 2018

OuterSPACE: Multiply Phase Column of A Row of B Figure: OuterSPACE architecture for the multiply phase. Talk more about the architecture; talk about what’s in the PEs HPCA 2018

OuterSPACE: Multiply Phase Figure: Reconfigured architecture for the multiply phase. L0 caches contain elements of rows of B that are shared between PEs L1 caches act as as “victim” caches Hold hot data that may get evicted from L0 if multiple PEs are working simultaneously on different rows Talk about the PE architecture itself! Animate data movement through the architecture HPCA 2018

OuterSPACE: Merge Phase 1 2 2 Partial rows 2 3 5 1 2 2 1 2 Figure: Reconfigured architecture for the merge phase. The L0 cache-crossbar blocks are reconfigured to accommodate the change in data access pattern L0 reconfigured into smaller, private caches and private scratchpads A PE-pair merges all the partial products belonging to one row They work on each final row independent of other PE-pairs Thus, the merge phase involves no data sharing among PEs HPCA 2018

OuterSPACE: Merge Phase Figure: Reconfigured architecture for the merge phase. Half of the PEs are turned off and the rest work in pairs to merge outer products A “fetcher” PE initiates loads to fetch the partial products A “sorter” PE sorts the previously-fetched partial products and merges upon collision Turning off PEs saves power and regulates bandwidth allocation No synchronization => no coherence! HPCA 2018

Overview Introduction Algorithm Architecture Evaluation Conclusion HPCA 2018

Simulation Methodology Modeled the PEs, cache- crossbar hierarchy and HBM using gem5 Trace-based simulation Generated memory instruction traces offline Fed them to the PEs in the simulator Processing Element (PE) 1.5 GHz, 64-entry requestQ 16 PEs/tile for multiply 8 PEs/tile for merge L0 cache/scratchpad (SPM) 16 kB, 4-way per tile for multiply 2 kB cache, 2 kB SPM per tile for merge L1 cache 4 kB, 2-way Crossbar 16x16 and 4x4, swizzle-switch based Main memory HBM 2.0, 8000 MB/s per channel, 16 channels Make the table bigger Table: gem5 simulation parameters Evaluated against state-of-the-art SpGEMM packages Intel MKL on 6-core Intel Broadwell CPU NVIDIA cuSPARSE & CUSP on K40 GPU HPCA 2018

Performance of Matrix-Matrix Multiplication Figure: Evaluation of SpGEMM on UFL SuiteSparse and SNAP matrices Red circle around bars when talking about specific matrices; Refer to photo you clicked HPCA 2018

Performance of Matrix-Matrix Multiplication Figure: Evaluation of SpGEMM on UFL SuiteSparse and SNAP matrices Sync with the animations; color code boxes based on point; explain each bar; color in the matrices represent nonzeros Greater speedups for irregular matrices; inner product incurs large # of comparisons MKL performs poorly on power law graphs Uneven non-zero distribution increases merge time HPCA 2018

Performance of Matrix-Vector Multiplication Matrix dimension (ms) Figure: Speedups for multiplication with vector of varying density (r) Figure: Performance scaling for multiplication with a dense vector Each matrix has 1 million uniformly distributed non-zeros ∼10x gain in speedup with 10x reduction in vector density for SpM-SpV multiplication color code the table cells The outer product method only accesses matrix columns that match the indices of the non-zero elements of the vector Eliminates all redundant accesses to the matrix The outer product algorithm scales with number of non-zeros and not with the density/dimensions of the matrix HPCA 2018

Table: Power and area estimates for OuterSPACE in 32 nm Total chip area for OuterSPACE: ∼87 mm2 @ 24 W power budget Average throughput of 2.9 GFLOPS  126 MFLOPS/W GPU achieves avg. 67 MFLOPS @ 85 W for UFL/SNAP matrices Mention peak gflops GPUs achieve on dense matrix mult/ NN or other compute bound workloads; compare that with gflops achieved for sparse workloads; highlight 150x power efficiency improvement over GPU HPCA 2018

Overview Introduction Algorithm Architecture Evaluation Conclusion HPCA 2018

Summary Explored the outer product approach for SpGEMM Discovered inefficiencies in existing hardware leading to sub-optimal performance Designed a custom architecture following an SPMD paradigm that reconfigures to support different data access patterns Our architecture minimizes memory accesses though efficient use of the cache-crossbar hierarchy Demonstrated acceleration of SpGEMM, a prominent memory-bound kernel Evaluated the outer product algorithm on artificial and real-world workloads OuterSPACE achieves speedups of 7.9-14.0x over commercial SpGEMM libraries on CPU and GPU Mention Speedup numbers HPCA 2018

Questions? HPCA 2018