University of Michigan Electrical Engineering and Computer Science Adaptive Input-aware Compilation for Graphics Engines Mehrzad Samadi 1, Amir Hormati 2, Mojtaba Mehrara 3, Janghaeng Lee 1 and Scott Mahlke University of Michigan - Ann Arbor 2 Microsoft Research 3 NVIDIA Research

University of Michigan Electrical Engineering and Computer Science GPU Performance Gap High performance at low cost Peak performance is difficult to achieve 2 GeForce GTX 480 GeForce GTX 280 GeForce 8800 GTX GeForce 7800 GTX GeForce GTX 590 GeForce GTX 680 In Practice

University of Michigan Electrical Engineering and Computer Science TMV Performance on Various Input 3 Square Matrix Rectangular Matrix

University of Michigan Electrical Engineering and Computer Science GPU Execution Model 4 Grid 1 SM 0 Shared Regs SM 1 Shared Regs SM 2 Shared Regs SM 3 Shared Regs SM 7 Shared Regs Executes Thread

University of Michigan Electrical Engineering and Computer Science Transposed Matrix Vector Multiplication (4 x 1M) 5 SM 0 Block 0 Thread 0 ~ 15 Block 3 Block 1 Block Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared IDLE

University of Michigan Electrical Engineering and Computer Science Transposed Matrix Vector Multiplication (1M x 4) 6 SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared SM Regs Shared Block 0 ~ 7 Block 8 ~ 15 Block 1,000, ,000 blocks / SM

University of Michigan Electrical Engineering and Computer Science GPU Programming Challenge - Portability 7 GPU Architectures Input Matrix SizeSource Code 4 x 1MGTX285_MV_4_1M.cu 128 x 32KGTX285_MV_128_32K.cu 32K x 128GTX285_MV_32K_128.cu 1M x 4GTX285_MV_1M_4.cu 4 x 1MGTX580_MV_4_1M.cu 128 x 32KGTX580_MV_128_32K.cu 32K x 128GTX580_MV_32K_128.cu 1M x 4GTX580_MV_1M_4.cu 4 x 1MGTX680_MV_4_1M.cu 128 x 32KGTX680_MV_128_32K.cu 32K x 128GTX680_MV_32K_128.cu 1M x 4GTX680_MV_1M_4.cu Fastest Matrix-Vector Multiplication for any GPU for any input size Cores : 240 Cores : 512 Cores :

University of Michigan Electrical Engineering and Computer Science Adaptic Adaptive Input-aware Compilation for GPUs –Device-Portable –Input-Portable –Programmers can focus on the algorithms without concerning about low-level details Streaming Language –Higher-level of abstraction –Separating Memory Access from Algorithm –e.g) StreamIt 8

University of Michigan Electrical Engineering and Computer Science Stream It Higher-level of abstraction Decoupling computation and memory accesses Coarse grain exposed parallelism, exposed communication Streaming actors use buffers to communicate A lot of recent works on extending portability of streaming applications 9

University of Michigan Electrical Engineering and Computer Science Compilation Flow in Adaptic 10 Input-aware Optimization Input-unaware Optimization StreamIt Code Target GPUInput Range Offline Compilation Performance Model Memory Access Optimization Actor Segmentation Actor Integration Why? Global Memory Accesses Large access latency Optimizations Memory Restructuring Coalesced Access Neighboring Access Data Reuse Splits Actors More blocks will be generated Alleviate resource under-utilization Optimizations Stream Reduction Intra-actor Parallelization Integrate Actors Merge several actors into one Alleviate high resource contention Optimizations Vertical Integration Horizontal Integration Executable Smallest Input Largest Input Small Input Large Input Input size? Launch Kernel Kernel 0Kernel 1Kernel 2Kernel 3 Several CUDA Kernels for various input range

University of Michigan Electrical Engineering and Computer Science Memory Optimization Global Memory - Large access latency Not access the words in sequence No coalescing 11 A[i, j]  Actor A has i pops and j pushes Thread 1 Thread 2 Thread 3 Thread A[4,4] Global Memory A[4,4]

University of Michigan Electrical Engineering and Computer Science Memory Optimization Global Memory - Large access latency Not access the words in sequence No coalescing 12 Thread 1 Thread 2 Thread 3 Thread A[4,4] Global Memory A[4,4] A[i, j]  Actor A has i pops and j pushes

University of Michigan Electrical Engineering and Computer Science Actor Segmentation 13 4 x 1M Transposed Matrix-Vector Multiplication Block 0 Block 3 Block 1 Block 2 Block 96 Block 32 Block 64 ~ Block 0 Block 31

University of Michigan Electrical Engineering and Computer Science Actor Integration Merges several actors or threads to balance threads’ workloads Vertical integration: reducing off-chip memory traffic by storing intermediate results in the shared memory. Horizontal integration : reducing synchronization overhead and also lets the merged actors share instructions. 14

University of Michigan Electrical Engineering and Computer Science Experimental Setup CPU - Intel Xeon X5650 GPU –NVidia Telsa C2050 3GB GDDR 5 –NVidia GTX 285 2GB GDDR 2 Benchmarks –CUBLAS Library 3.2 –NVidia SDK

University of Michigan Electrical Engineering and Computer Science Result( Matrix Vector Multlipication) 16

University of Michigan Electrical Engineering and Computer Science Results (Speedup) 17 Input Size

University of Michigan Electrical Engineering and Computer Science Results(BiCGSTAB) 18 Input unaware

University of Michigan Electrical Engineering and Computer Science Summary Performance of GPU is affected by –GPU Model / Input CUDA / OpenCL Programming Model –Lacks Architecture and Input Portability Scientific Applications use irregular input –Hard to get optimized performance Proposed Adaptic –Architecture and input portable /w streaming language –Showed speedup over CUBLAS / SDK in various input range 19

University of Michigan Electrical Engineering and Computer Science Q & A 20