Download presentation
Presentation is loading. Please wait.
1
Chimera: Collaborative Preemption for Multitasking on a Shared GPU
Jason Jong Kyu Park1, Yongjun Park2, and Scott Mahlke1 1University of Michigan, Ann Arbor 2Hongik University
2
GPUs in Modern Computer Systems
GPU is now a default component in modern computer systems Servers, desktops, laptops, etc. Mobile devices Offloads data-parallel kernels CUDA, OpenCL, and etc.
3
GPU Execution Model threads thread blocks
4
Multitasking Needs in GPUs
Augmented Reality Bodytrack 3D rendering Graphics Data parallel algorithm ….
5
Traditional Context Switching
GTX 780 (Kepler) 256kB registers + 48kB shared memory 288.4 GB/s for 12 SMs ~88us per SM (~1us for CPUs) K1 Context Save Context Load K2 K2 launches Time
6
Challenge 1: Preemption latency
GTX 780 (Kepler) 256kB registers + 48kB shared memory Too long for latency-critical kernels 288.4 GB/s for 12 SMs ~88us per SM K1 Context Save Context Load K2 K2 launches Time
7
Challenge 2: Throughput overhead
GTX 780 (Kepler) 256kB registers + 48kB shared memory 288.4 GB/s for 12 SMs ~88us per SM K1 Context Save Context Load K2 No useful work is done K2 launches Time
8
Objective of This Work Prior work Switch Thread Block Progress (%)
Preemption Cost Prior work 0% 100% Thread Block Progress (%)
9
SM draining [Tanasic’ 14]
No issue Thread block K1 K2 K2 launches Time
![SM draining [Tanasic’ 14]](http://slideplayer.com./slide/3542121/12/images/9/SM+draining+%5BTanasic%E2%80%99+14%5D.jpg "No issue. Thread block. K1. K2. K2 launches. Time.")
10
Chimera Opportunity Switch!! Drain!! Switch Drain
Preemption Cost Opportunity Drain 0% 100% Thread Block Progress (%)
11
Idempotent if read state is not modified
SM Flushing Instant preemption Throw away what was running on the SM Re-execute from the beginning later (Idempotent kernel) CPU Idempotent if read state is not modified GPU Global Memory Only observable state
12
Finding Relaxed Idempotence
Detected by compiler CUDA __global__ void kernel_cuda(const float *in, float* out, float* inout) { … = inout[idx]; … atomicAdd(…); … out[idx] = …; inout[idx] = …; } Atomic Operation Idempotent Region Global Overwrite
13
Chimera Flush near the beginning Context switch in the middle
Preemption Cost Optimal Drain 0% 100% Thread Block Progress (%) Flush near the beginning Context switch in the middle Drain near the end
14
Independent Thread Block Execution
No shared state between SMs and thread blocks Each SM/thread block can be preempted with different preemption technique GPU SM No shared state … Thread block SM
15
Chimera Collaborative Preemption … Progress Flush GPU Thread SM Drain
Thread block … SM Switch
16
Thread Block Scheduler
Architecture SM Scheduling Policy How many SMs will each kernel have? Two level scheduler Kernel scheduler + thread block scheduler Kernel Scheduler SM Scheduling Policy TB-to-Kernel Mapping Thread Block Scheduler Chimera
17
Thread Block Scheduler
Architecture Chimera Which SM will be preempted? Which preemption technique to use? Kernel Scheduler SM Scheduling Policy TB-to-Kernel Mapping Thread Block Scheduler Chimera
18
Architecture Thread block scheduler
Which thread block will be scheduled? Carry out preemption decision Kernel Scheduler Thread Block Queue per Kernel Preempted TB TB-to-Kernel Mapping Next TB Thread Block Scheduler Preempt
19
Cost Estimation: Preemption Latency
Switch Context size / (Memory bandwidth / # of SMs) Drain Instructions measured in a warp granularity Progress in insts Average execution insts: Estimated remaining insts x CPI = Estimated preemption latency Flush Zero preemption latency
20
Cost Estimation: Throughput
Switch IPC * Preemption latency * 2 Doubled due to context saving and loading Drain Instructions measured in a warp granularity Progress in insts Most progressed in the same SM: Overhead Flush Executed instructions in a warp granularity
21
Chimera Algorithm Preemption victim Least throughput overhead
Meets preemption latency constraint Flush GPU SM Flush : … Switch: Thread Drain : SM Thread block Switch
22
Chimera Algorithm Preemption victim Least throughput overhead
Meets preemption latency constraint Constraint Flush GPU SM Latency : … Overhead : Thread Latency : SM Thread block Overhead : Switch
23
Experimental Setup GPGPU-Sim v3.2.2 Workloads
GPU Model: Fermi architecture Up to 32,768 (128 kB) registers Up to 48 kB shared memory Workloads 14 benchmarks from Nvidia SDK, Parboil, and Rodinia GPGPU benchmark + Synthetic benchmark Mimics periodic, real-time task (e.g. Graphics kernel) Period: 1ms Execution Time: 200us GPGPU benchmark + GPGPU benchmark Baseline: Non-preemptive First-come First-served
24
Preemption Latency Violations
GPGPU benchmark + Synthetic benchmark 15 us preemption latency constraint (real-time task) 0.2% Non-idempotent kernel with short thread block execution time Estimated shorter preemption latency
25
System Throughput Case study: LUD + Other GPGPU benchmark
LUD has many kernel launches with varying number of thread blocks Drain has lower average normalized turnaround time (5.17x for Drain, 5.50x for Chimera)
26
Preemption Technique Distribution
GPGPU benchmark + Synthetic benchmark
27
Summary Context switch can have high overhead on GPUs Chimera Flushing
Preemption latency, and throughput overhead Chimera Flushing Instant preemption Collaborative preemption Flush + Switch + Drain Almost always meets preemption latency constraint 0.2% violations (estimated shorter preemption latency) 5.5x ANTT improvement, 12.2% STP improvement For GPGPU benchmark + GPGPU benchmark combinations
28
Questions? Drain Context Switch Flush
Similar presentations
