Portable Inter-workgroup Barrier Synchronisation for GPUs Tyler Sorensen, Alastair Donaldson Imperial College London, UK Mark Batty University of Kent, UK Ganesh Gopalakrishnan, Zvonimir Rakamaric University of Utah OOPSLA 2016

Barriers Provided as primitives MPI OpenMP Pthreads GPU (intra-workgroup) T0 T1 T2 T3 …… barrier ……

GPU programming Threads Global Memory

GPU programming Global Memory Workgroup 0 Workgroup 1 Workgroup n Threads Local memory for WG0 Local memory for WG1 Local memory for WGn Global Memory

Inter-workgroup barrier Not provided as primitive Building blocks provided: Inter-workgroup shared memory

Experimental results Chip Vendor Compute Units OpenCL Version Type GTX 980 Nvidia 16 1.1 Discrete Quadro K500 12 Iris 6100 Intel 47 2.0 Integrated HD 5500 24 Radeon R9 AMD 28 Radeon R7 8 T628-4 ARM 4 1.2 T628-2 2 integrated

Challenges Scheduling Memory consistency

Occupancy bound execution Program with 5 workgroups w5 w4 w2 w1 w0 workgroup queue CU CU CU GPU with 3 compute units

Occupancy bound execution Program with 5 workgroups w5 w4 w2 w1 w0 workgroup queue CU CU CU GPU with 3 compute units

Occupancy bound execution Program with 5 workgroups w5 w4 w4 workgroup queue w0 w1 w2 CU CU CU GPU with 3 compute units

Occupancy bound execution Program with 5 workgroups w5 w4 w4 workgroup queue w0 w1 CU CU CU w2 GPU with 3 compute units finished workgroups

Occupancy bound execution Program with 5 workgroups w5 w4 workgroup queue w0 w1 w4 CU CU CU w2 GPU with 3 compute units finished workgroups

Occupancy bound execution Program with 5 workgroups w5 w4 workgroup queue w1 w4 CU CU CU w2 w0 GPU with 3 compute units finished workgroups

Occupancy bound execution Program with 5 workgroups w4 workgroup queue w5 w1 w4 CU CU CU w2 w0 GPU with 3 compute units finished workgroups

Occupancy bound execution Program with 5 workgroups w4 workgroup queue Finished! CU CU CU w2 w0 w5 w1 w4 GPU with 3 compute units finished workgroups

Occupancy bound execution Program with 5 workgroups w5 w4 w2 w1 w0 workgroup queue CU CU CU GPU with 3 compute units

Occupancy bound execution Program with 5 workgroups w5 w4 w2 w1 w0 workgroup queue CU CU CU GPU with 3 compute units

Occupancy bound execution Program with 5 workgroups w5 w4 w4 workgroup queue w0 w1 w2 Cannot synchronise with workgroups in queue CU CU CU Barrier gives deadlock! GPU with 3 compute units

Occupancy bound execution Program with 5 workgroups w5 w4 w4 workgroup queue w0 w1 w2 Barrier is possible if we know the occupancy CU CU CU GPU with 3 compute units

Occupancy bound execution Launch as many workgroups as compute units?

Occupancy bound execution Launch as many workgroups as compute units? Program with 5 workgroups w5 w4 w2 w1 w0 workgroup queue CU CU CU GPU with 3 compute units

Occupancy bound execution Launch as many workgroups as compute units? Program with 5 workgroups w4 workgroup queue w0 w1 w4 w2 w5 Depending on resources, multiple wgs can execute on CU CU CU CU GPU with 3 compute units

Recall of occupancy discovery Chip Compute Units Occupancy Bound GTX 980 16 Quadro K500 12 Iris 6100 47 HD 5500 24 Radeon R9 28 Radeon R7 8 T628-4 4 T628-2 2

Recall of occupancy discovery Chip Compute Units Occupancy Bound GTX 980 16 32 Quadro K500 12 Iris 6100 47 6 HD 5500 24 3 Radeon R9 28 48 Radeon R7 8 T628-4 4 T628-2 2

Our approach (scheduling) wa w9 w8 w7 w6 Program w5 w4 w2 w1 w0 workgroup queue CU CU CU GPU with 3 compute units

Our approach (scheduling) w8 w6 Program w4 w5 w1 workgroup queue w2 w9 w7 w0 w4 wa CU CU CU GPU with 3 compute units

Our approach (scheduling) w8 w6 Program w4 w5 w1 workgroup queue w2 w9 w7 w0 w4 wa Dynamically estimate occupancy CU CU CU GPU with 3 compute units

Our approach (scheduling) w8 w6 Program w4 w5 w1 workgroup queue w2 w9 w7 w0 w4 wa Dynamically estimate occupancy CU CU CU GPU with 3 compute units

Finding occupant workgroups Executed by 1 thread per workgroup Two phases: Polling Closing

Finding occupant workgroups Executed by 1 thread per workgroup Three global variables: Mutex: m Bool poll flag: poll_open Integer counter: count

Finding occupant workgroups lock(m) if (poll_open) { count++; unlock(m); } else { return false; poll_open = false unlock(m) return true; Executed by 1 thread per workgroup Three global variables: Mutex: m Bool poll flag: poll_open Integer counter: count

Finding occupant workgroups Polling phase lock(m) if (poll_open) { count++; unlock(m); } else { return false; poll_open = false unlock(m) return true; Executed by 1 thread per workgroup Three global variables: Mutex: m Bool poll flag: poll_open Integer counter: count

Finding occupant workgroups Polling phase lock(m) if (poll_open) { count++; unlock(m); } else { return false; poll_open = false unlock(m) return true; Executed by 1 thread per workgroup Three global variables: Mutex: m Bool poll flag: poll_open Integer counter: count Closing phase

Finding occupant workgroups lock(m) if (poll_open) { count++; unlock(m); } else { return false; poll_open = false unlock(m) return true; Executed by 1 thread per workgroup Three global variables: Mutex: m Bool poll flag: poll_open Integer counter: count

Finding occupant workgroups lock(m) if (poll_open) { count++; unlock(m); } else { return false; poll_open = false unlock(m) return true; Executed by 1 thread per workgroup Three global variables: Mutex: m Bool poll flag: poll_open Integer counter: count

Recall of occupancy discovery Chip Compute Units Occupancy Bound Spin Lock Ticket Lock GTX 980 16 32 Quadro K500 12 Iris 6100 47 6 HD 5500 24 3 Radeon R9 28 48 Radeon R7 8 T628-4 4 T628-2 2

Recall of occupancy discovery Chip Compute Units Occupancy Bound Spin Lock Ticket Lock GTX 980 16 32 3.1 Quadro K500 12 2.3 Iris 6100 47 6 3.5 HD 5500 24 3 2.7 Radeon R9 28 48 7.7 Radeon R7 8 4.6 T628-4 4 T628-2 2 2.0

Recall of occupancy discovery Chip Compute Units Occupancy Bound Spin Lock Ticket Lock GTX 980 16 32 3.1 32.0 Quadro K500 12 2.3 12.0 Iris 6100 47 6 3.5 6.0 HD 5500 24 3 2.7 3.0 Radeon R9 28 48 7.7 48.0 Radeon R7 8 4.6 16.0 T628-4 4 4.0 T628-2 2 2.0

Challenges Scheduling Memory consistency

Our approach (memory consistency) …… barrier m1 ……

Our approach (memory consistency) …… barrier HB m1 ……

Our approach (memory consistency) …… Barrier implementation creates HB web Using OpenCL 2.0 atomic operations ……

Barrier implementation Start with an implementation from Xiao and Feng (2010) Written in CUDA (ported to OpenCL) No formal memory consistency properties

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 barrier barrier spin(x0 != 1) spin(x1 != 1)

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 barrier barrier spin(x0 != 1) spin(x1 != 1) barrier barrier

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 barrier barrier x0 = 1 x1 = 1 spin(x0 != 1) spin(x1 != 1) barrier barrier

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 barrier barrier x0 = 1 x1 = 1 spin(x0 != 1) spin(x1 != 1) barrier spin(x0 != 0) spin(x1 != 0) barrier barrier

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 barrier barrier x0 = 1 x1 = 1 spin(x0 != 1) spin(x1 != 1) barrier x0 = 0 x1 = 0 spin(x0 != 0) spin(x1 != 0) barrier barrier

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 barrier barrier x0 = 1 x1 = 1 spin(x0 != 1) spin(x1 != 1) barrier x0 = 0 x1 = 0 spin(x0 != 0) spin(x1 != 0) barrier barrier

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 x0 = 1 x1 = 1 spin(x0 != 1) spin(x1 != 1) x0 = 0 x1 = 0 spin(x0 != 0) spin(x1 != 0)

Barrier memory consistency Device release-acquire rule T0 and T1 in different workgroups T0 T1 Wrel y = 1 Racq y = 1

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 x0 = 1 x1 = 1 spin(x0 != 1) spin(x1 != 1) x0 = 0 x1 = 0 spin(x0 != 0) spin(x1 != 0)

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 Wrel x0 = 1 Wrel x1 = 1 spin(x0 != 1) spin(x1 != 1) Wrel x0 = 0 Wrel x1 = 0 spin(x0 != 0) spin(x1 != 0)

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 Wrel x0 = 1 Wrel x1 = 1 spin(Racqx0 != 1) spin(Racqx1 != 1) Wrel x0 = 0 Wrel x1 = 0 spin(Racqx0 != 0) spin(Racqx1 != 0)

Slave Workgroup 0 Master Workgroup Slave Workgroup 1 T1 T0 T0 T1 T0 T1 Wrel x0 = 1 Wrel x1 = 1 spin(Racqx0 != 1) spin(Racqx1 != 1) Wrel x0 = 0 Wrel x1 = 0 spin(Racqx0 != 0) spin(Racqx1 != 0)

Barrier vs. multi-kernel Pannotia Target AMD Radeon HD 7000 Written in OpenCL 1.x 4 graph algorithms applications https://github.com/pannotia/pannotia

Barrier vs. multi-kernel Device Host GPU_linear_algebra_routine1; global_barrier(); GPU_linear_algebra_routine2; global_barrier(); GPU_linear_algebra_routine3; global_barrier(); GPU_linear_algebra_routine1; GPU_linear_algebra_routine2; GPU_linear_algebra_routine3; Loop until a fixed point is reached. Loop until a fixed point is reached.

Barrier vs. multi-kernel Up to 2.2x speedup (ARM) Always observed a speedup on Intel (1.25x median) Broke even with Nvidia (1.0x median) In some cases a slowdown (0.22x on ARM)

Barrier vs. multi-kernel Up to 2.2x speedup (ARM) Always observed a speedup on Intel (1.25x median) Broke even with Nvidia (1.0x median) In some cases a slowdown (0.22x on ARM) Fine grained performance model for tuning needed!

Portable GPU Barrier Synchronisation Designed a GPU inter-workgroup barrier portable under occupancy bound execution and OpenCL memory model Evaluated on 8 GPUs (4 vendors) Nearly perfect recall on modern GPUs Can provide substantial performance gains in applications Try it out now! https://github.com/mc-imperial/gpu_discovery_barrier