Presentation is loading. Please wait.

Presentation is loading. Please wait.

CUDA Performance Considerations (1 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2011.

Published byCuthbert Stevenson Modified over 9 years ago

Similar presentations

Presentation on theme: "CUDA Performance Considerations (1 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2011."— Presentation transcript:

1 CUDA Performance Considerations (1 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2011

2 Administrivia Presentation topic due today  via email by 11:59pm  Or I pick your topic  Not bonus day eligible Assignment 4 due Friday, 03/04

3 Agenda Warps Revisited Parallel Reduction Revisited Warp Partitioning Memory Coalescing Dynamic Partitioning of SM Resources Data Prefetching

4 Warps Recall a warp  Group of threads from a block G80 / GT200 – 32 threads per warp  Each thread executes the same instruction on different data  The unit of scheduling  An implementation detail Not if you want performance

5 Warps 32 threads per warp but 8 SPs per SM. What gives?

6 Warps 32 threads per warp but 8 SPs per SM. What gives? When an SM schedules a warp:  Its instruction is ready  8 threads enter the SPs on the 1 st cycle  8 more on the 2 nd, 3 rd, and 4 th cycles  Therefore, 4 cycles are required to execute any instruction

7 Warps Question  A kernel has 1 global memory read ( 200 cycles) 4 non-dependent multiples/adds  How many warps are required to hide the memory latency?

8 Warps Solution  Each warp has 4 multiples/adds 16 cycles  We need to cover 200 cycles 200 / 16 = 12.5 ceil(12.5) = 13  13 warps are required

9 Efficient data- parallel algorithms Optimizations based on GPU Architecture Maximum Performance + =

10 Parallel Reduction 0 1 5 2 3 4 6 7 Recall Parallel Reduction (sum)

11 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913

12 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22

13 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22 28

14 Parallel Reduction Similar to brackets for a basketball tournament log( n ) passes for n elements How would you implement this in CUDA?

15 __shared__ float partialSum[]; //... load into shared memory unsigned int t = threadIdx.x; for (unsigned int stride = 1; stride < blockDim.x; stride *= 2) { __syncthreads(); if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; } Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html

16 __shared__ float partialSum[]; //... load into shared memory unsigned int t = threadIdx.x; for (unsigned int stride = 1; stride < blockDim.x; stride *= 2) { __syncthreads(); if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; } Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html Computing the sum for the elements in shared memory

17 __shared__ float partialSum[]; //... load into shared memory unsigned int t = threadIdx.x; for (unsigned int stride = 1; stride < blockDim.x; stride *= 2) { __syncthreads(); if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; } Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html Stride : 1, 2, 4, …

18 __shared__ float partialSum[]; //... load into shared memory unsigned int t = threadIdx.x; for (unsigned int stride = 1; stride < blockDim.x; stride *= 2) { __syncthreads(); if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; } Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html Why?

19 __shared__ float partialSum[]; //... load into shared memory unsigned int t = threadIdx.x; for (unsigned int stride = 1; stride < blockDim.x; stride *= 2) { __syncthreads(); if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; } Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html Compute sum in same shared memory As stride increases, what do more threads do?

20 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22 28 Thread 0 Thread 1

21 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22 28 Thread 0 Thread 1 1 st pass: threads 1, 3, 5, and 7 don’t do anything  Really only need n/2 threads for n elements

22 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22 28 Thread 0 Thread 1 2 nd pass: threads 2 and 6 also don’t do anything

23 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22 28 Thread 0 Thread 1 3 rd pass: thread 4 also doesn’t do anything

24 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Parallel Reduction 0 1 5 2 3 4 6 7 1 5 913 6 22 28 Thread 0 Thread 1 In general, number of required threads cuts in half after each pass

25 Parallel Reduction What if we tweaked the implementation?

26 Parallel Reduction 0 1 5 2 3 4 6 7

27 0 1 5 2 3 4 6 7 4 6 810

28 Parallel Reduction 0 1 5 2 3 4 6 7 4 6 810 1216

29 Parallel Reduction 0 1 5 2 3 4 6 7 4 6 810 1216 28

30 Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html stride : …, 4, 2, 1 __shared__ float partialSum[] //... load into shared memory unsigned int t = threadIdx.x; for(unsigned int stride = blockDim.x >> 1; stride > 0; stride >> 1) { __syncthreads(); if (t < stride) partialSum[t] += partialSum[t + stride]; }

31 Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html __shared__ float partialSum[] //... load into shared memory unsigned int t = threadIdx.x; for(unsigned int stride = blockDim.x >> 1; stride > 0; stride >> 1) { __syncthreads(); if (t < stride) partialSum[t] += partialSum[t + stride]; }

32 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Thread 0 Thread 1 Parallel Reduction 0 1 5 2 3 4 6 7 4 6 810 1216 28

33 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Thread 0 Thread 1 Parallel Reduction 0 1 5 2 3 4 6 7 4 6 810 1216 28 1 st pass: threads 4, 5, 6, and 7 don’t do anything  Really only need n/2 threads for n elements

34 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Thread 0 Thread 1 Parallel Reduction 0 1 5 2 3 4 6 7 4 6 810 1216 28 2 nd pass: threads 2 and 3 also don’t do anything

35 Thread 7 Thread 6 Thread 5 Thread 4 Thread 3 Thread 2 Thread 0 Thread 1 Parallel Reduction 0 1 5 2 3 4 6 7 4 6 810 1216 28 3 rd pass: thread 1 also doesn’t do anything

36 Parallel Reduction 0 1 2 3 4 5 6 7 What is the difference? stride = 1, 2, 4, … stride = 4, 2, 1, …

37 Parallel Reduction What is the difference? if (t < stride) partialSum[t] += partialSum[t + stride]; if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; stride = 1, 2, 4, … stride = 4, 2, 1, …

38 Warp Partitioning Warp Partitioning: how threads from a block are divided into warps Knowledge of warp partitioning can be used to:  Minimize divergent branches  Retire warps early

39 Understand warp partitioning make your code run faster

40 Warp Partitioning Partition based on consecutive increasing threadIdx

41 Warp Partitioning 1D Block  threadIdx.x between 0 and 512 (G80/GT200)  Warp n Starts with thread 32n Ends with thread 32(n + 1) – 1  Last warp is padded if block size is not a multiple of 32 0…3132...6364...9596...127 Warp 0Warp 1Warp 2Warp 3 …

42 Warp Partitioning 2D Block  Increasing threadIdx means Increasing threadIdx.x Starting with row threadIdx.y == 0

43 Warp Partitioning Image from http://courses.engr.illinois.edu/ece498/al/textbook/Chapter5-CudaPerformance.pdf 2D Block

44 Warp Partitioning 3D Block  Start with threadIdx.z == 0  Partition as a 2D block  Increase threadIdx.z and repeat

45 Image from: http://bps10.idav.ucdavis.edu/talks/03-fatahalian_gpuArchTeraflop_BPS_SIGGRAPH2010.pdf Warp Partitioning Divergent branches are within a warp!

46 Warp Partitioning For warpSize == 32, does any warp have a divergent branch with this code: if (threadIdx.x > 15) { //... }

47 Warp Partitioning For any warpSize, does any warp have a divergent branch with this code: if (threadIdx.x > warpSize) { //... }

48 Warp Partitioning Given knowledge of warp partitioning, which parallel reduction is better? if (t < stride) partialSum[t] += partialSum[t + stride]; if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; stride = 1, 2, 4, … stride = 4, 2, 1, …

49 Warp Partitioning stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 Pretend warpSize == 2

50 Warp Partitioning 1 st Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 4 divergent branches 0 divergent branches

51 Warp Partitioning 2 nd Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 2 divergent branches 0 divergent branches

52 Warp Partitioning 2 nd Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 1 divergent branch 1 divergent branch

53 Warp Partitioning 2 nd Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 1 divergent branch 1 divergent branch Still diverge when number of elements left is <= warpSize

54 Warp Partitioning Good partitioning also allows warps to be retired early.  Better hardware utilization if (t < stride) partialSum[t] += partialSum[t + stride]; if (t % (2 * stride) == 0) partialSum[t] += partialSum[t + stride]; stride = 1, 2, 4, … stride = 4, 2, 1, …

55 Warp Partitioning stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 Parallel Reduction

56 Warp Partitioning 1 st Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 0 warps retired 2 warps retired

57 Warp Partitioning 1 st Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3

58 Warp Partitioning 2 nd Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3 1 warp retired 2 warps retired

59 Warp Partitioning 2 nd Pass stride = 1, 2, 4, … stride = 4, 2, 1, … Warp 0 Warp 1 Warp 2 Warp 3 Warp 0 Warp 1 Warp 2 Warp 3

60 Memory Coalescing Given a matrix stored row-major in global memory, what is a thread’s desirable access pattern? Image from: http://bps10.idav.ucdavis.edu/talks/03-fatahalian_gpuArchTeraflop_BPS_SIGGRAPH2010.pdf M 2,0 M 1,1 M 1,0 M 0,0 M 0,1 M 3,0 M 2,1 M 3,1 M 2,0 M 1,0 M 0,0 M 3,0 M 1,1 M 0,1 M 2,1 M 3,1 M 1,2 M 0,2 M 2,2 M 3,2 M 1,2 M 0,2 M 2,2 M 3,2 M 1,3 M 0,3 M 2,3 M 3,3 M 1,3 M 0,3 M 2,3 M 3,3 M

61 Memory Coalescing Given a matrix stored row-major in global memory, what is a thread’s desirable access pattern? Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter5-CudaPerformance.pdf Md Nd W I D T H WIDTH Thread 0 Thread 1 Thread 0 Thread 1 a) column after column?b) row after row?

62 Memory Coalescing Given a matrix stored row-major in global memory, what is a thread’s desirable access pattern?  a) column after column Individual threads read increasing, consecutive memory address  b) row after row Adjacent threads read increasing, consecutive memory addresses

63 Memory Coalescing Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter5-CudaPerformance.pdf a) column after column

64 Memory Coalescing Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter5-CudaPerformance.pdf b) row after row

65 Memory Coalescing Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter5-CudaPerformance.pdf Recall warp partitioning; if these threads are in the same warp, global memory addresses are increasing and consecutive across warps.

66 Memory Coalescing Global memory bandwidth (DRAM)  G80 – 86.4 GB/s  GT200 – 150 GB/s Achieve peak bandwidth by requesting large, consecutive locations from DRAM  Accessing random location results in much lower bandwidth

67 Memory Coalescing Memory coalescing – rearrange access patterns to improve performance Useful today but will be less useful with large on-chip caches

68 Memory Coalescing The GPU coalesce consecutive reads in a half-warp into a single read  What makes this possible? Strategy: read global memory in a coalesce-able fashion into shared memory  Then access shared memory randomly at maximum bandwidth Ignoring bank conflicts – next lecture See Appendix G in the NVIDIA CUDA C Programming Guide for coalescing alignment requirements

69 SM Resource Partitioning Recall a SM dynamically partitions resources: Registers Thread block slots Thread slots Shared memory SM

70 SM Resource Partitioning Recall a SM dynamically partitions resources: Registers Thread block slots Thread slots Shared memory SM 8 768 8K registers / 32K memory 16K G80 Limits

71 SM Resource Partitioning We can have  8 blocks of 96 threads  4 blocks of 192 threads  But not 8 blocks of 192 threads Registers Thread block slots Thread slots Shared memory SM 8 768 8K registers / 32K memory 16K G80 Limits

72 SM Resource Partitioning We can have (assuming 256 thread blocks)  768 threads (3 blocks) using 10 registers each  512 threads (2 blocks) using 11 registers each Registers Thread block slots Thread slots Shared memory SM 8 768 8K registers / 32K memory 16K G80 Limits

73 SM Resource Partitioning We can have (assuming 256 thread blocks)  768 threads (3 blocks) using 10 registers each  512 threads (2 blocks) using 11 registers each Registers Thread block slots Thread slots Shared memory SM 8 768 8K registers / 32K memory 16K G80 Limits More registers decreases thread- level parallelism Can it ever increase performance?

74 SM Resource Partitioning Performance Cliff: Increasing resource usage leads to a dramatic reduction in parallelism  For example, increasing the number of registers, unless doing so hides latency of global memory access

75 Data Prefetching Independent instructions between a global memory read and its use can hide memory latency float m = Md[i]; float f = a * b + c * d; float f2 = m *f;

76 Data Prefetching Independent instructions between a global memory read and its use can hide memory latency float m = Md[i]; float f = a * b + c * d; float f2 = m *f; Read global memory

77 Data Prefetching Independent instructions between a global memory read and its use can hide memory latency float m = Md[i]; float f = a * b + c * d; float f2 = m *f; Execute instructions that are not dependent on memory read

78 Data Prefetching Independent instructions between a global memory read and its use can hide memory latency float m = Md[i]; float f = a * b + c * d; float f2 = m *f; Use global memory after the above line from enough warps hide the memory latency

79 Data Prefetching Prefetching data from global memory can effectively increase the number of independent instructions between global memory read and use

80 Data Prefetching Recall tiled matrix multiply: for (/*... */) { // Load current tile into shared memory __syncthreads(); // Accumulate dot product __syncthreads(); }

81 Data Prefetching Tiled matrix multiply with prefect: // Load first tile into registers for (/*... */) { // Deposit registers into shared memory __syncthreads(); // Load next tile into registers // Accumulate dot product __syncthreads(); }

82 Data Prefetching Tiled matrix multiply with prefetch: // Load first tile into registers for (/*... */) { // Deposit registers into shared memory __syncthreads(); // Load next tile into registers // Accumulate dot product __syncthreads(); }

83 Data Prefetching Tiled matrix multiply with prefetch: // Load first tile into registers for (/*... */) { // Deposit registers into shared memory __syncthreads(); // Load next tile into registers // Accumulate dot product __syncthreads(); } Prefetch for next iteration of the loop

84 Data Prefetching Tiled matrix multiply with prefetch: // Load first tile into registers for (/*... */) { // Deposit registers into shared memory __syncthreads(); // Load next tile into registers // Accumulate dot product __syncthreads(); } These instructions executed by enough threads will hide the memory latency of the prefetch

Download ppt "CUDA Performance Considerations (1 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2011."

Similar presentations

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 1 More on Performance Considerations.

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 1 More on Performance Considerations.
Taking CUDA to Ludicrous Speed Getting Righteous Performance from your GPU 1.

Taking CUDA to Ludicrous Speed Getting Righteous Performance from your GPU 1.
1 ITCS 6/8010 CUDA Programming, UNC-Charlotte, B. Wilkinson, Jan 28, 2011 GPUMemories.ppt GPU Memories These notes will introduce: The basic memory hierarchy.

1 ITCS 6/8010 CUDA Programming, UNC-Charlotte, B. Wilkinson, Jan 28, 2011 GPUMemories.ppt GPU Memories These notes will introduce: The basic memory hierarchy.
Instructor Notes This lecture discusses three important optimizations The performance impact of mapping threads to data on the GPU is subtle but extremely.

Instructor Notes This lecture discusses three important optimizations The performance impact of mapping threads to data on the GPU is subtle but extremely.
CS 193G Lecture 5: Performance Considerations. But First! Always measure where your time is going! Even if you think you know where it is going Start.

CS 193G Lecture 5: Performance Considerations. But First! Always measure where your time is going! Even if you think you know where it is going Start.
L15: Review for Midterm. Administrative Project proposals due today at 5PM (hard deadline) – handin cs6963 prop March 31, MIDTERM in class L15: Review.

L15: Review for Midterm. Administrative Project proposals due today at 5PM (hard deadline) – handin cs6963 prop March 31, MIDTERM in class L15: Review.
L9: Control Flow CS6963. Administrative Project proposals Due 5PM, Friday, March 13 (hard deadline) MPM Sequential code and information posted on website.

L9: Control Flow CS6963. Administrative Project proposals Due 5PM, Friday, March 13 (hard deadline) MPM Sequential code and information posted on website.
L8: Memory Hierarchy Optimization, Bandwidth CS6963.

L8: Memory Hierarchy Optimization, Bandwidth CS6963.
Programming with CUDA WS 08/09 Lecture 9 Thu, 20 Nov, 2008.

Programming with CUDA WS 08/09 Lecture 9 Thu, 20 Nov, 2008.
Prepared 10/11/2011 by T. O’Neil for 3460:677, Fall 2011, The University of Akron.

Prepared 10/11/2011 by T. O’Neil for 3460:677, Fall 2011, The University of Akron.
Ceng 545 Performance Considerations. Memory Coalescing High Priority: Ensure global memory accesses are coalesced whenever possible. Off-chip memory is.

Ceng 545 Performance Considerations. Memory Coalescing High Priority: Ensure global memory accesses are coalesced whenever possible. Off-chip memory is.
© David Kirk/NVIDIA and Wen-mei W. Hwu Taiwan, June 30 – July 2, 2008 1 Taiwan 2008 CUDA Course Programming Massively Parallel Processors: the CUDA experience.

© David Kirk/NVIDIA and Wen-mei W. Hwu Taiwan, June 30 – July 2, Taiwan 2008 CUDA Course Programming Massively Parallel Processors: the CUDA experience.
CuMAPz: A Tool to Analyze Memory Access Patterns in CUDA

CuMAPz: A Tool to Analyze Memory Access Patterns in CUDA
CUDA Performance Considerations Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2011.

CUDA Performance Considerations Patrick Cozzi University of Pennsylvania CIS Spring 2011.
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 1 ECE 498AL Lectures 8: Memory Hardware in G80.

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 1 ECE 498AL Lectures 8: Memory Hardware in G80.
Introduction to CUDA (1 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2012.

Introduction to CUDA (1 of 2) Patrick Cozzi University of Pennsylvania CIS Spring 2012.
Introduction to CUDA 1 of 2 Patrick Cozzi University of Pennsylvania CIS 565 - Fall 2012.

Introduction to CUDA 1 of 2 Patrick Cozzi University of Pennsylvania CIS Fall 2012.
L8: Writing Correct Programs, cont. and Control Flow L8: Control Flow CS6235.

L8: Writing Correct Programs, cont. and Control Flow L8: Control Flow CS6235.
CUDA Performance Considerations (1 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2012.

CUDA Performance Considerations (1 of 2) Patrick Cozzi University of Pennsylvania CIS Spring 2012.
© David Kirk/NVIDIA and Wen-mei W. Hwu ECE408/CS483/ECE498al, University of Illinois, 2007-2012 1 ECE408 Applied Parallel Programming Lecture 12 Parallel.

© David Kirk/NVIDIA and Wen-mei W. Hwu ECE408/CS483/ECE498al, University of Illinois, ECE408 Applied Parallel Programming Lecture 12 Parallel.

Similar presentations

Ads by Google