1. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 1 ECE 498AL Lecture 13: Application Lessons When the tires hit the road… (Part 2)

2. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 2 Build Scan From Partial Sums T347045611 T347 4560 T30744 616 Iterate log(n) times. Each thread adds value stride elements away to its own value, and sets the value stride elements away to its own previous value. Iteration 2 2 threads Stride 4 Stride 2 Each corresponds to a single thread.

3. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 3 Build Scan From Partial Sums T347045611 T347 4560 T30744 616 T03411 151622 Done! We now have a completed scan that we can write out to device memory. Total steps: 2 * log(n). Total work: 2 * (n-1) adds = O(n) Work Efficient! Iteration log(n) n/2 threads Stride 2 Stride 4 Stride 1 Each corresponds to a single thread.

4. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 4 Bank Conflicts Current scan implementation has many shared memory bank conflicts –These really hurt performance on hardware We can add some simple modifications to our memory addressing to save a lot of cycles

5. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 5 Bank Conflicts Occur when multiple threads access the same shared memory bank with different addresses No penalty if all threads access different banks –Or if all threads access exact same address Access costs 2*N cycles if there is a conflict –Where N is max number of threads accessing single bank

6. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 6 Build the Sum Tree T31704163 T34774569 Stride 1 Iteration 1, n/2 threads Iterate log(n) times. Each thread adds value stride elements away to its own value Each corresponds to a single thread.

7. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 7 Initial Bank Conflicts on Load Each thread loads two shared mem data elements Tempting to interleave the loads temp[2*thid] = g_idata[2*thid]; temp[2*thid+1] = g_idata[2*thid+1]; Threads:(0,1,2,…,8,9,10,…)  banks:(0,2,4,…,0,2,4,…) Better to load one element from each half of the array temp[thid] = g_idata[thid]; temp[thid + (n/2)] = g_idata[thid + (n/2)];

8. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 8 Bank Conflicts in the tree algorithm When we build the sums, each thread reads two shared memory locations and writes one: Th(0,8) access bank 0 0123456789101112131415012... 3170416358203319457… 0123456789101112131415012... 347745695132236110497… Bank: First iteration: 2 threads access each of 8 banks. … Each corresponds to a single thread. Like-colored arrows represent simultaneous memory accesses

9. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 9 When we build the sums, each thread reads two shared memory locations and writes one: Th(1,9) access bank 2, etc. Bank Conflicts in the tree algorithm 0123456789101112131415012... 3170416358203319457… 0123456789101112131415012... 347745695132236110497… Bank: First iteration: 2 threads access each of 8 banks. … Each corresponds to a single thread. Like-colored arrows represent simultaneous memory accesses

10. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 10 2 nd iteration: even worse! –4-way bank conflicts; for example: Th(0,4,8,12) access bank 1, Th(1,5,9,13) access Bank 5, etc. Bank Conflicts in the tree algorithm 0123456789101112131415012... 347445695132236110497… 0123456789 1112131415012... 347114561451321536116497… Bank: 2 nd iteration: 4 threads access each of 4 banks. … Each corresponds to a single thread. Like-colored arrows represent simultaneous memory accesses

11. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 11 Bank Conflicts in the tree algorithm We can use padding to prevent bank conflicts –Just add a word of padding every 16 words: No more conflicts! 01234567891011121314150123… 3170416358203319P457… 01234567891011121314150123... 347745695132236110P497… Bank: Now, within a 16-thread half-warp, all threads access different banks. (Note that only arrows with the same color happen simultaneously.) …

12. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 12 Use Padding to Reduce Conflicts This is a simple modification to the last exercise After you compute a shared mem address like this: Address = 2 * stride * thid; Add padding like this: Address += (address >> 4); // divide by NUM_BANKS This removes most bank conflicts –Not all, in the case of deep trees

13. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 13 Scan Bank Conflicts (1) A full binary tree with 64 leaf nodes: Multiple 2-and 4-way bank conflicts Shared memory cost for whole tree –1 32-thread warp = 6 cycles per thread w/o conflicts Counting 2 shared mem reads and one write (s[a] += s[b]) –6 * (2+4+4+4+2+1) = 102 cycles –36 cycles if there were no bank conflicts (6 * 6)

14. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 14 Scan Bank Conflicts (2) It’s much worse with bigger trees! A full binary tree with 128 leaf nodes –Only the last 6 iterations shown (root and 5 levels below) Cost for whole tree: –12*2 + 6*(4+8+8+4+2+1) = 186 cycles –48 cycles if there were no bank conflicts! 12*1 + (6*6)

15. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 15 Scan Bank Conflicts (3) A full binary tree with 512 leaf nodes –Only the last 6 iterations shown (root and 5 levels below) Cost for whole tree: –48*2+24*4+12*8+6* (16+16+8+4+2+1) = 570 cycles –120 cycles if there were no bank conflicts!

16. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 16 Fixing Scan Bank Conflicts Insert padding every NUM_BANKS elements const int LOG_NUM_BANKS = 4; // 16 banks on G80 int tid = threadIdx.x; int s = 1; // Traversal from leaves up to root for (d = n>>1; d > 0; d >>= 1) { if (thid <= d) { int a = s*(2*tid); int b = s*(2*tid+1) a += (a >> LOG_NUM_BANKS); // insert pad word b += (b >> LOG_NUM_BANKS); // insert pad word shared[a] += shared[b]; }

17. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 17 Fixing Scan Bank Conflicts A full binary tree with 64 leaf nodes No more bank conflicts! –However, there are ~8 cycles overhead for addressing For each s[a] += s[b] (8 cycles/iter. * 6 iter. = 48 extra cycles) –So just barely worth the overhead on a small tree 84 cycles vs. 102 with conflicts vs. 36 optimal

18. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 18 Fixing Scan Bank Conflicts A full binary tree with 128 leaf nodes –Only the last 6 iterations shown (root and 5 levels below) No more bank conflicts! –Significant performance win: 106 cycles vs. 186 with bank conflicts vs. 48 optimal

19. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 19 Fixing Scan Bank Conflicts A full binary tree with 512 leaf nodes –Only the last 6 iterations shown (root and 5 levels below) Wait, we still have bank conflicts –Method is not foolproof, but still much improved –304 cycles vs. 570 with bank conflicts vs. 120 optimal

20. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 20 Fixing Scan Bank Conflicts It’s possible to remove all bank conflicts –Just do multi-level padding –Example: two-level padding: const int LOG_NUM_BANKS = 4; // 16 banks on G80 int tid = threadIdx.x; int s = 1; // Traversal from leaves up to root for (d = n>>1; d > 0; d >>= 1) { if (thid <= d) { int a = s*(2*tid); int b = s*(2*tid+1) int offset = (a >> LOG_NUM_BANKS); // first level a += offset + (offset >>LOG_NUM_BANKS); // second level offset = (b >> LOG_NUM_BANKS); // first level b += offset + (offset >>LOG_NUM_BANKS); // second level temp[a] += temp[b]; }

21. © David Kirk/NVIDIA and Wen-mei W. Hwu, 2007 ECE 498AL, University of Illinois, Urbana-Champaign 21 Fixing Scan Bank Conflicts A full binary tree with 512 leaf nodes –Only the last 6 iterations shown (root and 5 levels below) –No bank conflicts But an extra cycle overhead per address calculation Not worth it: 440 cycles vs. 304 with 1-level padding –With 1-level padding, bank conflicts only occur in warp 0 Very small remaining cost due to bank conflicts Removing them hurts all other warps