CS179: GPU Programming Lecture 5: Memory

Today GPU Memory Overview CUDA Memory Syntax Tips and tricks for memory handling

Memory Overview Very slow access: Between host and device Slow access: Global Memory Fast access: Shared memory, constant memory, texture memory, local memory Very fast access: Register memory

Global Memory Read/write Shared between blocks and grids Same across multiple kernel executions Very slow to access No caching!

Constant Memory Read-only in device Cached in multiprocessor Fairly quick Cache can broadcast to all active threads

Texture Memory Read-only in device 2D cached -- quick access Filtering methods available

Shared Memory Read/write per block Memory is shared within block Generally quick Has bad worst-cases

Local Memory Read/write per thread Not too fast (stored independent of chip) Each thread can only see its own local memory Indexable (can do arrays)

Register Memory Read/write per thread function Extremely fast Each thread can only see its own register memory Not indexable (cant do arrays)

Syntax: Register Memory Default memory type Declare as normal -- no special syntax int var = 1; Only accessible by current thread

Syntax: Local Memory Global variables for threads Can modify across local functions for a thread Declare with __device__ __local__ keyword __device__ __local__ int var = 1; Can also just use __local__

Syntax: Shared Memory Shared across threads in block, not across blocks Cannot use pointers, but can use array syntax for arrays Declare with __device__ __shared__ keyword __device__ __shared__ int var[]; Can also just use __shared__ Dont need to declare size for arrays

Syntax: Global Memory Created with cudaMalloc Can pass pointers between host and kernel Transfer is slow! Declare with __device__keyword __device__ int var = 1;

Syntax: Constant Memory Declare with __device__ __constant__ keyword __device__ __constant__ int var = 1; Can also just use __constant__ Set using cudaMemcpyToSymbol (or cudaMemcpy) cudaMemcpyToSymbol(var, src, count);

Syntax: Texture Memory To be discussed later…

Memory Issues Each multiprocessor has set amount of memory Limits amount of blocks we can have (# of blocks) x (memory used per block) <= total memory Either get lots of blocks using little memory, or fewer blocks using lots of memory

Memory Issues Register memory is limited! Similar to shared memory in blocks Can have many threads using fewer registers, or few threads using many registers Former is better, more parallelism

Memory Issues Global accesses: slow! Can be sped up when memory is contiguous Memory coalescing: making memory contiguous Coalesced accesses are: Contiguous accesses In-order accesses Aligned accesses

Memory Coalescing: Aligned Accesses Threads read 4, 8, or 16 bytes at a time from global memory Accesses must be aligned in memory! Good: Bad: Which is worse, reading 16 bytes from 0xABCD0 or 0xABCDE? 0x00 0x040x14 0x00 0x070x14

Memory Coalescing Aligned Accesses Also bad: beginning unaligned

Memory Coalescing: Aligned Accesses Built-in types force alignment float3 (12B) takes up the same space as float4 (16B) float3 arrays are not aligned! To align a struct, use __align__(x) // x = 4, 8, 16 cudaMalloc aligns the start of each block automatically cudaMalloc2D aligns the start of each row for 2D arrays

Memory Coalescing: Contiguous Accesses Contiguous = memory is together Example: non-contiguous memory Thread 3 and 4 swapped accesses!

Memory Coalescing: Contiguous Accesses Which is better? index = threadIdx.x + blockDim.x * (blockIdx.x + gridDim.x * blockIdx.y); index = threadIdx.x + blockDim.y * (blockIdx.y + gridDim.y * blockIdx.x);

Memory Coalescing: Contiguous Accesses Case 1: Contiguous accesses

Memory Coalescing: Contiguous Accesses Case 1: Contiguous accesses

Memory Coalescing: In-order Accesses In-order accesses Do not skip addresses Access addresses in order in memory Bad example: Left: address 140 skipped Right: lots of skipped addresses

Memory Coalescing Good example:

Memory Coalescing Not as much of an issue in new hardware Many restrictions relaxed -- e.g., do not need to have sequential access However, memory coalescing and alignment still good practice!

Memory Issues Shared memory: Also can be limiting Broken up into banks Optimal when entire warp is reading shared memory together Banks: Each bank services only one thread at a time Bank conflict: when two threads try to access same block Causes slowdowns in program!

Bank Conflicts Bad: Many threads trying to access the same bank

Bank Conflicts Good: Few to no bank conflicts

Bank Conflicts Banks service 32-bit words at a time at addresses mod 64 Bank 0 services 0x00, 0x40, 0x80, etc., bank 1 services 0x04, 0x44, 0x84, etc. Want to avoid multiple thread access to same bank Keep data spread out Split data that is larger than 4 bytes into multiple accesses Be careful of data elements with even stride

Broadcasting Fast distribution of data to threads Happens when entire warp tries to access same address Memory will get broadcasted to all threads in one read

Summary Best memory management: Balances memory optimization with parallelism Break problem up into coalesced chunks Process data in shared memory, then copy back to global Remember to avoid bank conflicts!

Next Time Texture memory CUDA Applications in graphics