Shekoofeh Azizi Spring

 CUDA is a parallel computing platform and programming model invented by NVIDIA  With CUDA, you can send C, C++ and Fortran code straight to GPU, no assembly language required. 2

 Development Environment  Introduction to CUDA C  CUDA programming model  Kernel call  Passing parameter  Parallel Programming in CUDA C  Example : summing vectors  Limitations  Hierarchy of blocks and threads  Shared memory and synchronizations  CUDA memory model  Example : dot product 3

 The prerequisites to developing code in CUDA C :  CUDA-enabled graphics processor  NVIDIA device driver  CUDA development toolkit  Standard C compiler 4

 Every NVIDIA GPU since the 2006 has been CUDA-enabled.  Frequently Asked Questions  How can I find out which GPU is in my computer?  Do I have a CUDA-enabled GPU in my computer? 5

 Control Panel → "NVIDIA Control Panel“ or "NVIDIA Display“ 6

 Complete list on 7

 System software that allows your programs to communicate with the CUDA-enabled hardware  Due to graphics card and OS can find on :    CUDA-enabled GPU + NVIDIA’s device driver = Run compiled CUDA C code. 8

 Two different processors  CPU  GPU  Need two compilers  One compiler will compile code for your CPU.  One compiler will compile code for your GPU  NVIDIA provides the compiler for your GPU code on:   Standard C compiler : Microsoft Visual Studio C compiler 9

 Development Environment  Introduction to CUDA C  CUDA programming model  Kernel call  Passing parameter  Parallel Programming in CUDA C  Example : summing vectors  Limitations  Hierarchy of blocks and threads  Shared memory and synchronizations  CUDA memory model  Example : dot product 10

 Host : CPU and System’s memory  Device : GPU and its memory  Kernel : Function that executes on device  Parallel threads in SIMT architecture 11

12

 An empty function named kernel() qualified with __global__  A call to the empty function, embellished with >> 13

 __global__  CUDA C needed a linguistic method for marking a function as device code  It is shorthand to send host code to one compiler and device code to another compiler.  >>  Denote arguments we plan to pass to the runtime system  These are not arguments to the device code  These will influence how the runtime will launch our device code 14

15

 Allocate the memory on the device → cudaMalloc()  A pointer to the pointer you want to hold the address of the newly allocated memory  Size of the allocation you want to make  Access memory on a device → cudaMemcpy()  cudaMemcpyHostToDevice  cudaMemcpyDeviceToHost  cudaMemcpyDeviceToDevice  Release memory we’ve allocated with cudaMalloc()→ cudaFree() 16

 Restrictions on the usage of device pointer:  You can pass pointers allocated with cudaMalloc() to functions that execute on the device.  You can use pointers allocated with cudaMalloc() to read or write memory from code that executes on the device.  You can pass pointers allocated with cudaMalloc() to functions that execute on the host.  You cannot use pointers allocated with cudaMalloc() to read or write memory from code that executes on the host. 17

 Development Environment  Introduction to CUDA C  CUDA programming model  Kernel call  Passing parameter  Parallel Programming in CUDA C  Example : summing vectors  Limitations  Hierarchy of blocks and threads  Shared memory and synchronizations  CUDA memory model  Example : dot product 18

 Example : Summing vectors 19

20

21 GPU Code : add >>

 Allocate 3 array on device → cudaMalloc()  Copy the input data to the device → cudaMemcpy()  Execute device code → add >> (dev_a, dev_b, dev_c)  first parameter: number of parallel blocks  second parameter: the number of threads per block  N blocks x 1 thread/block = N parallel threads  Parallel copies→ blocks 22

23

24

 Change the index computation within the kernel  Change the kernel launch 25

26

27 Grid : The collection of parallel blocks Blocks Threads

 Development Environment  Introduction to CUDA C  CUDA programming model  Kernel call  Passing parameter  Parallel Programming in CUDA C  Example : summing vectors  Limitations  Hierarchy of blocks and threads  Shared memory and synchronizations  CUDA memory model  Example : dot product 28

29

 Per block  registers  shared memory  Per thread  local memory  Per grid  Global memory  Constant memory  Texture memory 30

 __shared__  The CUDA C compiler treats variables in shared memory differently than typical variables.  Creates a copy of the variable for each block that you launch on the GPU.  Every thread in that block shares the memory  Threads cannot see or modify the copy of this variable that is seen within other blocks  Threads within a block can communicate and collaborate on computations 31

32

 The computation consists of two steps:  First, we multiply corresponding elements of the two input vectors  Second, we sum them all to produce a single scalar output.  Dot product of two four-element vectors 33

34

 Buffer of shared memory: cache→ store each thread’s running sum  Each thread in the block has a place to store its temporary result.  Need to sum all the temporary values we’ve placed in the cache.  Need some of the threads to read the values from this cache.  Need a method to guarantee that all of these writes to the shared array cache[] complete before anyone tries to read from this buffer.  When the first thread executes the first instruction after __syncthreads(), every other thread in the block has also finished executing up to the __syncthreads(). 35

 Reduction: the general process of taking an input array and performing some computations that produce a smaller array of results a.  having one thread iterate over the shared memory and calculate a running sum and take time proportional to the length of the array  do this reduction in parallel and take time that is proportional to the logarithm of the length of the array 36

 Parallel reduction:  Each thread will add two of the values in cache and store the result back to cache.  Using 256 threads per block, takes 8 iterations of this process to reduce the 256 entries in cache to a single sum.  Before read the values just stored in cache, need to ensure that every thread that needs to write to cache has already done. 37

38

1.Allocate host and device memory for input and output arrays 2.Fill input array a[] and b[] 3.Copy input arrays to device using cudaMemcpy() 4.Call dot product kernel using some predetermined number of threads per block and blocks per grid 39

Thanks Any question? 40

41