1. 1

2. GPU – History  First true 3D graphics started with early display controllers (video shifters)  They acted as pass between CPU and display  RCA’s “Pixie” video chip (CDP1861) in 1976 capable of outputting video signal at 62x128 resolution  In 1977 this chip was followed by Television Interface Adapter (TIA) 1A 2

3. GPU – History  TIA was integrated into Atari 2600 for generating the screen display, sound effects and reading the input from the controller 3

4. GPU – History  Basically vertices transformed into pixels  Computing by “hand” and that was very slow  Early 80’s to late 90’s early GPUs work with fixed-function pipeline 4

5. GPU – History 5

6.  Later general programming extended to shader stage  Data independence is also explored  In 2006 NVidia GeForce 8800 mapped separate graphics stage to a unified array of processors(for vertex shading, geometry and pixel processing)  In 2007 NVidia release of CUDA 6

7. GPU  Similar to computer CPU but designed for the purpose of computing very complex mathematical and geometric calculation  In past all this work has been done by CPU which put strain on CPU and degraded performance  GPU improves performance because of its parallel processing architecture which allows it to perform multiple calculation at same time 7

8. GPU  Some of the fastest GPU has much more transistors than average CPU  GPU due to intensive calculation and speed produce a lot of heat so on the motherboard it is usually located under heat sink or fan  GPU typically interface with motherboard using PCI Express bus or accelerated graphic port and can be replaced or upgraded easily  There can be multiple GPUs do draw images simultaneously to the screen increasing the processing power (Google Tango Project) 8

9. GPU vs CPU  Architecturally CPU consist of few cores that can handle multiple threads at the time  GPU consist of hundreds of cores that can handle thousands of threads simultaneously 9

10. GPU vs CPU Discrepancy in floating point capability between CPU and GPU is that GPU is specialized for compute- intensive, highly parallel computation - exactly what graphic rendering is about and so 80% of transistors are devoted for data processing 10

11. GPU vs CPU Same function is executed on each element of data with high arithmetic intensity 11

12. Benefits of using GPU vs CPU  GPU has many benefits such as more computing power, larger memory bandwidth and lower power consumption but regarding its high computing there are some constraints  Developing a code with GPU takes more time and requires highly skilled work  GPU code runs in parallel so data partition and synchronization is needed 12

13. Benefits of using GPU vs CPU  It is hard to answer this question since it is application dependant  Simply GPU is very good following straight line of processing but not so good when processing different processing path  Code should be executed on GPU when it must be executed many times in parallel 13

14. Benefits of using GPU vs CPU  Example we can blend pixels from A to B and put them all in C  This task when executed on CPU would be:  For (int i = 0; i < pixelCount; i++) C[i] = A[i] + B[i]; This code can be slow when many pixels 14

15. Benefits of using GPU vs CPU  Code C[i] = A[i] + B[i]; and then we can populate cores with this code assigning value i for each  This is where GPU is at its best because all cores execute program at same time  Example where GPU is not very fast is conditional branching which implies making copy of the program that follows branch A and populate all cores with this code  Execute until first logical operation  Evaluate all elements and continue processing all elements that follow branch A and enqueue all processes that chose path B  Problem is there is no program for B and now all cores that chose B must be idle 15

16. Benefits of using GPU vs CPU  Possible worst case from prev point?  Only one core executes A branch and all others idle  Once cores executing A are done we could activate branch B version of the program (copying oit from memory buffer to core memory)  Execute B branches and if needed merge results 16

17. Benefits of using GPU vs CPU  GPU is designed for multithreaded calculations  GPU makers can easily add more cores whenever they want to add computational power but the problem is that some problems can not be divided in smaller problems  Lecture point: Not every problem lends itself to parallelism  Ex: n th in Fibonacci series (CPU much faster here) 17

18. Benefits of using GPU vs CPU  GPU can be more efficient for other reasons beside parallel computing  More restrictive memory access  Does not support as many data types  GPUs have limited instruction sets to perform specialized calculations  GPUs are highly optimized for floating point calculations  Integer point calculation is not necessarily faster on GPU 18

19. AMD FirePro™ D-Series GPU  Newest star of GPU in new Mac Pro  3 models D300, D500, D700  Main difference between above is number of stream processors, VRAM, width of memory bus, memory bandwidth and teraflop performance  More processing power for video editing, 3D modeling and animation and photography  GPU computing using OpenCL (more on it later) 19

20. AMD FirePro™ D-Series GPU  Architecture of this particular GPU supports OpenCL 2.0 and lower D300 model supports 256 bit memory bus that delivers 160 GB per second memory bandwidth meaning large amounts of data can be read quickly  With support of OpenCL 2.0 it is now possible for application to run both on GPU and CPU simultaneously and AMD refers to this as Accelerated Parallel Programming 20

21. AMD FirePro™ D-Series GPU 21

22. GPU Accelerated Computing  It is basically use of GPU together with CPU to accelerate scientific, analytics, consumer and enterprise applications  Started in 2007 by NVIDIA  GPUs are currently accelerating applications in platforms ranging from cars, mobile phones, risk management etc.  GPU Accelerated computing offers better performance by offloading compute – intensive portions of applications to GPU while remainder of the code still runs on CPU 22

23. GPU Accelerated Computing  We can see from the image on the left that some part of the code runs on GPU and some part runs on GPU and from users perspective applications simply runs faster 23

24. OpenCL  Open Computing Language  From the makers of OpenGL  Wide industry support: AMD, Apple, NVidia, Samsung etc.  OpenCL model : 24

25. OpenCL Architecture Host controls multiple compute devices 25

26. OpenCL Architecture  Each of these compute devices consist of multiple compute units  Compute units (execution units and arithmetic's processing units) contain processing elements  Processing elements execute OpenCL kernels (these are just a functions written by programmer in OpenCL language (C with restrictions and special keywords and data types)  Kernels are basic unit of executable code  Program is collection of kernels and other functions 26

27. OpenCL Architecture  We should also be aware that OpenCL program is divided in two parts  One part that executes on device (GPU)  Second part that executes on host (CPU)  Device part is where we need to write special functions called kernels 27

28. OpenCL Architecture – Device  Device is GPU  Kernel is written which is function executed on GPU (not only one)  Kernels are entry points into device program (only functions that can be called from host)  We need to program kernels ourselves 28

29. How to program a kernel - SIMT  SIMT: Single instruction multiple thread which reflects how instructions are executed on device  Same code is executed in parallel by a different thread and each thread executes the code with different data 29

30. How to program a kernel – Work Item  Work items are equivalent to threads and are smallest execution entity  Every time kernel is launched, lots of work items (a number specified by programmer) are launched and each one is executing same code  Each work item has an ID which is accessible from the kernel and it is used to distinguish the data to be processed by each work item 30

31. How to program a kernel – Work Group  Work groups are there to allow communication and cooperation between work items  They also reflect how work items are organized  N dimensional grid of work groups (N = 1,2 or 3)  Work groups also have ID which can be called from kernel 31

32. How to program a kernel – ND Range  ND Range is next organizational level specifying how work groups are organized  N dimensional grid of work groups where N = 1,2 or 3 32

33. Kernel Example - CPU void vector_add_cpu (const float* src_a, const float* src_b, float* res, const int num) { for (int i = 0; i < num; i++) res[i] = src_a[i] + src_b[i]; } This is kernel that adds 2 vectors. Here basically we have one thread iterating through all elements. 33

34. Kernel Example - GPU __kernel void vector_add_gpu (__global const float* src_a, __global const float* src_b, __global float* res, const int num) { /* get_global_id(0) returns the ID of the thread in execution. As many threads are launched at the same time, executing the same kernel, each one will receive a different ID, and consequently perform a different computation.*/ const int idx = get_global_id(0); /* Now each work-item asks itself: "is my ID inside the vector's range?" If the answer is YES, the work-item performs the corresponding computation*/ if (idx < num) res[idx] = src_a[idx] + src_b[idx]; } 34

35. Kernel Example - GPU  Each thread computing one elements  “kernel” reserved word which specifies that the function is kernel  Kernel functions always return void  In similar ways we can program host device as well 35

36. Kernel Example - GPU 36

37. Parallel Processing and OpenCL  OpenCL data parallel programming model is very hierarchical which can be specified in two ways  Explicitly – programmer defines total number of items to execute in parallel as well as how to group them  Implicitly - programmer defines total number of items to execute in parallel and OpenCL manages grouping them 37

38. OpenCL and Synchronization  The two domains of synchronization in OpenCL are work items in single work group and command queue in a single context  Work group barriers enable synchronization or work items in work group – barrier()  Barrier and memory fences synchronize threads in a work group  All threads are required to reach barrier before any of them can continue 38

39. OpenCL and Synchronization 39

40. OpenCL and Synchronization  Built in functions mem_fence() and barrier()  mem_fence(CLK_LOCAL_MEM_FENCE and/or CLK_GLOBAL_MEM_FENCE)  waits until all reads/writes to local and/or global memory made by the calling work item prior to mem_fence () are visible to all threads in the work group  barrier(CLK_LOCAL_MEM_FENCE and/or CLK_GLOBAL_MEM_FENCE)  waits until all work items in the work group have reached this point and calls mem_fence (CLK_LOCAL_MEM_FENCE and/or CLK_GLOBAL_MEM_FENCE 40

41. OpenCL and Synchronization  Two types of synchronization between commands in command queue  Command Queue barrier – enforces ordering with single queue and any resulting changes in memory are available to next command in the queue  Events – enforces ordering between or within queues  Enqueued commands in OpenCL return event identifying command as well as memory object updated by it 41

42. OpenCL – Memory Model  OpenCL had 4 address space  private – specific to work item and not visible to other work items  local – specific to work group and accessible only to work items belonging to that work group  global – accessible to all work items executing in context as well as to the host  constant – read only region for host allocated objects that are not changed during kernel execution 42

43. OpenCL – Memory Model  There is also host accessible region for application data structure and program data  Pci memory part of host (CPU) memory accessible from and modifiable by host program and GPU device  Modifying this memory requires synchronization between GPU compute device and the CPU 43

44. OpenCL – Communication  Communication and data transfer between host and GPU occur on PCIe channel  Actual transfer performance is CPU dependant  Transfer from the host to the GPU are done by the command processor  GPU device can read and write system memory directly through kernel instructions over PCIe bus 44

45. OpenCL – Processing API Calls  Host application does not interact with GPU device directly (data structures for the host)  Driver layer translates and issues commands to the hardware  Most commands to the GPU are buffered in command queue on the host side  Queue of commands is sent to and processed by the GPU  There is no guarantee as to when commands from command queue are executed but only that they are executed in order 45

46. OpenCL – Scheduling  GPU devices are very efficient in parallelizing large numbers of work items in manner transparent to application  Each GPU device uses large number of wavefronts to hide memory access latencies by having scheduler switch the active wavefront in given compute unit whenever the current wavefront is waiting for a memory access to complete 46

47. OpenCL – Scheduling 47

48. OpenCL – Scheduling 48

49. Data Parallelism in OpenCL  Define N dimensional computation domain (N = 1, 2 or 3)  Each independent element of execution in ND domain is called a work item  The ND domain defines the total number of work items that execute in parallel  E.g., process a 1024 x 1024 image: Global problem dimensions:  1024 x 1024 = 1 kernel execution per pixel: 1,048,576 total executions 49

50. Data Parallelism in OpenCL 50

51. Data Parallelism in OpenCL  Kernels executed across a global domain of work items  Global dimensions define the range of computation one work item per computation, executed in parallel  Work items are grouped in local workgroups  Local dimensions define the size of the workgroups  Executed together on one device and share local memory and synchronization 51

52. OpenCL C (quick glance)  Derived from ISO C99 (with some restrictions)  Language Features Added (Work items and work groups, vector types and synchronization  Included large set of built in functions for image manipulation, work item manipulation and math functions 52

53. OpenCL C language restriction  Pointers to functions are not allowed  Pointers to pointers allowed within a kernel, but not as an argument  Variable length arrays and structures are not supported  Recursion is not supported  3D Image writes are not supported 53

54. OpenCL C optional extension  Extensions are optional features exposed through OpenCL  The OpenCL working group has already approved many extensions to the OpenCL specification such as double precision floating point types, built in functions to support doubles, byte addressable stores (write to pointers to types < 32 bits) 54

55. Work Items and Work Groups 55

56. Work Items and Work Groups 56

57. OpenCL Data Types  Scalar data types (bool, char, cl_char, unsigned char, uchar, cl_uchar, short, cl_short, unsigned short, etc.)  Image types (image2d_t, image3d_t, image2d_array_t, image1d_t, etc.)  Vector data types (charn, ucharn, shortn, ushortn, intn, uintn etc.)  Supported values of n are 2, 3, 4, 8, and 16 for all vector data types 57

58. Q & A 58