Parallel Processing and GPUs with Daniel L. Silver, Ph.D. Christian Frey, BBA April 11-12, 2017

Benefits of GPUs Hundreds, or thousands of cores to allow for parallel processing of data, such as computing the forward pass through the network for each example in parallel. Memory on a GPU is faster and closer to the GPU cores, further improving processing speed. GPUs are designed specifically for vector addition or multiplication, which comes up often in deep learning

Drawbacks of GPU’s Moving data between the CPU and GPU is very costly, so unnecessary movement slows down the processing. Difficult to add additional memory to a GPU if the network size exceeds GPU memory. CPU’s can use up to 1TB of RAM, whereas GPU’s max out around 24-32 GB. Each core is less powerful than a CPU core, so operations than cannot be parallelized are slower than on a CPU.

Field Programmable Gate Array (FPGA) An integrated circuit that can be programmed in the field to handle a specific function. Tend to be faster than GPU for specific tasks, but are less versatile Can be used with embedded microprocessors to form a complete system More power efficient than GPUs due to their reduced ability at general purpose computing.

Application Specific Integrated Circuit (ASIC) An integrated circuit designed for a specific purpose, such as multiplying matrices. Can get away with lower precision (even 8 bit) when calculating weight updates or outputs of nodes for faster training. Google has created the TPU (Tensor Processing Unit), which powered AlphaGo in its competition against Lee Sedol Faster and uses less power than Traditional GPUs, but can cost millions of dollars to create the circuitry. Only useful for larger scale operations (such as Google).