Skew Handling in Aggregate Streaming Queries on GPUs Georgios Koutsoumpakis 1, Iakovos Koutsoumpakis 1 and Anastasios Gounaris 2 1 Uppsala University, Sweden 2 Aristotle University of Thessaloniki, Greece

Talk Outline 1.Setting of our work 2.Our load-balancing framework 3.Load balancing techniques 4.Experimental results 5.Conclusions and future work Anastasios Gounaris

Target applications Data-intensive continuous aggregate queries –E.g., continuously report the average share price of each company in all European stock markets. –They form the basis of many online analysis tasks. –They implicitly assume a (possibly infinite) data stream Anastasios Gounaris

Scalability requirements CQs may be CPU-intensive due to the –Sheer amount of data –Possibly complex aggregate tasks CQs may also be memory-intensive. –E.g., continuously report the median share price of each company in all European stock markets in the last secs. –We need to keep all the values within a (sliding) window of appropriate size. The standard solution is parallelism. –Partitioned parallelism has been widely investigated and used for CQs Anastasios Gounaris

Imbalance problems In partitioned parallelism each group is allocated to a distinct processor unit (PU). If the workload is predictable, we can allocate equal amount of work to each PU. But often, it is not! –E.g., continuously report the median size of messages originated from each IP taking into account the last messages. Skew problems arise when groups incur different amounts of workload Anastasios Gounaris

Our Goal Parallelise CQs on GPUs using CUDA. Balance the load on-the-fly. –Revise the assignment of groups to PUs Anastasios Gounaris

A brief note on CUDA CUDA stands for “Compute Unified Device Architecture” It is a general purpose programming model that makes it easy batches of threads to run on the GPU. The GPU acts as a dedicated super-threaded, massively data parallel co- processor Serial Code (host)‏... Parallel Kernel (device)‏ KernelA >>(args); Serial Code (host)‏ Parallel Kernel (device)‏ KernelB >>(args); The material of this slide is from David Kirk/NVIDIA and Wen-mei W. Hwu Anastasios Gounaris

Talk Outline 1.Setting of our work 2.Our load-balancing framework 3.Load balancing techniques 4.Experimental results 5.Conclusions and future work Anastasios Gounaris

Main rationale Data arrive continuously and we buffer them in batches, –which are processed in iterations. CPU responsibilities: To prepare the data in order to achieve coalesced memory access. To detect and correct imbalances. GPU responsibilities: To perform the actual data processing Anastasios Gounaris

Mappings on the CPU We assume a fixed number of threads. –Each group is fully processed by a single GPU thread. We keep 2 hashmaps for group-to-thread and thread-to- group mappings: GroupThread …… Group 01, ,5 3… 4… …… Anastasios Gounaris

id:5, attr: 1 id:2, attr: 4 id:3, attr: 1 id:1, attr: 5 id:2, attr: 2 id:6, attr:1 … Data Stream id:3, attr: 1 id:1, attr: 5 id:2, attr: 4 id:2, attr: 2 id:5, attr: 1 id:6, attr:1 thread0thread1thread2 1. Copies the next batch of the streaming data to a new matrix 2. Counts the number of tuples of each thread id:5, attr: 1 id:2, attr: 4 id:3, attr: 1 id:1, attr: 5 id:2, attr: 2 id:6, attr:1 1. Reorders data so that groups of the same thread are together 2.creates matrix threadDataIndicator Reordered data matrix Data matrix 024 threadDataIndicator repeat Operations on the CPU Check/correct imbalances Copy data to GPU /launch the kernel Anastasios Gounaris

Data on the GPU Copied from the CPU Maintained on the GPU id:3, attr: 1 id:1, attr: 5 id:2, attr: 4 id:2, attr: 2 id:5, attr: 1 id:6, attr:1 thread0thread1thread2 Reordered data matrix 024 threadDataIndicator Windows Group nextPos Anastasios Gounaris

Talk Outline 1.Setting of our work 2.Our load-balancing framework 3.Load balancing techniques 4.Experimental results 5.Conclusions and future work Anastasios Gounaris

id:5, attr: 1 id:2, attr: 4 id:3, attr: 1 id:1, attr: 5 id:2, attr: 2 id:6, attr:1 … Data Stream id:3, attr: 1 id:1, attr: 5 id:2, attr: 4 id:2, attr: 2 id:5, attr: 1 id:6, attr:1 thread0thread1thread2 1. Copies the next batch of the streaming data to a new matrix 2. Counts the number of tuples of each thread id:5, attr: 1 id:2, attr: 4 id:3, attr: 1 id:1, attr: 5 id:2, attr: 2 id:6, attr:1 1. Reorders data so that groups of the same thread are together 2.creates matrix threadDataIndicator Reordered data matrix Data matrix 024 threadDataIndicator repeat Operations on the CPU Check/correct imbalances Copy data to GPU/ launch the kernel Anastasios Gounaris

Load balancing algorithms - 1 Try to smooth differences between the workload of threads We use two heaps in order to detect tmax and tmin in O(1) Anastasios Gounaris

Load balancing algorithms - 2 getFirst simply chooses the first group upon detection of the most imbalanced pair. checkAll examines all the groups of the most loaded threaded and moves the biggest one. probCheck makes a probabilistic choice of the biggest group in the most loaded threaded. bestBalance examines all the groups of the most loaded threaded and moves the one that leads to the smallest difference in the workload. shift allows moves of groups only to neighboring threads. –E.g., the first group of thread 14 can be moved only to thread 13. shiftLocal does not detect tmax/tmin and checks only adjacent threads Anastasios Gounaris

Experimental setting Two systems used. –PC1: Intel Core2 Duo E6750 CPU at 2.66GHz NVidia 460GTX (GF104) graphics processor at 810 Mhz on a PCIe v2.0 x16 slot (5GB/s transfer rate). –PC2: Intel P4 550 CPU at 3.4 GHz NVidia 550GTX Ti (GF116) at 910 MHz on a PCIe v1.1 x16 (2.5GB/s transfer rate) slot. Three datasets. –DS1: no imbalance –DS2: high imbalance, group sizes follow a zipf distribution –DS3: low imbalance, group sizes follow a zipf distribution but groups are randomly permuted Fixed parameters: –Block size is fixed to 256 threads. –Batch size is fixed to 50K tuples. –Window size is 100 and there are always 40K groups Anastasios Gounaris

Impact of imbalance PC1 w/o load balancing – time to process 100M tuples (2K iterations) Grid size = 4 Anastasios Gounaris

High Imbalance Speedups of up to 4.27 are observed. Increasing the grid size seems to work …but it is not always applicable! Simple heuristics perform similarly to (if not better than) the most sophisticated ones. Less sophisticated and approximate load balancing techniques are more appropriate for GPGPU –Basically because they require less computational effort for the balancing itself Grid size = 4Grid size = 64 Anastasios Gounaris

Low imbalance No technique is actually effective Grid size = 4Grid size = 64 Anastasios Gounaris

Talk Outline 1.Setting of our work 2.Our load-balancing framework 3.Load balancing techniques 4.Experimental results 5.Conclusions and future work Anastasios Gounaris

Summary In this work we presented: 1.A GPGPU load balancing framework. 2.Load balancing algorithms. Lessons learnt: –Load imbalances can lead to serious performance degradations. –In high imbalances, we have achieved speedups of more than 4 times. –Load balancing techniques need not be very sophisticated. –Small imbalances cannot be tackled Anastasios Gounaris

Future Work - Points not considered Varying dynamically the grid/block/batch size. Investigation in light of the most recent dynamic parallelism extensions in Kepler architectures. Handling of cases where the gpu capacity is lower than the data arrival rate –Use of approximate/load shedding techniques Anastasios Gounaris

Thank you! … and apologies to all reviewers, whose comments have not been addressed due to tight time contraints Anastasios Gounaris

Back-up slides - Overheads For grid size 4, the CPU operations are (almost) fully hidden Grid size = 4Grid size = 64 Anastasios Gounaris

Anastasios Gounaris