1. The Problem Finding a needle in haystack An expert (CPU)
A group of non-experts (GPU)

2. Micro-benchmarking GPU micro-architectures
Suhas Thejaswi Muniyappa
Department of Computer Science Aalto University

3. Overview Micro-processor trend Micro-benchmarking
CPU micro-processor trend
GPU micro-processor trend
Micro-benchmarking
Pointer chase
Fine-grain pointer chase
Piecewise linear fine-grain pointer chase
Hardware characteristics

4. CPU micro-processor trend
Hardware support for advanced instructions.
Availability of hardware documentation.
Expensive hardware.
No significant change in per-core performance over the decade.
Parallelize the execution to achieve the speedup.

5. GPU micro-processor trend
Low hardware cost.
High arithmetic and memory bandwidth.
Thousands of cores.
Built for processing graphics.
No hardware support for advanced instructions.
Limited documentation of memory hierarchy.
How to overcome the limitations of GPUs?

6. Micro-benchmarking
Hacking into the system to reveal hardware details.
Using access latency to determine hardware architecture.
Details of memory system is necessary to achieve optimal hardware performance.

7. Pointer chase Saavedra et al. (1996) benchmarking approach for CPUs.
Array element is initialized with index of next memory access.
Access latency depends on the stride size.
Average memory access latency is stored.

8. Fine-grain pointer chase
Record and analyze every memory access latency.
Mei and Chu (2016) designed fine-grain benchmarks for GPUs.
Access latency stored in shared memory.
Shared memory not sufficient for large arrays.

9. Piecewise fine-grain pointer chase
Disk storage
After each iteration shared memory contents are stored into disk.
Sliding window approach to record access latency.

10. Hardware characteristics
L1 cache
Using the access latency the hardware characteristics are deduced.

11. Summary GPUs can be used for general purpose computations.
GPUs provide an environment for executing algorithms which can scale.
Details of memory system is necessary to achieve optimal hardware performance.
Benchmarking reveals characteristics of the hardware, which is not revealed by the hardware manufacturers.

12. References
[1] Mei, X., and Chu, X. Dissecting memory hierarchy through microbenchmarking. IEEE Transaction on Parallel and Distributed Systems Preprint, 99 (2016), 1.
[2] Mei, X., Zhao, C., and Chu, X. Benchmarking the memory hierarchy of modern GPUs. Network and Parallel Computing: 11th IFIPWG International Conference Proceedings (NPC) (2014),
[3] Saavedra, R.H., and Smith, A.J. Measuring cache and TLB performance and their effect on benchmark runtimes. IEEE transactions on computers 44, 10 (1995),
[4] Saavedra, R.H. CPU performance evaluation and execution time prediction using narrow spectrum benchmarking. PhD thesis, university of California, Berkley, 1992.

15. Questions ?

16. Thank you