1. Measuring Performance
These notes introduce:
Timing Program Execution
How to measure time of execution of CUDA programs
CUDA “events”
Synchronous and asynchronous CUDA routines
Bandwidth measures
Computation measures – floating point operations/sec
ITCS 4/5145 GPU Programming, B. Wilkinson, Nov 12, CUDATiming.ppt

2. Ways to measure time of execution
Generally instrument code. Measure time at two places and get difference
Routines to use to measure time:
C clock() or time() routines
CUDA “events” (seems the best way)
CUDA SDK timer

3. Timing with clock()
If program uses cudaMemcpy, which is synchronous and waits for previous operations to complete and returns when it is complete, could use clock():
#include <time.h> // needed for clock()
int main() {
clock_t start, stop; // return types are clock_t, int’s …
start = clock(); // number of clock ticks since prog launched
cudaMemcpy … ;
mykernel<<<B,T>>>(); // kernel call
cudaMemcpy … ;
stop = clock();
printf(“Execution time is %f seconds\n",
(float) (stop-start)/(CLOCKS_PER_SEC) ;
return 0; }

4. If just measuring time of asynchronous kernel with clock()
Important to remember that kernel calls asynchronous and return immediately and before kernels have fully executed.
Hence need to wait for kernel to complete.
Can be achieved using cudaThreadSynchronize():
start = clock();
mykernel<<<B,T>>>(); // kernel call
cudaThreadSynchronize();
stop = clock();
(We will discuss synchronization within a computation later.)

5. CUDA event timer In general, better to use CUDA event timer.
First need to create event objects.
cudaEvent_t event1;
cudaEventCreate(&event1);
cudaEvent_t event2;
cudaEventCreate(&event2);
creates two “event” objects, event1 and event2.

6. Recording Events
cudaEventRecord(event1, 0) record an “event” into default “stream” (0).
Device will record a timestamp for the event when it reaches that event in the stream, that is, after all preceding operations have completed.
(Default stream 0 will mean completed in CUDA context)
NOTE: This operation is asynchronous and may return before recording event!

7. Making event actually recorded
cudaEventSynchronize(event1) -- waits until named event actually recorded.
Event recorded when all work done by threads to complete prior to specified event
(Not strictly be necessary if synchronous CUDA call in code.)

8. Measuring time between two events
cudaEventElapsedTime(&time, event1, event2) will return (pointer argument) the time elapsed between two events, in milliseconds.
Resolution approx ½ millisecond.
Timing measured using GPU clock.

9. Timing GPU Execution with CUDA events
Code
cudaEvent_t start, stop;
float elapsedTime;
cudaEventCreate(&start); // create event objects
cudaEventCreate(&stop);
cudaEventRecord(start, 0); // Record start event .
cudaEventRecord(stop, 0); // record end event
cudaEventSynchronize(stop); // wait for all device work to complete
cudaEventElapsedTime(&elapsedTime, start, stop); //time between events
cudaEventDestroy(start); //destroy start event
cudaEventDestroy(stop);); //destroy stop event
Time period

10. Issues to watch for
First kernel launch will be more timing consuming than subsequent kernel executions because of code being transferred to GPU.
Asynchronous CUDA routines returning before they are complete – a big issue.

11. Asynchronous and synchronous calls
Kernels
Kernel starts after all previous CUDA calls completed
Control returned to CPU immediately (asynchronous, non-blocking)
cudaMemcpy
Copy starts after all previous CUDA calls completed
Returns after copy complete (synchronous)
NEW NVIDIA now says this applies only for transfers > 64KB.
From “CUDA C Programming Guide” October 2012, page 29.

12. Asynchronous CUDA routines
Control is returns before device has completed request tasked:
Kernel launches
Memory copies between two addresses in same device memory (Device to device memory copies)
Host to device memory copy (<= 64KB)
Memory copies with Async suffix
Memory set function calls
From “CUDA C Programming Guide” October 2012, page 29.

13. Timing within Kernel -- Using clock()
Possible to use clock() within kernel
See NVIDIA CUDA C Programming Guide, page 115:
“B.10 Time Function
clock_t clock();
when executed in device code, returns the value of a per-multiprocessor counter that is incremented every clock cycle. Sampling this counter at the beginning and at the end of a kernel, taking the difference of the two samples, and recording the result per thread provides a measure for each thread of the number of clock cycles taken by the device to completely execute the thread, but not of the number of clock cycles the device actually spent executing thread instructions. The former number is greater that the latter since threads are time sliced.”

14. Timing within Kernel -- Using events
Appears possible to use event timer within kernel.
Events can be recorded in specific “stream” objects – sequences of in-order code operating on a data set.
Events in default “stream 0” completed when all preceding operations completed by device
See NVIDIA CUDA C Programming Guide, page 39 for more details on streams.

15. Bandwidth Bandwidth is the rate at which data is transferred.
Physical connection will define the maximum system bandwidth.
Maximum bandwidth
K20 (cci-grid08) 208 GB/sec
C2050 (grid06/7) 144 GB/sec
GT 320M/330M (in Mac pro laptops) 25.6 GB/sec
Pentium Core i7 with Quickpath 25.6 GB/sec
Xbox 6.4 GB/sec
Wikipedia: Comparison of Nvidia graphics processing units

16. K20 bandwidthTest (cci-grid08) [abw@cci-grid08 ~]$ bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...
Device 0: Tesla K20c
Quick Mode
Host to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(MB/s)
Device to Host Bandwidth, 1 Device(s)
Device to Device Bandwidth, 1 Device(s)
Result = PASS
~]$

17. C2050 bandwidthTest (cci-grid07) [abw@cci-grid07 ~]$ bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...
Device 0: Tesla C2050
Quick Mode
Host to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(MB/s)
Device to Host Bandwidth, 1 Device(s)
Device to Device Bandwidth, 1 Device(s)
Result = PASS
~]$

18. Effective Bandwidth = (number_Bytes/time) * 10-9 GB/s
Effective bandwidth is the actual bandwidth achieved by a program. If we measure the effective bandwidth of a program, we can compare that to the maximum possible.
Effective bandwidth achieved by a program/kernel given by:
Effective Bandwidth = (number_Bytes/time) * 10-9 GB/s
where:
number_Bytes is total number of bytes read or written
time is the time period in seconds
GB/s = Gigabytes per second = 1,000,000,000 Bytes/s
Use effective bandwidth as a metric for measuring performance/optimization benefits*
* from NVIDIA CUDA C Best Practices Guide, Version 3.2, 8/20/2010

19. ( (N2 x b x 2) / time) x 10-9 GB/sec
Bandwidth of Matrix Copy Operation
Copying an N x N matrix:
( (N2 x b x 2) / time) x 10-9 GB/sec
where there are b bytes in each number. Need to know size of variables.
Integers, int (32 bits) b = 4 bytes
Floating point (32 bits) b = 4 bytes
Double (64 bits) b = 8 bytes
2 transfers -- Read plus write.
From NVIDIA CUDA C Best Practices Guide, Version 3.2, 8/20/2010

20. Computational Measures
GFLOPS
Classical measure in high performance computing (HPC) to measure performance is number of floating point operations. Systems have peak GFLOPs, and GFLOPs for doing LINPACK benchmark programs (single and double precision):
Tianhe PFLOPs
Titan PFLOPs (Linpack)
Cray Jaguar 1.75 PFLOPS
K20 (cci-grid08)) GFLOPS (single prec. peak)
C2050 (coit-grid06/cci-grid07) GFLOPS (single prec. peak)
GT 330M (in Mac pro laptops) 182 GFLOPS
Pentium Core i GFLOPS
* All numbers approximate as may not be not comparing under same conditions.
Petaflops, 1015 FLOPS, Gflops = 109 FLOPS)

21. Actual FLOPS
Measured using standard benchmark programs such as LINPACK
If measure it on your program, can see how close it get to the peak (which presumably is doing only floating point operations).

22. Sample partial code to measure performance on GPU
#define N // a big number up to INT_MAX, 2,147,483,647
__global__ void gpu_compute(float *result) {
int i, j;
float a = 0.0;
int tid = blockIdx.x * blockDim.x + threadIdx.x;
for (i = 0; i < N; i++)
for (j = 0; j < N; j++) a = a ; // do something, N x N floating pt operations
result[tid] = a; // store result
return; }
int main(int argc, char *argv[]) {
int T = 1, B = 1; // threads per block and blocks per grid
float cpu_result, *gpu_result, ans[T * B]; // result from gpu, to make sure computation is being done
cudaEvent_t start, end; // using cuda events to measure time
float time; // which is applicable for asynchronous code also
cudaEventCreate(&start); // instrument code to measure start time
cudaEventCreate(&end);
cudaEventRecord(start, 0 );
cudaMalloc((void**) &gpu_result, T * B * sizeof(float));
gpu_compute<<<B,T>>>(gpu_result);
cudaMemcpy(ans,gpu_result, T * B * sizeof(float),cudaMemcpyDeviceToHost);
cudaEventRecord(end, 0 ); // instrument code to measure end time
cudaEventSynchronize(end);
cudaEventElapsedTime(&time, start, end);
printf("GPU, Answer thread 0, %e\n", ans[0]);
printf("GPU Number of floating pt operations done %e\n", (double) N * N * T * B);
printf("GPU Time using CUDA events: %f ms\n", time); // time is in ms
cudaEventDestroy(start);
cudaEventDestroy(end);
return 0;
Sample partial code to measure performance on GPU

23. Questions