1. CUDA Linear Algebra Library and Next Generation
Yukai Hung
Department of Mathematics National Taiwan University

2. Sparse Matrix-Vector Multiplication

3. Sparse Matrix-Vector Multiplication
Dense approach is wasteful
- unclear how to map work to parallel processors
- irregular elements accessing for global memory 3

4. Sparse Matrix-Vector Multiplication
structured
unstructured
DIA - diagonal format
ELL - ellpack format
CSR - compressed row format
HYB - hybrid format
COO - coordinate format 4

5. global memory coalescing
Sparse Matrix-Vector Multiplication
Diagonal format
- diagonal should be mostly populated format
- high parallelism to map one thread for one row
- good parallel efficiency and good memory behavior
global memory coalescing 5

6. Sparse Matrix-Vector Multiplication
Ellpack format
- assign one thread to compute one row again
- but the load imbalance hurts parallel efficiency 6

7. Sparse Matrix-Vector Multiplication
Coordinate format
- insensitive to sparsity pattern but slower than ellpack
- assign one thread for one element and combine the
results from all elements in a row to get output element 7

8. Sparse Matrix-Vector Multiplication
Hybrid format
- combine regular ellpack format and flexible coo format
typical
exceptional 8

9. fixed number of nonzeros and variable matrix size
Sparse Matrix-Vector Multiplication
Property comparison
fixed number of nonzeros and variable matrix size
Matrix Format
Granularity
Coalescing
DIA
thread/row
full
ELL
CSR(scalar)
rare
CSR(vector)
warp/row
partial
COO
thread/nonzero
HYB 9

10. Sparse Matrix-Vector Multiplication
Sparse matrices for parallel efficiency: ellpack format
- one thread per row is efficient for memory accessing
Sparse matrices for load imbalance: coordinate format
- one thread per element is insensitive to matrix structure
Conclusion for all structures
- hybrid structure gives the best performance averagely
- irregularity is manageable if regularize the common case 10

11. Sparse Matrix-Vector Multiplication
Performance comparison 11

12. Sparse Matrix-Vector Multiplication
Performance comparison 12

13. Sparse Matrix-Vector Multiplication
Performance comparison 13

14. Linear Algebra Library

15. Linear Algebra Library
CUBLAS: CUDA Basic Linear Algebra Subroutines
- implement basic linear algebra subroutines on runtime level
- only available for single device not implement for multiple devices
CUFFT: CUDA Fast Fourier Transforms Library
- use divide-and-conquer algorithm for discrete transform
- support real and complex data for in-place or out-of-place
- support the stream operation for simultaneous execution
- use complex-to-complex to replace real-to-complex
- problem size in power-of-two gives best performance 15

16. Linear Algebra Library
CUDPP: CUDA Data Parallel Primitive Library
- a library of data-parallel algorithm primitives
- parallel prefix-sum and sorting and data reduction
- stream compaction and random number generator 16

17. comparison with multicore CPU
Linear Algebra Library
CUDPP: CUDA Data Parallel Primitive Library
comparison with multicore CPU 17

18. Linear Algebra Library
CULA: GPU-Accelerated Linear Algebra Library
- implement LAPACK function for variant language interface
linear system solves
least square solvers
orthogonal factorization
symmetric eigenproblem
non-symmetric eigenproblem
singular value decompositions 18

19. Linear Algebra Library
CULA: GPU-Accelerated Linear Algebra Library
- implement LAPACK function for variant language interface
double precision QR-factorization
double precision LU-factorization 19

20. Linear Algebra Library
CULA: GPU-Accelerated Linear Algebra Library
- implement LAPACK function for variant language interface
double precision symmetric eigenvalue problem
double precision QR-factorization 20

21. Linear Algebra Library
CULA: GPU-Accelerated Linear Algebra Library
- implement LAPACK function for variant language interface
double precision symmetric eigenvalue problem
double precision singular value decomposition 21

22. Linear Algebra Library
MAGMA: Matrix Algebra on GPU and Multicore Architecture
- open source project to develop a dense linear algebra library
similar to basic linear algebra package but for heterogeneous
and hybrid architecture with manycore CPUs and GPUs systems 22

23. Linear Algebra Library
MAGMA: Matrix Algebra on GPU and Multicore Architecture
single precision QR-factorization
double precision matrix-matrix multiplication 23

24. Linear Algebra Library
MAGMA: Matrix Algebra on GPU and Multicore Architecture
solving Ax=b by using LU-factorization
single precision QR-factorization 24

25. Linear Algebra Library
MAGMA: Matrix Algebra on GPU and Multicore Architecture
single precision Cholesky-factorization
solving Ax=b by using LU-factorization 25

26. Linear Algebra Library
Thrust
- thrust is a CUDA library of parallel algorithm with an interface
resembling the C++ Standard Template Library STL to provide
flexible high-level interface that greatly enhance productivity 26

27. Linear Algebra Library
int main(int argc,char** argv) {
//allocate memory space on the host
thrust::host_vector<float> hvec(1024);
//generate random number on the host
thrust::generate(hvec.begin(),hvec.end(),rand);
//allocate and transfer data to device
thrust::device_vector<float> dvec=hvec;
//manipulate device values from the host
dvec[0]=(float)rand()/(float)(RAND_MAX-1);
dvec[1]=(float)rand()/(float)(RAND_MAX-1);
//sum all data on device by parallel reduction
sum=thrust::reduce(dvec.begin(),dvec.end());
//sort all data on device by radix sort
thrust::sort(dvec.begin(),dvec.end());
//transfer final data back to host
thrust::copy(dvec.begin(),dvec.end(),hvec.begin()); } 27

28. Linear Algebra Library
int main(int argc,char** argv) {
//create list container on the host
std::list<int> hlist;
hlist.push_back(13);
hlist.push_back(27);
//copy host data from list into device vector
thrust::device_vector<int> dvec(hlist.size());
thrust::copy(hlist.begin(),hlist.end(),dvec.begin());
//alternative method to convert from host to device
thrust::device_vector<int> dvec(hlist.begin(),hlist.end());
//obtain raw pointer from device memory
int* dpointer=thrust::raw_pointer_cast(dvec);
//launch device kernel function
kernel<<<blocknum,blocksize>>>(dpointer,dvec.size());
//deallocate device memory
cudaFree(dpointer); } 28

29. Linear Algebra Library
CUSP: Generic Parallel Algorithm for Sparse Matrix Computations
- cusp provides a high-level and flexible interface for manipulating
sparse matrix and solving sparse linear systems by iterative method
- cusp is implemented on the thrust template interface structure
"Implementing Sparse Matrix-Vector Multiplication on Throughput-Oriented
Processors“ Nathan Bell and Michael Garland, in "Supercomputing 09", 2009 29

30. Linear Algebra Library
CUSP: Generic Parallel Algorithm for Sparse Matrix Computations 30

31. Linear Algebra Library
Matrix format
- cusp natively supports several sparse matrix formats
- cusp make it is easy to transfer sparse matrix data between
host and device and convert between sparse matrix format
//allocate storage space for a CSR matrix on the
//host with 5 row 8 column and 12 nonzero elements
cusp::csr_matrix<int,float,cusp::host_memory> A(5,8,12);
//allocate and transfer from host to device memory
cusp::csr_matrix<int,float,cusp::device_memory> B=A;
//convert the CSR matrix format to HYB matrix format
cusp::hyb_matrix<int,float,cusp::device_memory> C=A; 31

32. Linear Algebra Library
Algorithm and iterative solver
- matrix-vector multiplication sand transpose
- conjugate gradient and biconjugate gradient stab
//matrix-vector multiplication
cusp::multiply(A,x,y)
//sparse matrix transpose
cusp::transpose(A,At)
//conjugate gradient
cusp::krylov::cg(A,x,b)
//biconjugate gradient stab
cusp::krylov::bicgstab(A,x,b) 32

33. Linear Algebra Library
int main(int argc,char** argv) {
//create an empty HYB sparse matrix structure
cusp::hyb_matrix<int,float,cusp::device_memory> A;
//load a matrix stored in the matrix market format
cusp::io::read_matrix_market_file(A,”5pt_10x10.mtx”);
//allocate storage for solution x and right-hand side b
cusp::array1d<float,cusp::device_memory> x(A.num_rows,0);
cusp::array1d<float,cusp::device_memory> b(A.num_rows,1);
//set the iteration and residual stopping criteria
cusp::verbose_monitor<ValueType> monitor(100,1e-6);
//setup the matrix preconditioner
cusp::precond::diagonal<ValueType,MemorySpace> M(A);
//solve the linear system with conjugate gradient method
cusp::krylov::cg(A,x,b,monitor,M);
return 0; } 33

34. Linear Algebra Library
OpenNL: Open Numerical Library
- efficient sparse matrix data structure
- sparse direct linear solver for SuperLU
- matrix preconditioner for Jacobi and SSOR
- iterative builder for sparse least-square problems
- iterative solvers for conjugate gradient, BICGSTAB, GMRES 34

35. Linear Algebra Library
ViennaCL
- a basic linear algebra for computations on GPUs based on OpenCL
- support basic linear algebra subroutines
- generalized minimal residual method
- direct linear system solver with LU-factorization
- sparse conjugate gradient and biconjugate gradient
- optimal incomplete LU preconditioner with threshold
GATLAS: GPU Automatically Tuned Linear Algebra Subroutines
- automatically tuned the kernel of level 3 BLAS based on OpenCL 35

36. Next Generation Architecture

37. Next Generation Architecture
Next GPU generation architecture is called Fermi 37

38. Next Generation Architecture
Next GPU generation architecture is called Fermi 38

39. Next Generation Architecture
Third generation Streaming Multiprocessor
dual thread/warp scheduler
32 processors for each SM
double precision 50% of single
(8X faster than GT200)
4 special function units
64 KB of RAM for shared memory and configurable L1 cache 39

40. Next Generation Architecture
Second generation Parallel Thread Execution
IEEE floating point standard, surpassing even the most advanced CPU
Fused multiply-add FMA instruction
for both single and double precision
Newly designed 32-bit integer ALU
and extended precision operations 40

41. Next Generation Architecture
Improved Memory System
first GPU architecture to support true cache hierarchy in combination with on-chip shared memory
L1 cache for each multiprocessor
improve bandwidth/reduce latency
unified L2 cache (768 KB)
coherent data sharing across all cores
ECC support GDDR5 memory interface
which is almost 2X faster than GDDR3 41

42. Next Generation Architecture
GigaThread Hardware Scheduler
Hierarchically manage thousands
of simultaneously active threads
10X faster application context switching to support concurrent
kernel execution 42

43. concurrent kernel execution + faster context switch
Next Generation Architecture
GigaThread Hardware Scheduler
concurrent kernel execution + faster context switch 43

44. Next Generation Architecture
GigaThread Hardware Scheduler
Dual DMA engines for simultaneous data transfer
to fully overlap with CPU and GPU processing time 44

45. Next Generation Architecture
Third generation Streaming Multiprocessor
fully pipeline of integer arithmetic logic unit and floating-point unit
improve floating-point arithmetic from IEEE to IEEE
to support FMA instruction
improve integer ALU from 24-bit precision into 32-bit precision 45

46. original multiply-add
Next Generation Architecture
What is NEW on the floating-point operation?
- support fused multiply-add instructions for both single and double
fused multiply-add
original multiply-add A x B =
product
retain all digits
truncate extra digits + C =
result 46

47. Next Generation Architecture
What is NEW on the floating-point operation?
- support subnormal numbers for both single and double precision
which are small numbers that lie between the zero and smallest
normalized number of a given floating point number system
- prior generation flush subnormal operand and results to zero
- CPU typically perform subnormal calculation in exception-handling
software taking thousands of cycles, but Fermi handle subnormal
calculations in hardware with no additional performance penalty 47

48. Next Generation Architecture
Third generation Streaming Multiprocessor
16 load/store units to allow source and destination addresses to be calculated for 16 threads per cycle
32 single precision FMA units
16 double precision FMA units 48

49. double precision application performance
Next Generation Architecture
Third generation Streaming Multiprocessor
double precision application performance 49

50. instruction dispatch units
Next Generation Architecture
Third generation Streaming Multiprocessor
two warp scheduler and
instruction dispatch units 50

51. Next Generation Architecture
Third generation Streaming Multiprocessor
dual warp scheduler allowing two warps to be issued and executed concurrently for 32 cores 51

52. Next Generation Architecture
Third generation Streaming Multiprocessor
two warp scheduler and
instruction dispatch units
64KB configurable shared memory and L1 cache 52

53. radix sort using shared memory
Next Generation Architecture
64KB configurable shared memory and L1 cache
- 48KB shared memory and 16KB L1 cache
- 16KB shared memory and 48KB L1 cache
radix sort using shared memory 53

54. Next Generation Architecture
Unified memory address space
- combine three separate addresses space for load and store
- this feature enable Fermi to support all C++ specific programs
virtual function, function pointer, new and delete object, try and catch 54

55. Next Generation Architecture
summary table 55

56. Next Generation Architecture
scheduler bottleneck L2 FB SP L1 TF
Thread Processor
Vtx Thread Issue
Setup / Rstr / ZCull
Work Distribution
Pixel Thread Issue
Input Assembler
Host 56

57. Next Generation Architecture
new bottleneck
old bottleneck 57

58. - Mark Harris http://www.markmark.net/
Reference
- Mark Harris
- Wei-Chao Chen
- Wen-Mei Hwu 58