1. Automatic Performance Tuning of Sparse Matrix Kernels
Berkeley Benchmarking and OPtimization (BeBOP) Project
James Demmel, Katherine Yelick
Richard Vuduc, Shoaib Kamil, Rajesh Nishtala, Benjamin Lee
Atilla Gyulassy, Chris Hsu
University of California, Berkeley
January 24, 2003

2. Outline Performance tuning challenges Automatic performance tuning
Demonstrate complexity of tuning
Automatic performance tuning
Overview of techniques and results
New results for Itanium 2
Structure of the Google matrix
What optimizations are likely to pay-off?
Preliminary experiments: 2x speedups possible on Itanium 2

3. Tuning Sparse Matrix Kernels
Sparse tuning issues
Typical uniprocessor performance < 10% machine peak
Indirect, irregular memory references—poor locality
High bandwidth requirements, poor instruction mix
Performance depends on architecture, kernel, and matrix
How to select data structures, implementations? at run-time?
Our approach: for each kernel,
Identify and generate a space of implementations
Search to find the fastest (models, experiments)
Early success: SPARSITY
sparse matrix-vector multiply (SpMV) [Im & Yelick ’99]

4. Sparse Matrix Example n = 16146 nnz = 1.0M kernel: SpMV
Source: NASA structural analysis problem
FEM discretization from a structural analysis problem.
Source: Matrix “olafu”, provided by NASA via the University of Florida Sparse Matrix Collection

5. Sparse Matrix Example (enlarged submatrix)
nnz = 1.0M
kernel: SpMV
Natural 6x6 dense block structure
Upon closer inspection, this matrix has a natural 6x6 dense block structure.
In general, could have multiple block sizes aligned irregularly.

6. Speedups on Itanium: The Need for Search
6x6
Best: 3x1
Worst:
3x6
[NOTE: This slide has some animation in it.]
Experiment: Try all block sizes that divide 6x6—16 implementations in all.
Platform: 800 MHz Itanium-1, 3.2 Gflop/s peak speed. Intel v6.0 compiler.
Good speedups (1.44) but at an unexpected block size (3x1).
Results are platform dependent: based on Gautam’s experiments with our Sparsity benchmark code on a 1 GHz Itanium 2, the best block size for this matrix would most likely have been 2x6 (~850 Mflop/s out of 4 Gflop/s peak).
Reference

7. Filling-In Zeros to Improve Efficiency
More complicated non-zero structure in general
An enlarged submatrix for “ex11”, from an FEM fluid flow application.
Original matrix: n = 16614, nnz = 1.1M
Source: UF Sparse Matrix Collection

8. Filling-In Zeros to Improve Efficiency
More complicated non-zero structure in general
One SPARSITY technique: uniform register-level blocking
Example: 3x3 blocking
Logical 3x3 grid
Suppose we know we want to use 3x3 blocking.
SPARSITY lays a logical 3x3 grid on the matrix…

9. Filling-In Zeros to Improve Efficiency
More complicated non-zero structure in general
One SPARSITY technique: uniform register-level blocking
Example: 3x3 blocking
Logical 3x3 grid
Fill-in explicit zeros
“Fill ratio” = 1.5
On Pentium III: 1.5x speedup!
The main point is that there can be a considerable pay-off for judicious choice of “fill” (r x c), but that allowing for fill makes the implementation space even more complicated.
For this matrix on a Pentium III, we observed a 1.5x speedup, even after filling in an additional 50% explicit zeros. Two effects:
Filling in zeros, but eliminating integer indices (overhead)
(2) Quality of r x c code produced by compiler may be much better for particular r’s and c’s.
In this particular example, overall data structure size stays the same, but 3x3 code is 2x faster than 1x1 code for a dense matrix stored in sparse format.

10. Approach to Automatic Tuning
Recall: for each kernel,
Identify and generate implementation space
Search space to find fastest
Selecting the r x c register block size
Off-line: Precompute performance Mflops of SpMV using dense A for various block sizes r x c
Only once per architecture
Run-time: Given A, sample to estimate Fill for each r x c
Choose r, c to maximize ratio Mflops/Fill
Using this approach, we do choose the right block sizes for the two previous examples.

11. I think we can mention the bounds without talking about them in detail
I think we can mention the bounds without talking about them in detail. The main point is that this picture is a guide to what performance we get without blocking (reference) and what we might expect (bounds).
Note that absolute performance is highly dependent on matrix structure.

12. Summary of Results: Pentium III
Plot of the best block size (exhaustive search). Points:
It helps for some matrices and not others, but not blocking is part of the implementation space so we never do worse.
We’re close to the bounds. We should emphasize that the Intel compiler is generating the code here, so kudos to the Intel compiler group.

13. Summary of Results: Pentium III (3/3)
And our heuristic always chooses a good block size (includes 1x1) in practice.
Similar results for other platforms in the “Extra Slides” section.

14. Preliminary Results (Matrix Set 1): Itanium 2
Dense
FEM
FEM (var)
Assorted LP

15. Preliminary Results (Matrix Set 2): Itanium 2
Web/IR
Max speedups are several times faster than on earlier architectures (c.f., 200 Mflop/s on Itanium 1-800, 150 Mflop/s on Power 3, 300 Mflop/s on Pentium GHz).
Categories:
Matrix 0-2: dense, dense lower triangle, dense upper triangle Matrix 3-18: FEM (uniform blocking) Matrix 19-38: FEM (variable blocking Matrix 39-42: Protein modeling Matrix 43: Quantum chromodynamics calculation Matrix 44-46: Chemistry and chemical engineering Matrix 47-49: Circuits Matrix 50-57: Macroeconomic modeling Matrix 58-66: Statistical modeling Matrix 67-68: Finite-field problems Matrix 69-77: Linear programming Matrix 78-81: Information retrieval
Dense
FEM
FEM (var)
Bio
Econ
Stat LP

16. Exploiting Other Kinds of Structure
Optimizations for SpMV
Symmetry (up to 2x speedup)
Diagonals, bands (up to 2.2x)
Splitting for variable block structure (1.3x—1.7x)
Reordering to create dense structure + splitting (up to 2x)
Cache blocking (1.5—4x)
Multiple vectors (2—7x)
And combinations…
Sparse triangular solve
Hybrid sparse/dense data structure (1.2—1.8x)
Higher-level kernels
AATx, ATAx (1.2—4.2x)
RART, Akx, …

17. Multiple Vector Performance
On a dense matrix in sparse format, with multiple vectors we can get close to peak machine speed (up to 3 Gflop/s, peak on this machine is 3.6 Gflop/s).

18. What about the Google Matrix?
Google approach
Approx. once a month: rank all pages using connectivity structure
Find dominant eigenvector of a matrix
At query-time: return list of pages ordered by rank
Matrix: A = aG + (1-a)(1/n)uuT
Markov model: Surfer follows link with probability a, jumps to a random page with probability 1-a
G is n x n connectivity matrix [n » 3 billion]
gij is non-zero if page i links to page j
Normalized so each column sums to 1
Very sparse: about 7—8 non-zeros per row (power law dist.)
u is a vector of all 1 values
Steady-state probability xi of landing on page i is solution to x = Ax
Approximate x by power method: x = Akx0
In practice, k » 25
The matrix to which Google applies the power method is the sum of the connectivity graph matrix G plus a rank one update. According to Cleve’s corner, alpha ~= 85.
Macroscopic connectivity structure: Kumar, et al. “The web as a graph.” PODS
Size of G as of May ’02: Cleve’s corner. “The world’s largest sparse matrix computation.”
“k = 25”: Taher Stanford (student of Jeff Ullman), private communication: “For 120M pages, [the computation takes] about 5 hours for 25 iterations, no parallelization. The limiting factor is the disk speed for sequentially reading in the 8 GB matrix (it is assumed to be larger than main memory).” [13 Mar ’02]

19. Portion of the Google Matrix: A Snapshot
A very “small” (1M x 1M, 3M non-zeros) picture of a portion of G.
Assembled from an archived portion of the web stored at the Stanford WebBase
(40M — 200M page crawl done in 2001).

20. Possible Optimization Techniques
Within an iteration, i.e., computing (G+uuT)*x once
Cache block G*x
On linear programming matrices and matrices with random structure (e.g., LSI), 1.5—4x speedups
Best block size is matrix and machine dependent
Reordering and/or splitting of G to separate dense structure (rows, columns, blocks)
Between iterations, e.g., (G+uuT)2x
(G+uuT)2x = G2x + (Gu)uTx + u(uTG)x + u(uTu)uTx
Compute Gu, uTG, uTu once for all iterations
G2x: Inter-iteration tiling to read G only once
We have established techniques for efficiently computing within an iteration.
What about computing across iterations?

21. The blue bars compare the effect of just reordering the matrix—no significant improvements.
The red bars compare the effect of combined reordering and register blocking. Each red bar is labeled by the block size chosen, and by the fill (in parentheses). We are currently looking at splitting the matrix to reduce this fill overhead while sustaining the performance improvement.
In terms of storage cost (MB for double precision values and 32-bit integers):
Size(natural, RCM, MMD, no blocking) ~= 35.5 MB
Size(natural, 2x1) ~= 44.3 MB, or ~1.25x
Size(RCM, 3x1) ~= 54.9 MB, or ~1.55x
Size(MMD, 4x1) ~= 66.3 MB, or ~1.87x
Let k = no. of non-zeros, f be the fill, and r x c be the block size. Also suppose sizeof(floating point data type) = sizeof(integer data type). Then the size of CSR storage is
sizeof(A, ‘csr’) = 2k (ignoring row pointers)
sizeof(A, ‘rxc’) = kf * (1+1/(rc))
So we expect a storage savings even with fill when sizeof(A, ‘rxc’) < sizeof(A, ‘csr’):
kf * (1 + 1/(rc)) < 2k  f < 2 / (1 + (1/rc))
As rc  infinity, this condition becomes f < 2. (This case corresponds to the floating point values and integers being both 32-bit or both 64-bit, for example.)
If sizeof(real) = 2*sizeof(int), then this condition becomes f < 1.5 as rc  infinity. (This case corresponds to storing the values in double precision, but storing the indices in 32-bit integers.)
If sizeof(real) = 0.5*sizeof(int), then this condition becomes f < 3 as rc  infinity. (This case corresponds to storing the values in single precision, but the indices in 64-bit integers.)

22. Cache Blocked SpMV on LSI Matrix: Ultra 2i
10k x 255k
3.7M non-zeros
Baseline:
16 Mflop/s
Best block size
& performance:
16k x 64k
28 Mflop/s
Best block size on this platform:
Optimal size is machine dependent.

23. Cache Blocking on LSI Matrix: Pentium 4
10k x 255k
3.7M non-zeros
Baseline:
44 Mflop/s
Best block size
& performance:
16k x 16k
210 Mflop/s
LSI matrix has nearly randomly distributed non-zeros.

24. Cache Blocked SpMV on LSI Matrix: Itanium
10k x 255k
3.7M non-zeros
Baseline:
25 Mflop/s
Best block size
& performance:
16k x 32k
72 Mflop/s

25. Cache Blocked SpMV on LSI Matrix: Itanium 2
10k x 255k
3.7M non-zeros
Baseline:
170 Mflop/s
Best block size
& performance:
16k x 65k
275 Mflop/s

26. Inter-Iteration Sparse Tiling (1/3)x1 t1 y1
Let A be 6x6 tridiagonal
Consider y=A2x
t=Ax, y=At
Nodes: vector elements
Edges: matrix elements aij x2 t2 y2 x3 t3 y3
This generalization of Strout algorithm for an arbitrary Akx stolen from my quals talk. x4 t4 y4 x5 t5 y5

27. Inter-Iteration Sparse Tiling (2/3)x1 t1 y1
Let A be 6x6 tridiagonal
Consider y=A2x
t=Ax, y=At
Nodes: vector elements
Edges: matrix elements aij
Orange = everything needed to compute y1
Reuse a11, a12 x2 t2 y2 x3 t3 y3 x4 t4 y4 x5 t5 y5

28. Inter-Iteration Sparse Tiling (3/3)x1 t1 y1
Let A be 6x6 tridiagonal
Consider y=A2x
t=Ax, y=At
Nodes: vector elements
Edges: matrix elements aij
Orange = everything needed to compute y1
Reuse a11, a12
Grey = y2, y3
Reuse a23, a33, a43 x2 t2 y2 x3 t3 y3
Depending on cache placement policies, could reuse even more grey edges since they were read on previous iteration. x4 t4 y4 x5 t5 y5

29. Inter-Iteration Sparse Tiling: Issuesx1 t1 y1
Tile sizes (colored regions) grow with no. of iterations and increasing out-degree
G likely to have a few nodes with high out-degree (e.g., Yahoo)
Mathematical tricks to limit tile size?
Judicious dropping of edges [Ng’01] x2 t2 y2 x3 t3 y3
Idea of dropping edges follows from the following paper.
Andrew Ng, Alice Zheng, Michael I. Jordan. “Link Analysis, Eigenvectors, and Stability.” IJCAI, 2001. x4 t4 y4 x5 t5 y5

30. Summary and Questions Need to understand matrix structure and machine
BeBOP: suite of techniques to deal with different sparse structures and architectures
Google matrix problem
Established techniques within an iteration
Ideas for inter-iteration optimizations
Mathematical structure of problem may help
Questions
Structure of G?
What are the computational bottlenecks?
Enabling future computations?
E.g., topic-sensitive PageRank  multiple vector version [Haveliwala ’02]
See for more info, including more complete Itanium 2 results.

31. Extra slides

32. Sparse Kernels and Optimizations
Sparse matrix-vector multiply (SpMV): y=A*x
Sparse triangular solve (SpTS): x=T-1*b
y=AAT*x, y=ATA*x
Powers (y=Ak*x), sparse triple-product (R*A*RT), …
Optimization techniques (implementation space)
Register blocking
Cache blocking
Multiple dense vectors (x)
A has special structure (e.g., symmetric, banded, …)
Hybrid data structures (e.g., splitting, switch-to-dense, …)
Matrix reordering
How and when do we search?
Off-line: Benchmark implementations
Run-time: Estimate matrix properties, evaluate performance models based on benchmark data

33. Exploiting Matrix Structure
Symmetry (numerical or structural)
Reuse matrix entries
Can combine with register blocking, multiple vectors, …
Matrix splitting
Split the matrix, e.g., into r x c and 1 x 1
No fill overhead
Large matrices with random structure
E.g., Latent Semantic Indexing (LSI) matrices
Technique: cache blocking
Store matrix as 2i x 2j sparse submatrices
Effective when x vector is large
Currently, search to find fastest size
Symmetry could be numerical or structural.
In matrix splitting, we try to overcome the issues with too much fill but splitting into “true” r x c blocks (i.e., without fill) and the remainder in conventional (CSR) storage.

34. Symmetric SpMV Performance: Pentium 4

35. SpMV with Split Matrices: Ultra 2i

36. Cache Blocking on Random Matrices: Itanium
Speedup on four banded
random matrices.

37. Sparse Kernels and Optimizations
Sparse matrix-vector multiply (SpMV): y=A*x
Sparse triangular solve (SpTS): x=T-1*b
y=AAT*x, y=ATA*x
Powers (y=Ak*x), sparse triple-product (R*A*RT), …
Optimization techniques (implementation space)
Register blocking
Cache blocking
Multiple dense vectors (x)
A has special structure (e.g., symmetric, banded, …)
Hybrid data structures (e.g., splitting, switch-to-dense, …)
Matrix reordering
How and when do we search?
Off-line: Benchmark implementations
Run-time: Estimate matrix properties, evaluate performance models based on benchmark data

38. Example: Register Blocking for SpMV
Store dense r x c blocks
Reduces storage overhead and bandwidth requirements
Fully unroll block multiplies
Improves register reuse
Fill-in explicit zeros: trade-off extra computation for improved efficiency
x speedups on FEM matrices

39. Tuning Sparse Matrix-Vector Multiply (SpMV)
Sparsity [Im & Yelick ’99]
Optimizes y=A*x for sparse A, dense x, y
Selecting the register block size
Precompute performance Mflops of of dense A*x for various block sizes r x c
Given A, sample to estimate Fill for each r x c
Choose r, c to maximize ratio Mflops/Fill
Multiplication by multiple dense vectors
Block across vectors (by vector block size, v)

40. Off-line Benchmarking: Register Profiles
Register blocking performance for a dense matrix in sparse format.
333 MHz
Sun
Ultra 2i
35 Mflop/s
73 Mflop/s

41. Off-line Benchmarking: Register Profiles
Register blocking performance for a dense matrix in sparse format.
333 MHz
Sun
Ultra 2i 35 73
500 MHz
Intel
Pentium
III 42
105
375 MHz
IBM
Power3 88
172
800 MHz
Itanium
110
250

42. Register Blocked SpMV: Pentium III

43. Register Blocked SpMV: Ultra 2i

44. Register Blocked SpMV: Power3

45. Register Blocked SpMV: Itanium

46. Multiple Vector Performance: Itanium

47. Sparse Kernels and Optimizations
Sparse matrix-vector multiply (SpMV): y=A*x
Sparse triangular solve (SpTS): x=T-1*b
y=AAT*x, y=ATA*x
Powers (y=Ak*x), sparse triple-product (R*A*RT), …
Optimization techniques (implementation space)
Register blocking
Cache blocking
Multiple dense vectors (x)
A has special structure (e.g., symmetric, banded, …)
Hybrid data structures (e.g., splitting, switch-to-dense, …)
Matrix reordering
How and when do we search?
Off-line: Benchmark implementations
Run-time: Estimate matrix properties, evaluate performance models based on benchmark data

48. Example: Sparse Triangular Factor
Raefsky4 (structural problem) + SuperLU + colmmd
N=19779, nnz=12.6 M
Dense trailing triangle: dim=2268, 20% of total nz

49. Tuning Sparse Triangular Solve (SpTS)
Compute x=L-1*b where L sparse lower triangular, x & b dense
L from sparse LU has rich dense substructure
Dense trailing triangle can account for 20—90% of matrix non-zeros
SpTS optimizations
Split into sparse trapezoid and dense trailing triangle
Use tuned dense BLAS (DTRSV) on dense triangle
Use Sparsity register blocking on sparse part
Tuning parameters
Size of dense trailing triangle
Register block size

50. Sparse/Dense Partitioning for SpTS
Partition L into sparse (L1,L2) and dense LD:
Perform SpTS in three steps:
Sparsity optimizations for (1)—(2); DTRSV for (3)

51. SpTS Performance: Itanium
(See POHLL ’02 workshop paper, at ICS ’02.)

52. SpTS Performance: Power3

53. Sparse Kernels and Optimizations
Sparse matrix-vector multiply (SpMV): y=A*x
Sparse triangular solve (SpTS): x=T-1*b
y=AAT*x, y=ATA*x
Powers (y=Ak*x), sparse triple-product (R*A*RT), …
Optimization techniques (implementation space)
Register blocking
Cache blocking
Multiple dense vectors (x)
A has special structure (e.g., symmetric, banded, …)
Hybrid data structures (e.g., splitting, switch-to-dense, …)
Matrix reordering
How and when do we search?
Off-line: Benchmark implementations
Run-time: Estimate matrix properties, evaluate performance models based on benchmark data

54. Optimizing AAT*x Kernel: y=AAT*x, where A is sparse, x & y dense
Arises in linear programming, computation of SVD
Conventional implementation: compute z=AT*x, y=A*z
Elements of A can be reused:
When ak represent blocks of columns, can apply register blocking.

55. Optimized AAT*x Performance: Pentium III

56. Current Directions Applying new optimizations
Other split data structures (variable block, diagonal, …)
Matrix reordering to create block structure
Structural symmetry
New kernels (triple product RART, powers Ak, …)
Tuning parameter selection
Building an automatically tuned sparse matrix library
Extending the Sparse BLAS
Leverage existing sparse compilers as code generation infrastructure
More thoughts on this topic tomorrow

57. Related Work Automatic performance tuning systems Code generation
PHiPAC [Bilmes, et al., ’97], ATLAS [Whaley & Dongarra ’98]
FFTW [Frigo & Johnson ’98], SPIRAL [Pueschel, et al., ’00], UHFFT [Mirkovic and Johnsson ’00]
MPI collective operations [Vadhiyar & Dongarra ’01]
Code generation
FLAME [Gunnels & van de Geijn, ’01]
Sparse compilers: [Bik ’99], Bernoulli [Pingali, et al., ’97]
Generic programming: Blitz++ [Veldhuizen ’98], MTL [Siek & Lumsdaine ’98], GMCL [Czarnecki, et al. ’98], …
Sparse performance modeling
[Temam & Jalby ’92], [White & Saddayappan ’97], [Navarro, et al., ’96], [Heras, et al., ’99], [Fraguela, et al., ’99], …

58. More Related Work Compiler analysis, models Sparse BLAS interfaces
CROPS [Carter, Ferrante, et al.]; Serial sparse tiling [Strout ’01]
TUNE [Chatterjee, et al.]
Iterative compilation [O’Boyle, et al., ’98]
Broadway compiler [Guyer & Lin, ’99]
[Brewer ’95], ADAPT [Voss ’00]
Sparse BLAS interfaces
BLAST Forum (Chapter 3)
NIST Sparse BLAS [Remington & Pozo ’94]; SparseLib++
SPARSKIT [Saad ’94]
Parallel Sparse BLAS [Fillipone, et al. ’96]

59. Context: Creating High-Performance Libraries
Application performance dominated by a few computational kernels
Today: Kernels hand-tuned by vendor or user
Performance tuning challenges
Performance is a complicated function of kernel, architecture, compiler, and workload
Tedious and time-consuming
Successful automated approaches
Dense linear algebra: ATLAS/PHiPAC
Signal processing: FFTW/SPIRAL/UHFFT

60. Cache Blocked SpMV on LSI Matrix: Itanium

61. Sustainable Memory Bandwidth

62. Multiple Vector Performance: Pentium 4

63. Multiple Vector Performance: Itanium

64. Multiple Vector Performance: Pentium 4

65. Optimized AAT*x Performance: Ultra 2i

66. Optimized AAT*x Performance: Pentium 4

67. Tuning Pays Off—PHiPAC

68. Tuning pays off – ATLAS
Extends applicability of PHIPAC; Incorporated in Matlab (with rest of LAPACK)

69. Register Tile Sizes (Dense Matrix Multiply)
333 MHz Sun Ultra 2i
2-D slice of 3-D space; implementations color-coded by performance in Mflop/s
16 registers, but 2-by-3 tile size fastest
A 2-D slice of the 3-D core matmul space, with all other generator options fixed. This shows the performance (Mflop/s) of various core matmul implementations on a small in-L2 cache workload. The “best” is the dark red square at (m0=2,k0=1,n0=8), which achieved 620 Mflop/s. This experiment was performed on the Sun Ultra-10 workstation (333 MHz Ultra II processor, 2 MB L2 cache) using the Sun cc compiler v5.0 with the flags -dalign -xtarget=native -xO5 -xarch=v8plusa.
The space is discrete and highly irregular. (This is not even the worst example!)

70. Search for Optimal L0 block size in dense matmul
IBM RS/6000 (pink) has a generous memory bandwidth. For instance, we can see that 10% of the implementations achieved at least 85% of theoretical machine peak (nothing got more than 95% peak).
The Cray T3E node (450 MHz DEC Alpha 21164) is the most “difficult” of the platforms (no L2 cache). Only 10% achieved even 30% of peak performance!
The other platforms fall in between.
The main point is that the distribution of performance is sensitive to the platform/compiler. We cannot assume a particular distribution (or average distribution) will accurately characterize all platforms. Moreover, since the distribution captures some aspect of platform-dependence, maybe we can make use of this information (we will!).

71. High Precision GEMV (XBLAS)

72. High Precision Algorithms (XBLAS)
Double-double (High precision word represented as pair of doubles)
Many variations on these algorithms; we currently use Bailey’s
Exploiting Extra-wide Registers
Suppose s(1) , … , s(n) have f-bit fractions, SUM has F>f bit fraction
Consider following algorithm for S = Si=1,n s(i)
Sort so that |s(1)|  |s(2)|  …  |s(n)|
SUM = 0, for i = 1 to n SUM = SUM + s(i), end for, sum = SUM
Theorem (D., Hida) Suppose F<2f (less than double precision)
If n  2F-f + 1, then error  1.5 ulps
If n = 2F-f + 2, then error  22f-F ulps (can be  1)
If n  2F-f + 3, then error can be arbitrary (S  0 but sum = 0 )
Examples
s(i) double (f=53), SUM double extended (F=64)
accurate if n  = 2049
Dot product of single precision x(i) and y(i)
s(i) = x(i)*y(i) (f=2*24=48), SUM double extended (F=64) 
accurate if n  = 65537