® Backward Error Analysis and Numerical Software Sven Hammarling NAG Ltd, Oxford

Plan of Talk Introduction –Stability –Error analysis –Condition The Basic Linear Algebra Subprograms (BLAS) LAPACK: state of the art Linear Algebra PACKage ScaLAPACK: distributed memory version of LAPACK (Scalable LAPACK) Other Aspects Summary

Stability The stability of a method for solving a problem is concerned with the sensitivity of the method to (rounding) errors in the solution process A method that guarantees as accurate a solution as the data warrants is said to be stable, otherwise the method is unstable

Example - Sample Variance

Error Analysis Error analysis is concerned with establishing whether or not an algorithm is stable for the problem in hand A forward error analysis is concerned with how close the computed solution is to the exact solution A backward error analysis is concerned with how well the computed solution satisfies the problem to be solved

Example

Example (cont’d)

Componentwise and Normwise

Condition The condition of a problem is concerned with the sensitivity of the problem to perturbations in the data A problem is ill-conditioned if small changes in the data cause relatively large changes in the solution. Otherwise a problem is well- conditioned

® Example

Condition Number for Linear Equations

Condition and Error Analysis

Example

Backward Error as Perturbations in the Data

Backward Error as … (cont’d)

Example

Backward Error Analysis

Backward Error and Perturbation Analysis

The Purpose of Error Analysis “The clear identification of the factors determining the stability of an algorithm soon led to the development of better algorithms. The proper understanding of inverse iteration for eigenvectors and the development of the QR algorithm by Francis are the crowning achievements of this line of research. “For me, then, the primary purpose of the rounding error analysis was insight.” Wilkinson, 1986 Bulletin of the IMA, Vol 22, p197

LAPACK and Error Bounds “In Addition to providing faster routines than previously available, LAPACK provides more comprehensive and better error bounds. Our goal is to provide error bounds for most quantities computed by LAPACK.” LAPACK Users’ Guide, Chapter 4 – Accuracy and Stability

The BLAS, LAPACK and ScaLAPACK

The Three Levels of BLAS The Level 1 BLAS are concerned with scalar and vector operations, such as the Level 2 BLAS with matrix-vector operations such as and the Level 3 BLAS with matrix-matrix operations such as

Why Higher Level BLAS? Registers L1 Cache L2 Cache Local Memory Remote Memory Secondary Memory BLASMem RefsOpsRatio Level 1 Level 2 Level 3 3 : 2 1 : 2 2 : n 2n2n 2n22n2 2n32n3 3n3n n2n2 4n24n2 The so called ‘surface to volume’ effect

BLAS for Performance Level 3 BLAS Level 2 BLAS Level 1 BLAS

Publication of the Specifications of the BLAS [1] C L. Lawson, R. J. Hanson, D. Kincaid, and F. T. Krogh. Basic Linear Algebra Subprograms for FORTRAN usage. ACM Trans. Math. Software, 5: , [2] J. J. Dongarra, J. Du Croz, S. Hammarling, and R. J. Hanson. An extended set of FORTRAN Basic Linear Algebra Subprograms. ACM Trans. Math. Software, 14:1-32, 399, (Algorithm 656). [3] J. J. Dongarra, J. Du Croz, I. S. Duff, and S. Hammarling. A set of Level 3 Basic Linear Algebra Subprograms. ACM Trans. Math. Software, 16:1-28, (Algorithm 679).

LAPACK Systems of linear equations Linear least squares problems Eigenvalue and singular value problems, including generalized problems Matrix factorizations Condition and error estimates The BLAS as a portability layer Linear Algebra PACKage for high-performance computers Dense and banded linear algebra for Shared Memory

Benefits of LAPACK Comprehensive and consistent set of routines for dense and banded linear algebra Efficient on a wide range of modern computers Additional functionality compared to previous packages Improved numerical algorithms Error and sensitivity estimates Portability layer between users applications and the machines

LAPACK Users’ Guide E. Anderson, Z. Bai, C. Bischof S. Blackford, J. Demmel, J. Dongarra J. Du Croz, A. Greenbaum, S. Hammarling A. McKenney, D. Sorensen SIAM, 1999 (3rd Edition) Proceeds to SIAM student travel fund L A P A C K L -A P -A C -K L A P A -C -K L -A P -A -C K L A -P -A C K L -A -P A C -K

1/4 1/L 1/L 1/L 1/L 1/A -1/A 1/A -1/A 1/P 1/P -1/P -1/P 1/A -1/A -1/A 1/A 1/C 1/C -1/C -1/C 1/K -1/K -1/K 1/K

Performance Improvements Efficiency in LAPACK has been principally achieved by restructuring algorithms so that as much computation as possible is performed by calls to: Level 3 and Level 2 BLAS In order to exploit the Level 3 BLAS we needed: BLOCK-PARTITIONED ALGORITHMS a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 A 11 A 12 A 13 A 21 A 22 A 23 A 31 A 32 A 33

Error Bounds in LAPACK The Users’ Guide gives details of error bounds and code fragments to compute those bounds. In many cases the routines return the bounds directly

Error Bounds in LAPACK: Example 1

Error Bounds in LAPACK: Example 2

Error Bounds in LAPACK: Example 2 (cont’d) Asymptotic error bounds for nonsymmetric eigenproblem See Golub and Van Loan – Matrix Computations

Error Bounds in LAPACK: Example 2 (cont’d) EPSMCH = DLAMCH(‘E’) DO 10 I = 1, N EERRBD(I) = EPSMCH*ABNRM/RCONDE(I) VERRBD(I) = EPSMCH*ABNRM/RCONDV(I) 10 CONTINUE EPSMCH is the machine precision

Availability of LAPACK LAPACK is freely available via netlib: Much of LAPACK is included in the NAG Fortran 77 Library and is the basis of the dense linear algebra in the NAG Fortran 90 Library Tuned versions of some LAPACK routines are in the NAG Fortran SMP Library

LAPACK - Summary LAPACK success from HP-48G to Cray T-90 Portable Fortran 77 (+BLAS), 766K lines (354K library, 412K testing and timing) PCs, workstations, vector machines, and shared memory computers Initial release February 29th, 1992 Now on the 3 rd release Linear systems, least squares, eigenproblems C, Java and Fortran 95 interfaces

ScaLAPACK Scalability, performance and portability Interfaces to stay as close to LAPACK as possible Use LAPACK algorithms wherever possible Principal additional consideration is the data distribution The PBLAS (BLAS + BLACS) as a portabilility layer Scalable Linear Algebra PACKage Aim: To port LAPACK to distributed memory environments As with LAPACK, ScaLAPACK is freely available: Most of ScaLAPACK included in NAG Parallel Library

ScaLAPACK Structure ScaLAPACK BLAS LAPACK BLACS PVM/MPI/... PBLAS Global Local

ScaLAPACK Summary Follow on to LAPACK - Designed for MPP’s Object based design Portability across wide range of architectures –will work on a heterogeneous platform Software, examples, prebuilt libraries available from netlib Users’ Guide, including CD available from SIAM Continuing development HPF interface to some driver routines available

Hierarchy of Building Blocks Users Applications Packages (Sca)LAPACK (P)BLAS Machines

Developments Since Wilkinson More problems solved, tighter bounds More emphasis on componentwise results – particularly for sparse and structured problems Use of standard precision iterative refinement Importance of scaling Iterative methods Importance of non-normality and pseudospectra

Diagonal Matrix Example

Backward Errors in Other Areas For references to other areas see: Higham, Accuracy and Stability of Numerical Algorithms, Sections 1.20 & Examples: –ODEs –Chaotic behaviour of iterations

What to do if Software does not Provide Estimates? Run the problem with perturbed data Better still, use a software tool such as PRECISE which allows you to perform a stochastic analysis Put pressure on developers to provide estimates

PRECISE Provides a module for statistical backward error analysis Provides a module for sensitivity analysis index.html

Quality of Reliable Software

References [1] E. Anderson, Z. Bai, C. H. Bischof, J. Demmel, J. J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Blackford & D. C. Sorensen. LAPACK Users’ Guide, 3 rd Edition, SIAM, [2] F. Chaitin-Chatelin & V. Frayssé. Lectures on Finite Precision Computations, SIAM, [3] G. H. Golub & C. F. Van Loan. Matrix Computations, 3 rd Edition, The Johns Hopkins University Press, [4] N. J. Higham. Accuracy and Stability of Numerical Algorithms, SIAM, 1996.

® Closing Remarks LAPACK and ScaLAPACK provide efficient and reliable portable software for numerical linear algebra Emphasis has been placed on error and condition (sensitivity) estimates Research and development is ongoing Available via the NAG Libraries Expect error estimates from numerical software Use software tools