TEMPLATE DESIGN © H. Che 2, E. D’Azevedo 1, M. Sekachev 3, K. Wong 3 1 Oak Ridge National Laboratory, 2 Chinese University of Hong Kong, 3 National Institute for Computational Sciences Motivation High Performance Linpack (HPL) benchmark often outperforms ScaLAPACK in solving a large dense system of equation, but is only commonly used for performance benchmark in double precision. => HPL for LU decomposition Single Instruction Multiple Data (SIMD) capability in most processors can achieve higher performance in 32-bit over 64-bit operations. => Iterative refinement procedure Goals ScaLAPACK Based Application Code Modifications Numerical Experiments Summary Acknowledgements Contact information Eduardo F. D’Azevedo, ORNL Kwai Wong, NICS Deliver a user library by extending HPL to perform single, double, complex, double complex, and mixed precisions calculations. Deliver an interface compatible to ScaLAPACK calling convention, which allows simple code modifications to achieve top performance using this modified HPL (libmhpl.a) library. Use mixed precision solver by performing costly LU factorization in 32- bit but achieve 64-bit accuracy by iterative refinement method. Mixed Precision Iterative Refinement Solver Procedure Using High Performance Linpack (HPL) This research is partially sponsored by the Office of Advanced Scientific Computing Research; U.S. Department of Energy. This research used resources of the National Institute for Computational Sciences (NICS), which is supported by the National Science Foundation (NSF). Summer internships for H. Che, T. Chan, D. Lee, and R. Wong were supported by the Department of Mathematics, The Chinese University of Hong Kong (CUHK). Internship opportunity was provided by the Joint Institute for Computational Sciences (JICS), the University of Tennessee, and the Oak Ridge National Laboratory. HPL based dense LU solver is more efficient than standard ScaLAPACK, and achieved about 75% of the peak performance. HPL performs parallel LU factorization in double but uses hybrid left/right-looking panel method and look-ahead algorithms. Application interface compatible to ScaLAPACK. Integrated to AORSA fusion INCITE application. High Performance Linpack (HPL) Performing simple modifications of ScaLAPACK source codes one can embed calling more efficient HPL functions, simply linked with libmhpl.a provided here. For example the test program provided by ScaLAPACK ( pdludriver.f ) should be modified as follow as follows ( pdhpl_driver.f ): 1.Two extra integers are declared: integer hpl_lld, hpl_ineed 2.HPL extra functions are called: call blacs_barrier(ictxt,'A') call hpl_dblacsinit( ictxt ) call hpl_dmatinit( n, NB, hpl_lld,hpl_ineed) call descinit(descA,n,n+1,NB,NB,0,0,ICTXT,hpl_lld,ierr(1)) 3.Original ScaLAPACK function CALL PDGETRF(M,N,MEM(IPA),1,1,DESCA,MEM(IPPIV),INFO ) is replaced by HPL function call hpl_pdgesv(n, mem(IPA), descA,mem(ippiv), info) Results Table 1: Performance comparison: HPL vs. ScaLAPACK HPL is written in portable C to evaluate parallel performance of Top500 computers by solving a (random) dense linear system in double precision (64-bit) arithmetic on distributed-memory computers. HPL uses a right-looking variant of the LU factorization of random matrix with row partial pivoting. 2-D block-cyclic distribution is employed. It has tunable parameters to implement multiple look-ahead depths, recursive panel factorization with pivot search and column broadcast combined, various virtual panel broadcast topologies, bandwidth reducing swap broadcast algorithm, and a ring broadcast algorithm in the backward substitution Methodology /* * Find norm_1( A ). */ if( nq > 0 ) { work = (double*)malloc( nq * sizeof( double ) ); if( work == NULL ) { HPL_pabort( __LINE__, "HPL_pdlange", "Memory allocation failed" ); } for( jj = 0; jj < nq; jj++ ) { s = HPL_rzero; for( ii = 0; ii < mp; ii++ ) { s += Mabs( *A ); A++; } work[jj] = s; A += LDA - mp; } float*float HPL_pslange (float)HPL_rzero Unchanged: Timing variables: /hpl/testing /ptimer, /timer Norm Variables: Anorm1 in HPL_pztest.c Residue Variables: resid0 in HPL_pztest.c Function return Data-type: double HPL_pzlange Mixed Precision Using HPL HPL updates upper triangular matrix only Need to update lower triangular matrix to prepare iterative refinement Find the global pivot vector from HPL and use it to swap the rows of the lower triangular matrix Update lower triangular (to be compatible with ScaLAPACK) Modified data structure code to assemble and return pivot vector Iterative Refinement A better performance is gained by using LU factorization in single precision (much faster than in double). Then using ScaLAPACK routines for triangular solve, perform matrix-vector multiply and iterative refinement to gain double precision. Solve the matrix −Using ‘ call hpl_psgesv(…) ’ to solve matrix by HPL −Instead of ‘ call psgetrf(…) ’ (ScaLAPACK) Major computational cost −Solve the matrix: O(N 3 ), N x N is the size of the matrix The numerical experiments were performed on the athena Cray XT4 supercomputer at the NICS. Athena nodes consist of a quad-core 2.3 GHz AMD Opteron processor with 4GB of memory. Using Streaming SIMD Extension (SSE), each core has peak performance of 9.2 Gflops (18.4 Gflops) in 64-bit (32-bit) arithmetic. Precision Processor Grid (P x Q) Size of matrix (N) GFLOPS% of peak HPLSCALAPACK (p?ludriver.f) HPL SCALAPACK (p?ludriver.f) SINGLE REAL s 128 x ,00018, , ,00056, , ,000108, , ,000174, ,200,000211, x 2 5, , DOUBLE REAL d 128 x ,00014, , ,00036, , ,00061, , ,00092, ,200,000106, x 2 5, , SINGLE COMPLEX c 128 x ,00048, , ,000103, , ,000150, , ,000211, x 2 5, , DOUBLE COMPLEX z 128 x ,00032, , ,00056, , ,00079, , x 2 5, , Processor Grid (P x Q) Size of matrix (N) REAL*8 ScaLAPACK LU, sec REAL*4 ScaLAPACK LU, sec REAL*4 HPL LU, sec Mixed Precision Solve, sec 128 x , , , , , x 32 80, x x x , x x Table 2: Performance of HPL Mixed Precision – REAL MATRIX Processor Grid (P x Q) Size of matrix (N) COMPLEX*16 ScaLAPACK LU, sec COMPLEX*8 ScaLAPACK LU, sec COMPLEX*8 HPL LU, sec Mixed Precision Solve, sec 128 x , , , , , x 32 80, x x x , x x Table 3: Performance of HPL Mixed Precision – COMPLEX MATRIX Results Data-type Change: Variable Data-type: double A → double complex A MPI communication Data-type: MPI_Send(A,…,MPI_DOUBLE,…) → MPI_Send(A,…,MPI_DOUBLE_COMPLEX,…) Function return type: double HPL_rand → double complex HPL_zrand For example, HPL_pdlange.c becomes HPL_pslange.c with the following change in content: Peak Rate of 16K nodes = TFLOPS (Double precision) Peak Rate of 16K nodes = TFLOPS (Single precision) Methodology To get three additional precisions rewrite original HPL source codes by modifying data types and function names using the following convention (same as ScaLAPACK) in naming files and functions: ‘s’ – stands for SINGLE REAL ‘d’ – stands for DOUBLE REAL ‘c’ – stands for SINGLE COMPLEX ‘z’ – stands for DOUBLE COMPLEX