1 Using the PETSc Parallel Software library in Developing MPP Software for Calculating Exact Cumulative Reaction Probabilities for Large Systems (M. Minkoff and A. Wagner) ANL (MCS/CHM) u Introduction u Problem Description u MPP Software Tools u Computations u Future Direction

2 Parallelization of Cumulative Reaction Probabilities (CRP) with PETSc M. Minkoff (ANL/MCS) and A. Wagner(ANL/CHM) Calculation of gas phase rate constants u Develop a highly scalable and efficient parallel algorithm for calculating the Cumulative Reaction Probability, P. u Use parallel subroutine libraries for higher generation parallel machines to develop parallel CRP simulation software. u Implementing Miller and Manthe (1994) method for time-independent solution of P in parallel. –P is determined for an eigenvalue problem with an operator involving two Green’s functions. The eigenvalues are obtained using a Lanczos method. The Green’s functions are evaluated via a GMRES iteration with diagonal preconditioner.

3 Benefits of using PETSc u Sparsity: PETSc allows arbitrarily sparse data structures u GMRES: PETSc has GMRES as an option for linear solves u Present tests involve problems in dimensions 3 to 6. Testing is underway using an SGI Power Challenge (ANL), and SGI/CRAY T3E (NERSC). (Portability is provided via MPI and PETSc, so higher dimensional systems are planned for future work).

4 Chemical Dynamics Theory 3 angles, 3 stretches 6 degrees of freedom

5 Chemical Dynamics Theory u How fast do chemicals react? u Rate constant “k” determines it –d[X] / dt = -k1[X][Y] + k2[Z][Y] –many rates at work in devices –rates express interactions in the chemistry –individual rates are measurable and calculable –rates depend on T, P.

6 Chemical Dynamics Theory N(E) = Tr[P(E)] u Rates are related to –Cumulative Reaction Probability (CRP), N(E) N(E) = 4 Tr[

7 Chemical Dynamics Theory u Probability Operator and It’s Inverse –Using probability method calculates a few large eigenvalues via iterative methods. The iterative evaluation involves the action of two Green’s function. –Using inverse probability method involves a direct calculation each iteration to obtain a few smallest eigenvalues. At each iteration the action of a vector by the Green’s function is required. This leads to solving linear systems involving the Hamiltonian.

8 Chemical Dynamics Theory u The Green’s functions have the form: G(E) = (E + i e - H) -1 and so we need to solve two linear systems (at each iteration) of the form: (E + i e - H) y = x where x is known. This system is solved via GMRES with preconditioning methods (initially diagonal scaling).

9 PETSc: Portable, Extensible Toolkit for Scientific Computing u Focus: data structures and routines for the scalable solution of PDE-based applications u Object-oriented design using mathematical abstractions u Freely available and supported research code  Available via u Usable in C, C++, and Fortran77/90 (with minor limitations in Fortran 77/90 due to their syntax) u Users manual, hyperlinked manual pages for all routines u Many tutorial-style examples  Support via Satish Balay, William Gropp, Lois McInnes, and Barry Smith MCS Division, Argonne National Laboratory

10 Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Profiling Interface PETSc PDE Application Codes Object-Oriented Matrices, Vectors, Indices Grid Management Linear Solvers Preconditioners + Krylov Methods Nonlinear Solvers, Unconstrained Minimization ODE Integrators Visualization Interface Application Codes Using PETSc Applications can interface to whatever abstraction level is most appropriate.

11 Compressed Sparse Row (AIJ) Blocked Compressed Sparse Row (BAIJ) Block Diagonal (BDIAG) DenseOther IndicesBlock IndicesStrideOther Index Sets Vectors Line SearchTrust Region Newton-based Methods Other Nonlinear Solvers Additive Schwartz Block Jacobi ILUICC LU (Sequential only) Others Preconditioners Euler Backward Euler Pseudo Time Stepping Other Time Steppers GMRESCGCGSBi-CG-STABTFQMRRichardsonChebychevOther Krylov Subspace Methods Matrices PETSc Numerical Components

MHz T3E, 2.8M vertices Sample Scalable Performance 3D incompressible Euler Tetrahedral grid Up to 11 million unknowns Based on a legacy NASA code, FUN3d, developed by W. K. Anderson Fully implicit steady-state Newton-Krylov-Schwarz algorithm with pseudo-transient continuation Results courtesy of Dinesh Kaushik and David Keyes, Old Dominion University

13 Computations via MPI and PETSc

14 5D/T3E Results for Varying Eigenvalue and G-S Method

15 Parallel Speedup 5D/6D ANL/SGI and NERSC/T3E

16 Storage Required for Higher Dimensions

17 Results and Future Work u Achieved parallelization with less effort –Suboptimal but perhaps 2X Optimal Performance u Testing for 6D and 7D underway. –MPP CPU and Memory can provide necessary resources –Many degrees of freedom can be approximated, so maximum dimension needed is ~10. u Develop block structured preconditioning methods