1 SciDAC TOPS PETSc Work SciDAC TOPS Developers Satish Balay Chris Buschelman Matt Knepley Barry Smith Hong Zhang Other PETSc Team Members Lois Curfman McInnes, Bill Gropp, Dinesh Kaushik

2 PETSc Portable, Extensible, Software for the scalable solution of nonlinear PDEs Focusing on algebraic solvers –Preconditioners, multigrid, Krylov methods, Newton, FAS, time integrators

3 Activities Derivative calculations – ADIC/ADIFOR –Matrix-free preconditioners –FAS (future) Accessing other software –HYPRE preconditioner usage –RAMG,SAMG –SuperLU –ESI SIDL/Babel/CCA – PETSc 3.0 plans SIDL/Babel MPI

4 Derivatives Automatic tools ADIC/ADIFOR are great but require manual processing (painful when developing code) Would like to have a variety of derivatives “for free” (or nearly) –Matrix element based Newton which requires explicit sparse Jacobian –Matrix-free Newton –SOR preconditioners and FAS require one equation at at time

5 To Fully Automate Code comments to indicate what needs to be processed Predefined calling sequences Libraries that can use generated code Make system automatically generates required derivative code

6.. /* Process adiC: FFL */ int FFL (DALocalInfo *info,Field **x, Field **f,void *ptr) { AppCtx *user = (AppCtx*)ptr;.. DMMGSetSNESLocal(dmmg,FFL,0,ad_FFL,admf_FFL); typedef struct { PassiveReal lidvelocity,prandtl,grashof; PetscTruth draw_contours; } AppCtx; typedef struct { double u,v,omega,temp; } Field;

7 > make BOPT=g_c++ ex19 Running adiC on function FFL Extracting structure Field Extracting structure AppCtx Extracting function FFL adiC -a -d gradient -i /sandbox/bsmith/petsc/bmake/adic.init ex19.c.tmp ADIC 1.1 beta 4 08/07/00 18:06:31 This software was created in the course of academic and/or research endeavors by Argonne National Laboratory. See the disclaimer in the g++ -fPIC -o ex19.o -c -Dad_GRAD_MAX=36 -I/sandbox/bsmith/petsc/include/adic -g -I/sandbox/bsmith/petsc..

8 Matrix-free Multiply Differencing (well understood) AD int mf_FFL(DALocalInfo *info,mf_Field **x, mf_Field **f,void *ptr) { typedef struct { double value;double grad[1]; } mfDERIV_TYPE; typedef struct { mfDERIV_TYPE u, v, omega, temp; }

9 Matrix-free Multigrid Preconditioners Krylov smoothing with –no preconditioner –(point block) Jacobi preconditioner –(point block) SOR preconditioning

10 Matrix-free Jacobi/SOR If function is provided an “(block) equation at a time” matrix-free multiply can be done an equation at a time – can be computed by providing a seed of –For SOR Similar computations with differencing

11 FAS – Full Approximation Scheme (Nonlinear Multigrid) Smoothing uses Newton to solve a single (block) equation at a time –Need the same “(block) equation at a time” and its derivative as for matrix-free Newton!

12 We are developing a common software infrastructure and user interface for many Newton methods and FAS which can use AD or differencing for derivatives.

13 Activities Derivative calculations – ADIC/ADIFOR –Matrix-free preconditioners –FAS (future) Accessing other software –HYPRE preconditioner usage –RAMG, SAMG –SuperLU –ESI SIDL/Babel/CCA – PETSc 3.0 plans SIDL/Babel MPI

14 Philosophy on Solvers Use any solver without changing the application code Add to application code to take advantage of special solver features (optimizations) –but additions cannot break other solvers PCILUSetDropTolerance(pc,1.e-5); PCSPAISetEpsilon(pc,1.e-2);

15 HYPRE-PETSc Matrix Element based preconditioners No vector copies Currently a matrix copy (plan to remove) -pc_type hypre -pc_hypre_type -pc_hypre_pilut_tol 1.e-4 -pc_hypre_parasails_thresh.1 -pc_hypre_parasails_filter.2

16 RAMG, SAMG – PETSc Sequential Preconditioners AMG – written by John Ruge, Klaus Stueben, and Rolf Hempel. A classic scalar PDE AMG code. SAMG – written by Klaus Stueben. A complete rewrite that supports systems of PDEs (interface coming soon). –PETSc interfaces provided by Domenico Lahaye

17 SuperLU_Dist MPI parallel LU factorization and solve written by Sherry Li and Jim Demmel Henry Tufo – Paul Fischer XYT fast direct parallel solver for “small problems”, i.e. the coarse grid problem in multigrid -mat_aij_superlu_dist -mat_aij_superlu_dist_colperm

18 ESI Equation Solver Interface A C++ “standard” of abstract classes for setting up and solving linear systems At least three implementations –Trilinos (Sandia, NM) –ISIS (Sandia, CA) –PETSc

19 PETSc ESI Goal Allow PETSc to use matrices and solvers written by others in ESI Allow anyone to use PETSc matrices and solvers from ESI –Achieved by using two sets of wrappers PETSc objects “look like” ESI objects ESI objects “look like” PETSc objects

20 -is_type esi –is_esi_type Petra_ESI_IndexSpace -vec_type esi -vec_esi_type Petra_ESI_Vector -mat_type esi –mat_esi_type Petra_ESI_CSR_Matrix PETSc using Trilinos objects ESI using PETSc objects esi::Vector *v = new esi::petsc::Vector (petscvector) esi::Operator *o = new esi::petsc::Matrix (petscmatrix)

21 Activities Derivative calculations – ADIC/ADIFOR –Matrix-free preconditioners –FAS (future) Accessing other software –HYPRE preconditioner usage –RAMG, SAMG –SuperLU –ESI SIDL/Babel/CCA – PETSc 3.0 plans SIDL/Babel MPI

22 SIDL/Babel (LLNL) Scientific Interface Definition Language Babel – generates language stubs for –C, C++, Fortran 77, Python, Java, … Provides –Object model –Exception (error) model Immature, but very exciting development

23 PETSc 3.0 SIDL/Babel for inter-language, object model and exception model Multiple precisions –Single for preconditioners? Vector and matrix sizes mutable –Adaptive mesh refinement Not a rewrite, just a rewrap!

24 Activities Derivative calculations – ADIC/ADIFOR –Matrix-free preconditioners –FAS (future) Accessing other software –HYPRE preconditioner usage –RAMG, SAMG –SuperLU –ESI SIDL/Babel/CCA – PETSc 3.0 plans SIDL/Babel MPI

25 Conclusions Many new tools and solvers Accessible via –PETSc 2.0 interface –ESI –(coming) SIDL/Babel/CCA We are happy to support users, especially those interested in nontrivial problems