Parallel Computing Activities at the Group of Scientific Software Xing Cai Department of Informatics University of Oslo

Numerical Simulation Phy.phenomMath.model Diffpack hardware Algorithm

Diffpack O-O software environment for scientific computation (C++)O-O software environment for scientific computation (C++) Rich collection of PDE solution components - portable, flexible, extensibleRich collection of PDE solution components - portable, flexible, extensible H.P.Langtangen, Computational Partial Differential Equations, Springer 1999H.P.Langtangen, Computational Partial Differential Equations, Springer 1999

Parallelization Objectives Flexible and user-friendly parallelizationFlexible and user-friendly parallelization High parallel efficiencyHigh parallel efficiency Full code portability (standard MPI)Full code portability (standard MPI) –SGI Cray Origin 2000, SGI Power Challenge –HP V2500 –IBM SP2 –Scali –Cluster of Linux PC nodes

Straightforward Parallelization Develop a sequential simulator, without paying attention to parallelismDevelop a sequential simulator, without paying attention to parallelism Use add-on libraries for parallelization specific functionalitiesUse add-on libraries for parallelization specific functionalities Add a few new statements for transformation to a parallel simulatorAdd a few new statements for transformation to a parallel simulator

Linear-algebra-level (LAL) Approach Parallelize matrix/vector operationsParallelize matrix/vector operations –local sequential operations + communication –inner-product of two vectors –matrix-vector product –preconditioning - block contribution from subgrids Keeps original sequential Diffpack libraries almost intact Keeps original sequential Diffpack libraries almost intact Needs inter-processor communication functionality Needs inter-processor communication functionality Easy to useEasy to use –access to all existing Diffpack iterative methods, preconditioners and convergence monitors –need only to add a few lines of new code –arbitrary choice of number of procs at run-time

Work Load Distribution Through grid partition -> sub-matrix, sub-vector, no global matrix/vector!Through grid partition -> sub-matrix, sub-vector, no global matrix/vector! Need good load balanceNeed good load balance Flexibility & extensibilityFlexibility & extensibility –Global grid -> a set of subgrids Arbitrary number of procs determined at run-timeArbitrary number of procs determined at run-time Non-overlapping partitionNon-overlapping partition Controllable addition of overlap (if desired)Controllable addition of overlap (if desired) –An existing set of subgrids (input from files)

An Add-on Parallelization Library Grid partition administrationGrid partition administration High-level inter-processor communication routines (hidden MPI, OpenMP?)High-level inter-processor communication routines (hidden MPI, OpenMP?) class GridPartAdmclass GridPartAdm –void GridPartAdm::prepareSubgrids() –void GridPartAdm::prepareCommunication() –void GridPartAdm::updateGlobalValues() –void GridPartAdm::matvec –void GridPartAdm::innerProd –void GridPartAdm::norm Fully portable (Origin 2000, IBM SP2, HP V )

A Simple Coding Example //... #ifdef PARALLEL_CODE adm->scan (menu); adm->prepareSubgrids (); adm->prepareCommunication (); lineq->attachCommAdm (*adm); #endif lineq->solve (); //... set subdomain list = DEFAULT set global grid = whole_grid.file set partition-algorithm = METIS set number of overlaps = 0

LAL Approach - Measurements 2D Poisson Equation on unit square

Parallel Vortex-Shedding Simulation

Simulation Snapshots Pressure

Electrical potential depolarization in human heart

Parallelization Approach 2 Inspired by overlapping Schwarz methodsInspired by overlapping Schwarz methods Multilevel methods & parallel computingMultilevel methods & parallel computing Simulator-Parallel: One subdomain is assigned with a sequential simulatorSimulator-Parallel: One subdomain is assigned with a sequential simulator A generic framework: add-on library No.2A generic framework: add-on library No.2 Systematic and flexibleSystematic and flexible –O-O programming enables extensive code reuse –Easy to incorporate multilevel algorithmic modification –Different grid types, local solution methods etc. on different subdomains Parallelization at a higher level!Parallelization at a higher level!

Generic Programming SubdomainSimulator SubdomainFEMSolver OldSimulator NewSimulator Incorporation of a subdomain solver

SP Approach - Measurements 2D Poisson Equation on unit square Fixed number of subdomains Overlap between subdomains Superior numerical efficiency!

SP Approach - Unstructured Grids Highly unstructured grid Highly unstructured grid Discontinuity in the coefficient K Discontinuity in the coefficient K Run multigrid in parallel Run multigrid in parallel

SP Approach - Measurements I HP V2500

SP Approach - Measurements II SGI Cray Origin 2000

SP Application - System of PDEs 2D pressure equation in reservoir simulation

SP Application - System of PDEs 3D Poisson equation in water wave simulation

Parallel Efficiency Fixed number of subdomains M =16.Fixed number of subdomains M =16. Subdomain grids from partition of a global 41x41x41 grid.Subdomain grids from partition of a global 41x41x41 grid. DD as preconditioner of CG for the Laplace eq.DD as preconditioner of CG for the Laplace eq. Multigrid V-cycle as subdomain solver.Multigrid V-cycle as subdomain solver.