An Object-Oriented Software Framework for Building Parallel Navier-Stokes Solvers Xing Cai Hans Petter Langtangen Otto Munthe University of Oslo

Parallel CFD’99 Outline of the Talk Mathematical model for N-S equationsMathematical model for N-S equations Different numerical methodsDifferent numerical methods An O-O framework in DiffpackAn O-O framework in Diffpack ParallelizationParallelization A fast FEM N-S solverA fast FEM N-S solver A test caseA test case

Parallel CFD’99 Mathematical Model Incompressible Navier-Stokes equations:

Parallel CFD’99 Different Solution Methods Fully implicit methodsFully implicit methods Artificial compressibilityArtificial compressibility Operator splitting (discrete & continuous)Operator splitting (discrete & continuous) Solenoidal approachesSolenoidal approaches Pressure filteringPressure filtering Many choices and possibilities!

Parallel CFD’99 An Observation Most solution processes have sub-stepsMost solution processes have sub-steps Each sub-step solves one/several PDEsEach sub-step solves one/several PDEs Many PDEs in the sub-steps are of the same typeMany PDEs in the sub-steps are of the same type Re-usable simulatorsRe-usable simulators Small adjustments neededSmall adjustments needed

Parallel CFD’99 An O-O Framework Goal: Flexible and extensible simulatorsGoal: Flexible and extensible simulators Take advantages of OO programmingTake advantages of OO programming Unified implementation of different base PDE solversUnified implementation of different base PDE solvers Unified organization of common dataUnified organization of common data Separate administration of problem specific data (flow conditions)Separate administration of problem specific data (flow conditions)

Parallel CFD’99 Implementation Structure SomeCaseSomeSolver NsCaseNsBaseNsPrms Use of OOP only on high administration level Fortran construct and c-array at low-level

Parallel CFD’99 Diffpack O-O software environment for scientific computationO-O software environment for scientific computation Rich collection of PDE solution components - portable, flexible, extensibleRich collection of PDE solution components - portable, flexible, extensible H.P.Langtangen, Computational Partial Differential Equations, Springer 1999H.P.Langtangen, Computational Partial Differential Equations, Springer 1999

Parallel CFD’99 Straightforward Parallelization Develop a sequential simulator, without paying attention to parallelismDevelop a sequential simulator, without paying attention to parallelism Follow the Diffpack coding standardsFollow the Diffpack coding standards Use add-on libraries for parallelization specific functionalitiesUse add-on libraries for parallelization specific functionalities Add a few new statements for transformation to a parallel simulatorAdd a few new statements for transformation to a parallel simulator

Parallel CFD’99 Linear-algebra-level Approach Parallelize matrix/vector operationsParallelize matrix/vector operations –inner-product of two vectors –matrix-vector product –preconditioning - block contribution from subgrids Keeps original sequential Diffpack libraries almost intact needs inter-processor communication functionalities Keeps original sequential Diffpack libraries almost intact needs inter-processor communication functionalities Easy to useEasy to use –access to all existing Diffpack iterative methods, preconditioners and convergence monitors –need only to add a few lines of new code –arbitrary choice of number of procs at run-time

Parallel CFD’99 Work Load Distribution Through grid partitionThrough grid partition Need good load balanceNeed good load balance Flexibility & extensibilityFlexibility & extensibility –Global grid -> a set of subgrids Arbitrary number of procs determined at run-timeArbitrary number of procs determined at run-time Non-overlapping partitionNon-overlapping partition Controllable addition of overlap (if desired)Controllable addition of overlap (if desired) –An existing set of subgrids (input from files)

Parallel CFD’99 Partition example

Parallel CFD’99 An Add-on Parallelization Library Grid partition administrationGrid partition administration High-level inter-processor communication functionalities (hidden MPI)High-level inter-processor communication functionalities (hidden MPI) class GridPartAdmclass GridPartAdm –void GridPartAdm::prepareSubgrids() –void GridPartAdm::prepareCommunication() –void GridPartAdm::updateGlobalValues() –void GridPartAdm::matvec –void GridPartAdm::innerProd –void GridPartAdm::norm Fully portable (Origin 2000, IBM SP2, HP V )

Parallel CFD’99 A Simple Coding Example //... #ifdef PARALLEL_CODE adm->scan (menu); adm->prepareSubgrids (); adm->prepareCommunication (); lineq->attachCommAdm (*adm); #endif lineq->solve (); //... set subdomain list = DEFAULT set global grid = grid1.file set partition-algorithm = METIS set number of overlaps = 0

Parallel CFD’99 LAL Parallelization Example Highly unstructured grid Highly unstructured grid Discontinuity in the coefficient K Discontinuity in the coefficient K

Parallel CFD’99 Measurements for example one 130,561 degrees of freedom 130,561 degrees of freedom Overlapping subgrids Overlapping subgrids BiCGStab + (block) RILU prec. BiCGStab + (block) RILU prec.

Parallel CFD’99 A Fast FEM N-S Solver Operator splittingOperator splitting Calculation of an intermediate velocityCalculation of an intermediate velocity

Parallel CFD’99 A Fast FEM N-S Solver Solution of a Poisson EquationSolution of a Poisson Equation Correction of the intermediate velocityCorrection of the intermediate velocity

Parallel CFD’99 O-O Implementation NsBaseNsSolverPredictor PressureCorrectorNsCaseExplicit Case1 FEM Case2

Parallel CFD’99 Test Case: Vortex-Shedding

Parallel CFD’99 Simulation Snapshots Pressure

Parallel CFD’99 Simulation Snapshots Pressure

Parallel CFD’99 Animated Pressure Field

Parallel CFD’99 Simulation Snapshots Velocity

Parallel CFD’99 Simulation Snapshots Velocity

Parallel CFD’99 Animated Velocity Field

Parallel CFD’99 Some CPU-Measurements

Parallel CFD’99 Parallelization Approach 2: DD Parallelization of multilevel methodsParallelization of multilevel methods One subdomain is assigned with a sequential simulatorOne subdomain is assigned with a sequential simulator A generic implementation frameworkA generic implementation framework Systematic and flexibleSystematic and flexible –O-O programming enables extensive code reuse –Easy to incorporate multilevel algorithm modification –Different grid types, local solution methods etc. on different subdomains

Parallel CFD’99 Parallel DD Application One 2D pressure equation in reservoir simulation

Parallel CFD’99 2-phase porous media flow PEQ: SEQ: Multigrid V-cycle in subdomain solves

Parallel CFD’99 Nonlinear water waves Fully nonlinear 3D water waves Primary unknowns:

Parallel CFD’99 Parallel DD Application Two 3D Poisson equation in water wave simulation

Parallel CFD’99 Parallel efficiency Fixed number of subdomains M =16.Fixed number of subdomains M =16. Subdomain grids from partition of a global 41x41x41 grid.Subdomain grids from partition of a global 41x41x41 grid. Simulation over 32 time steps.Simulation over 32 time steps. DD as preconditioner of CG for the Laplace eq.DD as preconditioner of CG for the Laplace eq. Multigrid V-cycle as subdomain solver.Multigrid V-cycle as subdomain solver.

Parallel CFD’99 Summary O-O implementation framework for N-S solversO-O implementation framework for N-S solvers Flexible & user-friendly parallelization at the level of linear algebraFlexible & user-friendly parallelization at the level of linear algebra Parallelization at the level of DDParallelization at the level of DD