Overture and MetaChoas for CISM Developers M. Wiltberger With thanks to Bill Henshaw and Alan Sussman

What is Overture? A Collection of C++ Classes that can be used to solve PDEs on overlapping grids Key Features High level interface for PDEs on adaptive and curvilinear grids Provides a library of finite differences operators Conservative/NonConservative 2 nd and 4 Th order Uses A++/P++ array class for serial parallel array operations Extensive grid generation capablities

Overture: A toolkit for solving PDEs Mappings (geometry) MappedGrid GridCollection MappedGridFunction GridCollectionFunction Operators div, grad, bc's Grid Generator Ogen Adaptive Mesh Refinement Solvers Oges, Ogmg, OverBlown A++P++ array class Data base (HDF) Graphics (OpenGL) Boxlib (LBL)

A++ Array Class Serial and Parallel Array operations Easily passed to Fortran Just recompile with different library for parallel code using MPI calls do j=1,8 do i=1,8 u(i,j)=0.25*(v(i+1,j)+v(i-1,j) +v(i,j+1+v(i,j+1) enddo enddo Fortran realArray u(10,10), v(10,10); Range I(8,8), J(8,8); v = 1; u(I,J) =0.25*(v(I+1,J)+v(I-1,J)+ v(I,J+1)+v(I,J-1)); C++

realArray x(10,10), u(10,10); Range I(1,8), J(1,8);... where( x(I,J) >.5 ) u(I,J) = sin( 2.*Pi*x(I,J) ); do j=1,8 do i=1,8 if( x(i,j).gt. 0.5 )then u(i,j) = sin( 2.*pi*x(i,j) ) end if end do C++ Fortran Using the where statement in A++

RealArray u(10), v(3); intArray ia(3); ia(0)=2; ia(1)=4; ia(2)=7; Range I(0,2);... v(I)=u(ia)+2.*u(ia+1); do i=0,2 v(i)=u(ia(i))+2.*u(ia(i)+1); end do C++ Fortran Indirect addressing with A++

Passing A++ arrays to Fortran C++ Fortran extern "C" { void myfortran_( int & m, int & n, real & u); } realArray u(20,10); myfortran_( u.getLength(0), u.getLength(1), *u.getDataPointer() ); subroutine myfortran( m,n,u ) real u(m,n)... end

A Mapping defines a continuous transformation x r Mapping domainDimension rangeDimension map( r,x,xr ) inverseMap( x,r,rx ) boundaryConditions singularities... SquareMapping AnnulusMapping SphereMapping HyperbolicMapping EllipticTransform MatrixMapping RevolutionMapping etc. unit square Each mapping has an optimized map and inverseMap map

A MappedGrid holds the grid points and other geometry info MappedGrid gridIndexRange Mapping vertex vertexDerivative cellVolume faceNormal... Mapping that defines the geometry grid points jacobian derivatives optionally computed geometry arrays Figure here

A GridCollection holds a list of MappedGrids GridCollection numberOfGrids operator [int g] refinementLevel[int l]... access to a MappedGrid, g=0,1,.. GridCollection for a refinement level base grids refinement grids

A MappedGridFunction holds solution values realMappedGridFunction numberOfComponents MappedGrid MappedGridOperators x,y,z,xx,... derivatives derived from an A++ realArray which implies it inherits all A++ operators scalar, vector, 2-tensor,.. Grid functions can be vertex-centered, cell-centered, face-centered etc. MappedGrid mg(mapping); realMappedGridFunction u(mg); u=1.;

A GridCollectionFunction is a list of MappedGridFunctions GridCollection gc(...); Range all; realGridCollectionFunction u(gc,all,all,all,2); u=1.; for( int grid=0; grid<gc.numberOfGrids(); grid++ ) u[grid]=1.; realGridCollectionFunction realMappedGridFunction

Operators The Operator classes define differential operators and boundary conditions. - 2nd/4th order approximations plus some conservative approximations MappedGrid mg(sphere); MappedGridOperators op(mg); floatMappedGridFunction u(mg), v(mg), w; v=u.x()+u.y(); w=u.laplacianCoefficients(); Compute the derivatives directly Form the sparse matrix for the differential operator

Overture can be used at a number of levels GridCollection gc(...); realGridCollectionFunction u(gc), v(gc); GridCollectionOperators op(gc); v = u.x(); for( int grid=0; grid<gc.numberOfGrids(); grid++ ) v[grid]=u[grid].x(); for( grid=0; grid<gc.numberOfGrids(); grid++ ) xFortran_( *v[grid].getDataPointer(), *u[grid].getDataPointer(),...); Operate at the collection level Call Fortran Operate at the single grid level

A sample Overture program int main() { CompositeGrid cg; // Create a Composite grid getFromADataBaseFile(cg,”mygrid.hdf”); floatCompositeGridFunction u(cg); u = 1.0; // Set initial conditions CompositeGridOperators og(cg); // Create Operators u.setOperators(op); float t=0, dt=0.005, a=1., b=1., nu=0.1; for(int step=0;step<200;step++) { u+=dt*(-a*u.x()-b*u.y()+nu*(u.xx()+u.yy()); t+=dt; u.interpolate(); // Exchange info between grids u.applyBoundaryConditions(dircichelt,allBoundaries); u.finishBoundaryConditions(); } return 0; } Solve u t +au x + bu y - (u xx +u yy ) on overlaping grid

Wave Equation Solver The Overture primer contains a short code showing how to write a solver for the wave equation. This example shows how a wave is able to pass through a grid interface with few artificial reflections.

Incompressible flow past two cylinders Fine grids are integrated implicitly, coarse grids explicitly.

What is MetaChoas? A runtime meta-library that achieves direct data transfers between data structures managed by different parallel libraries Runtime meta-library means that it interacts with data parallel libraries and languages used for separate applications (including MPI) Can exchange data between separate (sequential or parallel) applications Manage data transfers between different libraries in the same application This often referred to as the MxN problem in parallel programming

How does MetaChaos work? It starts with the Data Descriptor Information about how the data is distributed across the processors Usually supplied by the library/program developer Sussman et al. are working on generalizing this process MetaChaos then uses a linearization ( L S A ) of the data to be moved (the regions) to determine the optimal method to move data from set of regions in A ( S A ) to a set of regions in B ( S B ) Moving the data is a three step process L S A = l libX (S A ) L S B = L S A S B = l -1 libY (L S B ) Only constraint on this operation is each region must have the same number of elements

A Simple Example: Heat Eq Using P++ #include main(int argc, char **argv) { Optimization_Manager::Initialize_Virtual_Machine("",iNPEs,argc,argv); int i,j,iTimeStep, iNumX, iNumT; Index I(1,iNumX-1); doubleArray daSolution(iNumX+1,iNumT+1),daTemp(iNumX+1,iNumT+1); daSolution = 0.0; // Setup BC via scalar indexing for(i=1;i<=iNumX/2;i++) daSolution(i,0)=2*i*dDX; for(i=iNumX/2+1;i<=iNumX-1;i++)daSolution(i,0)=2*(1-i*dDX); // Solve Heat Eq for(int iStep=0;iStep<9;iStep++) { daSolution(I,iStep+1) = dR*(daSolution(I+1,iStep) 2*daSolution(I,iStep)+daSolution(I-1,iStep))+ daSolution(I,iStep); } Optimization_Manager::Exit_Virtual_Machine(); }

Split in two Using MetaChoas #include main(int argc, char **argv) { Optimization_Manager::Initialize_Virtual_Machine("",NPES,argc,argv); Index I(1,iNumX-1); this_pgme = InitPgme(pgme_name,NPES); other_pgme = WaitPgme(other_pgme_name,NPES_other); Sync2Pgme(this_pgme,other_pgme); doubleArray daSolution(iNumX+1,iNumT+1),daTemp(iNumX+1,iNumT+1); daSolution = 0.0; BP_Set = Alloc_setOfRegion(); // Solve Heat Eq for(int iStep=0;iStep<9;iStep++) { daSolution(I,iStep+1) = dR*(daSolution(I+1,iStep) 2*daSolution(I,iStep)+daSolution(I-1,iStep))+ daSolution(I,iStep); } Optimization_Manager::Exit_Virtual_Machine(); }

A Simple Example: Wave Eq Using P++ #include main(int argc, char **argv) { Optimization_Manager::Initialize_Virtual_Machine("",iNPES,argc,argv); doubleArray daUnm1(iNumX+2,iNumY+2),daUn(iNumX+2,iNumY+2); doubleArray daUnp1(iNumX+2,iNumY+2); Index I(1,iNumX), J(1,iNumY); // Indices for computational domain for(j=1;j<iNumY+1;j++) { daUnm1(I,j) = sin(dW*dTime + (daX(I)*2*dPi)/dLenX); daUn(If,j) = sin(dW*0 + (daX(If)*2*dPi)/dLenX); } // Apply BC Omitteed for space // Evole a step forward in time for(i=1;i<iNSteps;i++) { daUnp1(I,J) = ((dC*dC*dDT*dDT)/(dDX*dDX))* (daUn(I-1,J)-2*daUn(I,J)+daUn(I+1,J)) + 2*daUn(I,J) - daUnm1(I,J); // Apply BC Omitted for space } Optimization_Manager::Exit_Virtual_Machine(); }

Split into two using MetaChaos #include main(int argc, char **argv) { Optimization_Manager::Initialize_Virtual_Machine("",NPES,argc,argv); this_pgme = InitPgme(pgme_name,NPES); other_pgme = WaitPgme(other_pgme_name,NPES_other); Sync2Pgme(this_pgme,other_pgme); BP_set = Alloc_setOfRegion(); left[0] = 4; right[0] = 4; stride[0] = 1; left[1] = 5; right[1] = 5; stride[0] = 1; reg = Alloc_R_HDF(DIM,left,right,stride); Add_Region_setOfRegion(reg,BP_Set); BP_da = getPartiDescriptor(&daUn); sched = ComputeScheduleForSender(…,BP_da,BP_set,…); for(i=1;i<iNSteps;i++) { daUnp1(I,J) = ((dC*dC*dDT*dDT)/(dDX*dDX))* (daUn(I-1,J)-2*daUn(I,J)+daUn(I+1,J)) + 2*daUn(I,J) - daUnm1(I,J); iDataMoveSend(other_pgme,sched,daUn,getLocalArray().getDataPointer); iDataMoveRecv(other_pgme,sched,daUn,getLocalArray().getDataPointer); Sync2Pgme(this_pgme,other_pgme); } Optimization_Manager::Exit_Virtual_Machine(); }

How CISM will use Overture and MetaChoas