1. Coupler Design Issues from the Modular Ocean Data Assimilation Project Dr. Richard Loft Computational Science Section Scientific Computing Division National Center for Atmospheric Research Boulder, CO USA

2. Outline Project Introduction –goals / pariticipants / scope –http//:iom.asu.edu IOM: weak variational data assimilation –Inputs/outputs –Data components –Functional components IOM Coupling design –Strategy –Details –Role of Domain Specific Languages (DSL)

3. Modular Ocean Data Assimilation NSF ITR/AP Focus on the OSU Inverse Ocean Model (IOM) system for ocean data assimilation. –Variational data assimilation system –Weak assimilation –Iterative algorithm for solving nonlinear assimilation problems –Suite of diagnostics (posterior error statistics) –20 years in development by Bennett, et al.

4. Objectives of MODA Project Enhance the IOM System with modern information technology. –Modular software design –Hybrid parallel implementation –Coupled model strategy –Automated code generation Distribute it to the ocean modeling community. –5 ocean modeling partners Facilitate application of the system to coastal oceans, ocean basins and the global ocean.

5. NSF ITR MODA Collaboration Arizona State U. Muccino U. Colorado Moore NCAR Loft NCSA Baker U. North Carolina Leuttich Oregon State U Bennett, Egbert, Erwig Rutgers U. Arango, Haidvogel UCSD ( Scripps) Cornuelle, Miller InstitutionResearcher

6. IOM Team Members IOM-PEZ Parallelism –PI: Rich Loft, NCAR –Component: parallel super and Infra-structure (hybrid F90 module framework) Domain Specific Languages –PIs: Martin Erwig & Zhu Fu, OSU –Component: automated code generator (Haskell) Visualization –Pis: Polly Baker, NCSA/IU –Component: VisBenchTool Visualization Software

7. Participating Model Teams PEZ (Primitive Eq. Z-coordinate) model –Description: 3D, free-surface, z-coordinate Bryan-Cox –Grid: spherical polar, “B” grid –Language: F90 –Parallelism: SPMD, MPI ROMS (Regional Ocean Model System) –Description: 3D, free-surface, S-coordinate –Grid: Horizontal orthogonal curvilinear coordinate, “C” grid –Language: F90 –Parallelism: SPMD, OpenMP, MPI, SMS

8. Participating Model Teams SEOM (Spectral Element Ocean Model) –PI: Dale Haidvogel, Rutgers –Description: 3D, free-surface, S-coordinate –Grid: h-p finite element, quadrilateral element –Language: F90 –Parallelism: SPMD, MPI ADCIRC (Advanced Circulation) model –PIs: Julia Muccino, ASU; Rich Luettich, UNC –Description: 3D, free-surface, sigma- coordinate –Grid: finite element, linear triangular element Language: F90 –Parallelism: SPMD, MPI

9. Participating Model Teams OTIS (Internal Tides) –PI: Gary Egbert, OSU –Description: Laplace tidal equations plus 10 years of TOPEX/Poseidon –Description: Solid Earth mageto-tellurics (magnetic field of earth’s crust)

10. observing system –e.g. measured sea surface temperatures, isotherm depths and surface winds dynamics –e.g. the hydrostatic primitive equations hypothesis concerning the error –covariances of errors in the initial conditions boundary conditions and forcing estimator –space-time integrated weighted sum of squared errors optimization algorithm –iterative indirect representer algorithm Data Assimilation Checklist: Inputs

11. state estimate –hindcast over the period in the data collection –space-time fields for state variables data residuals, dynamic residuals –Space-time minimum residuals posterior error statistics –space-time covariances test statistics –chi square internal consistency tests model improvements –e.g. suggested from the distribution of errors observing system assessment Data Assimilation Checklist: Outputs

12. Inverse system data components… Vector of data to be assimilated Trajectories –multi-variate physical space-time fields –generated by tangent linear/adjoint model –user prescribed Parameters –covariance matrices –user prescribed

13. Inverse system functional components… Pure data space components –iterative solver Physical space components –tangent linear/ adjoint models Components that map between the two spaces. –measurement operator –impulse operator Looks like a coupled system

14. Coupler macro design: IOM + coupler + ocean models Fwd & adjoint Ocean model IOM Ocean Models: PEZ ROMS SEOM ADCIRC OTIS Physical space Data space Data Assimilations : IOM core Data Coupler State Both spaces Impulses / measurements: DSL autogeneration Control info grids

15. Data / Physical Space Interactions Ocean Model Time Integration (physical space) IOM Iterative Solve (data space) model PEs IOM PEs Interpolations

16. IOM System Pseudo-code IOM(){ read d first( ; uF) h = d - uF solve(h ;  ) final(  ; ) } d = data uF = meas. first guess h = misfit  = representer coefficients Key: Physical Space / user supplied Both / Auto-generated Data Space/IOM supplied Subroutine Notation: foo( in ; out ) Pure Data Space… ^ ^ ^

17. IOM First Guess Code first(;uF){ read F tanlin( F(x,t) ; UF(x,t) ) measure(UF(x,t) ; uF) } F(x,t) = Initial / boundary / interior forcing UF(x,t) = response to F uF = measured first guess

18. IOM Inner Solver Code solve(h ;  ){  = h while (  ≠  ){ comb(  ; D(x,t) ) adjoint( D(x,t) ;  (x,t) ) convolve(  (x,t) ;  (x,t) ) tanlin(  (x,t) ;  (x,t) ) measure(  (x,t) ;  ) stabilize( ,  ;  ) call precongrad(  ;  ) }  =  }  (x,t) = adjoint  (x,t) = forward D(x,t) = Dirac comb  = representer coefs  = measured   = scratch  = scratch ^ ^ ^

19. IOM Final Sweep final(  ; ){ comb(  ; D(x,t) ) adjoint( D(x,t) ;  (x,t) ) convolve(  (x,t) ;  (x,t) ) tanlin(  (x,t) + F(x,t); U(x,t) ) write U(x,t) } D(x,t) = Dirac comb U(x,t) = optimal estimate  (x,t) = optimal adjoint ^ ^ ^ ^ ^

20. IOM System Architecture Parallel Infrastructure IOM Solver IOM Coupling Layer Tangent Linear Model DSL Generated Code External libraries: MPI, NetCDF, … Adjoint Model

21. IOM Parallel Module Support parallel F90 Module Heirarchy: processthread virtual topology PEZ 9pt stencil IOM broadcast stencils buffers Solver global sums

22. IOM Coupling Design Details Key object: the observation –Simplest case: a point observation –is a distribution:  ( - ',  -  ', t-t' ) –In general smeared out over space-time. The observation (self describing) – F90 derived type – type of observation – units Associated methods must be supplied for mapping observations onto gridded state variables.

23. IOM Coupling Design Details Fortran/Unix specific (applications / code generator) Preferred mode: separate executables. –IOM + coupler + ocean models (constant functions) –IOM-PEZ components currently merged into one executable IOM supports parallel components (MPI/OMP/hybrid) Fixed number of processors –MPI_SPAWN and MPI_SPAWN_MULTIPLE not supported on many major platforms (e.g. IBM) ). Coupled component execution does not overlap. –Strategy adopted to enable easy interfacing with diverse ocean codes, with different internal forms of parallelism. Solver checkpoint/restarts IOM system, no internal state saved on TLM/adjoint model side.

24. Automatic Generation of Model- Specific IOM Tools Martin Erwin and Zhe Fu Oregon State University Idea: Ocean modelers: select simulation tools provide parameters for their models obtain a customized variational system Specify ocean modeling tools once Identify model-dependent parameters Generate Fortran programs from specification and values for parameters

25. IOM DSL System tool(p 1,...,p k ): obj[n] =...... Tool SpecificationModel Configurations p 1 =...... p k =... Compiler Fortran Code p 1 =...... p k =... module tool_ROMS... PEZ ROMS p 1 =...... p k =... module tool_PEZ...

26. IOM-DSL-Modeler Interactions tool(p 1,...,p k ): obj[n] =...... DSL Configuration p 1 =...... p k =... DSL-Compiler Fortran Computer Scientists Ocean Modelers IOM Developers

27. Questions?