Hernan G. Arango, Rutgers University Tal Ezer, Pricenton University FTP File: TOMS.tar A Community T errain-following O cean Modeling S ystem

YBennett et al. (FNMOC; OSU) YChassignet / Iskandarani et al. (RSMAS) YCornuelle / Miller (SIO) YGeyer (WHOI) YHetland (TAMU) YLermusiaux (Harvard) YMellor (Pricenton) YMoore (U. Colorado) YShchepetkin (UCLA) YSignell (SACLANT; USGS) COLLABORATORS

YChao / Song (JPL) YPreller / Martin (NRL) YNaval Operational Community YPOM Ocean Modeling Community YROMS / SCRUM Ocean Modeling Community OTHER COLLABORATORS

YTo design, develop and test an expert ocean modeling system for scientific and operational applications YTo support advanced data assimilation strategies YTo provide a platform for coupling with operational atmospheric models (like COAMPS) YTo support massive parallel computations YTo provide a common set of options for all coastal developers with a goal of defining an optimum coastal/relocatable model for the navy OBJECTIVES

YUse state-of-the-art advances in numerical techniques, subgrid-scale parameterizations, data assimilation, nesting, computational performance and parallelization YModular design with ROMS as a prototype YTest and evaluate the computational kernel and various algorithms and parameterizations YBuild a suite of test cases and application databases YProvide a web-based support to the user community and a linkage to primary developers APPROACH

“The complexity of physics, numerics, data assimilation, and hardware technology should be transparent to the expert and non-expert USER” CHALLENGE

YFree-surface, hydrostatic, primitive equation model YGeneralized, terrain-following vertical coordinates YBoundary-fitted, orthogonal curvilinear, horizontal coordinates on an Arakawa C-grid YNon-homogeneous time-stepping algorithm YAccurate discretization of the baroclinic pressure gradient term YHigh-order advection schemes YContinuous, monotonic reconstruction of vertical gradients to maintain high-order accuracy TOMS KERNEL ATTRIBUTES

Dispersive Properties of Advection  /2  /4 3  /4 kxkx 1/2 1 3/2 2 5/2 K(k)  x Parabolic Splines Vs Finite Centered Differences

YHorizontal mixing of tracers along level, geopotential, isopycnic surfaces YTransverse, isotropic stress tensor for momentum YLocal, Mellor-Yamada, level 2.5, closure scheme YNon-local, K-profile, surface and bottom closure scheme TOMS SUBGRID-SCALE PARAMETERIZATION

YAir-Sea interaction boundary layer from COARE (Fairall et al., 1996) YOceanic surface boundary layer (KPP; Large et al., 1994) YOceanic bottom boundary layer (inverted KPP; Durski et al., 2001) TOMS BOUNDARY LAYERS

YAir-Sea interaction boundary layer from COARE (Fairall et al., 1996) YOceanic surface boundary layer (KPP; Large et al., 1994) YOceanic bottom boundary layer (inverted KPP; Durski et al., 2001) TOMS BOUNDARY LAYERS YWave / Current / Sediment bed boundary layer (Styles and Glenn, 2000) YSediment transport

YLagrangian Drifters (Klinck, Hadfield) YTidal Forcing (Hetland, Signell) TOMS MODULES

Gulf of Maine M2 Tides Surface Elevation (m)

YLagrangian Drifters (Klinck, Hadfield) YTidal Forcing (Hetland, Signell) TOMS MODULES YRiver Runoff (Hetland, Signell, Geyer)

Distance (km) Depth (m) Salinity (PSS) Hudson River Estuary

YLagrangian Drifters (Klinck, Hadfield) YTidal Forcing (Hetland, Signell) Y River Runoff (Hetland, Signell, Geyer) TOMS MODULES YBiology Fasham-type Model (Moisan, Shchepetkin) YEcoSim Bio-Optical Model (Bissett)

YSystematic evaluation of numerical algorithms via robust test problems YData/Model comparisons YStudy optimal combination of algorithmic options for various coastal applications YDocumentation of testing procedures TOMS TESTING

YModular, efficient, and portable Fortran code (F77+, F90) YC-preprocessing managing YMultiple levels of nesting YLateral boundary conditions options for closed, periodic, and radiation YArbitrary number of tracers (active and passive) YInput and output NetCDF data structure YSupport for parallel execution on both shared- and distributed -memory architectures TOMS CODE DESIGN

YCoarse-grained parallelization TOMS PARALLEL DESIGN

} } Nx Ny PARALLEL TILE PARTITIONS 8 x 8

YCoarse-grained parallelization TOMS PARALLEL DESIGN YShared-memory, compiler depend directives MAIN (OpenMP standard) YDistributed-memory (MPI; SMS) YOptimized for cache-bound computers YZIG-ZAG cycling sequence of tile partitions YFew synchronization points (around 6) YSerial and Parallel I/O (via NetCDF) YEfficiency 4-64 threads

YNudging YOptimal Interpolation (OI) YTangent linear and Adjoint algorithms Y4D VARiational data assimilation (4DVAR) and Physical Statistical Analysis System (PSAS) algorithms YInverse Ocean Modeling System (IOMS) YEnsemble prediction platform based on singular value decomposition YError Subspace Statistical Estimation (ESSE) TOMS DATA ASSIMILATION

Historical, Synoptic, Future in Situ/Remote Field/Error Observations d 0 R 0 Field Initialization Central Forecast Sample Probability Density Mean Select Best Forecast Shooting ESSE Smoothing via Statistical Approximation Minimum Error Variance Within Error Subspace (Sequential processing of Observations) Measurement Model A Posteriori Residules dr (+) Performance/ Analysis Modules OA via ESSE Gridded Residules Synoptic Obs Measurement Model Measurement Error Covariance   ^  cf (-) ^     00 Options/ Assumptions Most Probable Forecast  mp (-) ^ Ensemble Mean  q {  j  ^ Adaptive Error Subspace Learning Convergence Criterion Continue/Stop Iteration Breeding Peripherals Analysis Modules Normalization SVD p Continuous Time Model Errors Q(t) Scalable Parallel Ensemble Forecast + PerturbationsError Subspace Initialization 1jq1jq    1jq1jq ^ ^ ^ u j (o,Ip) with physical constraints +  (+) ^ E(+)  (+) E0E0  (+) ^ Ea(+)a(+)Ea(+)a(+) Field Operation Assumption Key  (-) ^ E(-)  (-) d  C  (-) Data Residuals ^ /- + j=1 j=q + ESSE Flow Diagram

YDensity Jacobian Class (Blumberg and Mellor, 1987; Song 1998; Song and Wright 1998) u More Accurate u Error vanishes with linear density profiles YPressure Jacobian Class (Lin 1998; Shchepetkin and McWilliams, 2001) u JEBAR consistent u Conserve Energy PRESSURE GRADIENT FORCE

Seamount Test Case (64 x 64 x 20) dx = dy = 8 km

Models with 2nd order advection scheme POM ROMS Second Order Advection Scheme Surface Elevation Anomaly Stream Function Anomaly

Advection Schemes in ROMS (Seamount Case) V Second Order Centered Third Order Upstream Bias Fourth Order Centered

Pressure Gradient Errors POM (6th order) U (cm/s) V (cm/s) ROMS POM X (km)

Relative CPU per time step Percentage

XBuild TOMS from ROMS prototype uMellor-Yamada, level 2.5 uPassive and active open boundary conditions uTidal forcing uRiver runoff uLagrangian drifters uData assimilation XInter-comparison between POM and ROMS uEvaluation of time-stepping, advection, and pressure gradient algorithms XInitial development of TOMS web site RESULTS (YEAR 1)

Initial web page:

YTo Be Determined !!! YPotential Users: uNAVO uFNMOC uNOAA uUSCG TRANSITION PATHS

XChassignet et al., 2000: Damee modeling review XEzer, 2000: Mixed-layer evaluation XEzer and Mellor, 2000: POM Damee application XHaidvogel et al., 2000: ROMS Damee application XMalanotte-Rizzoli et al., 2000: ROMS Damee XMellor, 2001: Improved turbulence scheme XMellor et al., 2001: Generalized vertical coordinate PUBLICATIONS