ROMS/TOMS European Workshop Alcala de Henares, Spain, November 7, 2006 ROMS Framework and Algorithms Andrew M. Moore UCSC Emanuele Di Lorenzo Georgia Tech Bruce D. Cornuelle SIO, UCSD Arthur J. Miller SIO, UCSD Hernan G. Arango IMCS, Rutgers John L. Wilkin IMCS, Rutgers Javier Zavala-Garay IMCS, Rutgers

The Good…The Bad…The Ugly… F90 Adjoint Nesting Adjoint Maintenance Data Assimilation Infrequent Releases Copyright / Open Source Adjoint Parallelization Grid Generation ROMS Code Divergence Released Version 3.0 Open Boundaries Compiler Bugs WikiROMSDocumentationDocumentation Forum Activity Version Control Wetting and Drying ROMS Blog Post-processing Treatment of Rivers Conclusions

ROMS Framework

ROMS Directory Tree src Bin Adjoint Drivers External Include Modules Nonlinear Obsolete Programs SeaIce Utility Tangent Representer Version Compilers Lib makefile ROMS Master SWAN WRF

ROMS Drivers master.F #include “cppdefs.h” #if defined AIR_OCEAN # include "air_ocean.h" #elif defined WAVES_OCEAN # include "waves_ocean.h" #else # include "ocean.h" #endif ocean_control.F #include “cppdefs.h” #if defined AD_SENSITIVITY # include "adsen_ocean.h" #elif defined AFT_EIGENMODES # include "afte_ocean.h" #elif defined FT_EIGENMODES # include "fte_ocean.h" #elif defined FORCING_SV # include "fsv_ocean.h" #elif defined OPT_PERTURBATION # include "op_ocean.h" #elif defined OPT_OBSERVATIONS # include "optobs_ocean.h" #elif defined SO_SEMI # include "so_semi_ocean.h" #elif defined S4DVAR # include "s4dvar_ocean.h" #elif defined IS4DVAR # include "is4dvar_ocean.h" #elif defined W4DPSAS # include "w4dpsas_ocean.h" #elif defined W4DVAR # include "w4dvar_ocean.h" #else # if defined TLM_DRIVER # include "tl_ocean.h" # elif defined RPM_DRIVER # include "rp_ocean.h" # elif defined ADM_DRIVER # include "ad_ocean.h" # else # include "nl_ocean.h" # endif ocean.h # include “cppdefs.h” PROGRAM ocean USE ocean_control_mod, ONLY : initialize USE ocean_control_mod, ONLY : run USE ocean_control_mod, ONLY : finalize #ifdef DISTRIBUTE && defined MPI CALL mpi_init (MyError) CALL mpi_comm_rank (MPI_COMM_WORLD, MyRank, MyError) #endif CALL initialize CALL run CALL finalize #if defined DISTRIBUTE && defined MPI CALL mpi_finalize (MyError) #endif END PROGRAM ocean

ROMS Adjoint The Adjoint Model (ADM) of ROMS is exact and defined relative to the L2-norm inner-product Hand-written using the recipe of Giering and Kaminski (1998) Two Tangent Linear Models:  Perturbation Tangent Linear Model (TLM): Generalized Stability Theory (GST) Analyses, Strong and Weak Constraint 4DVar  Finite Amplitude Tangent Linear Model (RPM): Indirect Representers, Weak Constraint 4DVar The TLM is derived by linearizing the Nonlinear Model (NLM) around a small perturbation The RPM model is derived from the TLM by adding additional terms

Parallelization Coarse-grained parallelization: horizontal tiles The NLM, TLM, and RPM can be run in either shared-memory (OpenMP) or distributed-memory (MPI) The ADM can only be run in distributed-memory (ADM violates shared-memory collision rules) Aggregation of variables for MPI communications CALL ad_mp_exchange2d (ng, iADM, 3, Istr, Iend, Jstr, Jend, & & LBi, UBi, LBj, UBj, & & NghostPoints, EWperiodic, NSperiodic, & & ad_Zt_avg1, ad_DU_avg1, ad_DV_avg1)

Some Uses of Adjoint Models Data Assimilation: fit model solutions to data by adjusting initial conditions, boundary conditions and parameters. Sensitivity Analysis: study the response of the ocean circulation to variations in all physical attributes of the system Eigenmode Analysis: dynamic modes of variability (TLM normal modes, ADM optimal excitations) Singular Vectors: stability of the dynamical system (most rapidly growing perturbations) Stochastic Optimals: most disruptive patterns of ocean forcing Ensemble Prediction: initial condition perturbations along the most unstable directions of the state space Adaptive Sampling: design of optimal observational systems

4D Variational Data Assimilation (4DVAR) Strong Constraint   Conventional (S4DVAR): outer loop, NLM, ADM   Incremental (IS4DVAR): inner and outer loops, NLM, TLM, ADM (Courtier et al., 1994) Weak Constraint   Indirect Representer Method (W4DVAR): inner and outer loops, NLM, TLM, RPM, ADM (Egbert et al., 1994; Bennett et al, 1997)   Physical Space Statistical Analysis (W4DPSAS): inner and outer loops, NLM, TLM, ADM (Courtier, 1997)

Strong Constraint, Incremental 4DVAR Let’s introduce a new minimization variable  v, such that: J(  v k ) = ½(  v k ) T  v k + ½(H  x k – d k-1 ) T O -1 (H  x k – d k-1 )  v J =  v k + B T/2 H T O -1 (H  x k – d k-1 ) =  v k + B T/2  x J o =  v k + W -1/2 L T/2 GS B = SCS = S(GL 1/2 W -1/2 )(W -1/2 L T/2 G)S  x k = B 1/2  v k + x k-1 – x b  x k = B -1/2 (  x k + x k-1 – x b ) yielding The gradient of J in minimization-space, denoted  v J, is given by: The background-error covariance matrix can be factored as: where S is the background-error standard deviations, C is the background-error correlations which can be factorized as C = C 1/2 C T/2, G is the normalization matrix which ensures that the diagonal elements of C are equal to unity, L is a 3D self-adjoint filtering operator, and W is the grid cell area or volume.

Model/Background Error Covariance, B Use a generalized diffusion squared-root operator (symmetric) as in Weaver et al. (2003): B = S C S = S (G L 1/2 W -1/2 ) (W -1/2 L T/2 G) The normalization matrix, G, ensure that the diagonal elements of the correlation matrix, C, are equal to unity. They are computed using the exact (expensive) or randomization (cheaper) methods. The spatial convolution of the self-adjoint filtering operator, L 1/2, is split in horizontal and vertical components and discretized both explicitly and implicitly. The model/background standard deviation matrix, S, is computed from long (monthly, seasonal) simulations. The grid cell area or volume matrix, W -1/2, is assumed to be time invariant.

Strong Constraint, Incremental 4DVAR Misfit cost function between model (NLM+TLM) and observations Cost function gradient Compute TLM initial conditions using first guess conjugate gradient step size Compute change in cost function Compute TLM initial conditions Using refined conjugate gradient step size Compute NLM new initial conditions (NLM+TLM) Compute basic state trajectory and extract model at observations locations

Model/Background Error Correlation (C) Horizontal Hdecay = 100 km Vdecay = 100 m Vertical (implicit)

Model/Background Error Correlation Normalization Coefficients (G) SSHTemperature Bottom Level EAC

Generalized Stability Theory (GST) Dynamics/sensitivity/stability of flow to naturally occurring perturbations Dynamics/sensitivity/stability due to error or uncertainties in the forecast system Practical applications:   Ensemble prediction   Adaptive observations   Array design...

Optimal Perturbations A measure of the fastest growing of all possible perturbations over a given time interval R T (t,0)XR(0,t)u u

Ensemble Prediction Optimal perturbations / singular vectors and stochastic optimals can also be used to generate ensemble forecasts. Perturbing the system along the most unstable directions of the state space yields information about the first and second moments of the probability density function (PDF):  ensemble mean  ensemble spread Excite with dominant basis vectors

Ensemble Prediction For an appropriate forecast skill measure, s

East Australia Current (EAC) Example

EAC: Incremental 4DVar (IS4DVAR) SSH, SST, XBT Assimilation Observations SSH Temperature along XBT line Assimilation

0-days 10-days SV 2SV 6SV 7SV 8SV 10 Ritz Eigenvalues EAC: Optimal Perturbations +

EAC: Ensemble Prediction 15-days forecast1-day forecast8-days forecast

Profiling ROMS Kernel: EAC The NLM is run in the EAC application for 800 time-steps as standalone driver with the same CPP options as the TLM, RPM, ADM: TLM is 2.5 times more expensive than NLM RPM is 2.6 times more expensive than NLM  RPM is 3-percent more expensive than TLM ADM is 2.8 times more expensive than NLM  ADM is 13-percent more expensive than TLM Linux mdksmp #1 SMP Mon Jan 9 23:35:18 MST 2006 x86_64 Dual Core AMD Opteron(tm) Processor 265