1. Algorithms Overview Hernan G. Arango Institute of Marine and Coastal Sciences Rutgers University 2004 ROMS/TOMS European Workshop CNR-ISMAR, Venice, October 18-20

2. Outline ROMS/TOMS algorithms status ROMS/TOMS algorithms status ROMS/TOMS future releases ROMS/TOMS future releases How does one build an adjoint model? How does one build an adjoint model? Ensemble prediction Ensemble prediction Variational data assimilation: Variational data assimilation:  Strong constraint 4DVAR  Weak constraint 4DVAR Final remarks Final remarks

3. ROMS/TOMS 2.1 Features Fasham model revisited (Fennel) Fasham model revisited (Fennel) Bio-optical model (up to 84 components), EcoSim (Bissett) Bio-optical model (up to 84 components), EcoSim (Bissett) New bottom boundary layer (Blaas); Fixed Styles and Glenn BBL New bottom boundary layer (Blaas); Fixed Styles and Glenn BBL Sediment model revisited: stratigraphy with Nbed layers (Warner) Sediment model revisited: stratigraphy with Nbed layers (Warner) Momentum and tracer balances (Crowley) Momentum and tracer balances (Crowley) Time-averaged quadratic terms:,,,, Time-averaged quadratic terms:,,,, Isobaric Lagrangian trajectories (Warner) Isobaric Lagrangian trajectories (Warner)

4. ROMS/TOMS 2.1 Features Sequential and concurrent coupling with atmospheric models (Moore, Shaffer) Sequential and concurrent coupling with atmospheric models (Moore, Shaffer)  ESMF (initialize, run, finalize)  Atmospheric coupler: Modeling coupling toolkit (MCT, Argonne National Lab) and WRF I/O API  MPI communicator is split between atmosphere and ocean nodes

5. ROMS/TOMS 2.1 Fixed Bugs Horizontal viscosity Horizontal viscosity Parallel periodic boundaries Parallel periodic boundaries Tiling in serial applications Tiling in serial applications Added river mass transport to DU_avg1 and DV_avg1 arrays Added river mass transport to DU_avg1 and DV_avg1 arrays MPI parallel bug in restart of floats NetCDF MPI parallel bug in restart of floats NetCDF

6. ROMS/TOMS 2.2 Features Ice model Ice model Nesting / composed grids Nesting / composed grids Parallel IO Parallel IO Improvements to sediment model Improvements to sediment model Monotonic tracer advection Monotonic tracer advection

7. Serial Versus Parallel NetCDF (Yang, NCSA) NCSA IBM P690 16 Serial Parallel Serial Parallel Timestep Output Time (0.1 s) (246 x 240 x 16) (656 x 640 x 16)

8. Serial Versus Parallel NetCDF Serial 128 64 32 16 128 Serial Parallel (Yang, NCSA) NCAR IBM SP Cluster (WinterHawk II) Timestep Output Time (0.1 s) (656 x 640 x 16)

9. Sediment Model New Features Suspended-sediment stratification effects in wave boundary layer (neutral currently) Suspended-sediment stratification effects in wave boundary layer (neutral currently) Mechanics for cohesive versus non-cohesive bottom sediments Mechanics for cohesive versus non-cohesive bottom sediments Gravity-driven transport in bottom boundary layer Gravity-driven transport in bottom boundary layer Aggregation / dissaggregationAggregation / dissaggregation Wetting / drying Wetting / drying Bioturbation in sediment layers Bioturbation in sediment layers Bedload transport (with wave effects) Bedload transport (with wave effects) Radiation stresses Radiation stresses

10. ROMS/TOMS Adjoint and Data Assimilation Team Hernan G. Arango Boon Chua Bruce D. Cornuelle Emanuele Di Lorenzo Arthur J. Miller Andrew M. Moore Julio Sheinbaum Rutgers University Oregon State University Scripps Institute of Oceanography Georgia Institute of Technology Scripps Institute of Oceanography University of Colorado CICESE

11. Objectives To provide the ocean modeling community with analysis and prediction tools that are available in meteorology and Numerical Weather Prediction (NWP), using a community OGCM (ROMS/TOMS). To provide the ocean modeling community with analysis and prediction tools that are available in meteorology and Numerical Weather Prediction (NWP), using a community OGCM (ROMS/TOMS). To build a Generalized Stability Analysis (GSA) platform: eigenmodes, optimal perturbations / singular vectors, forcing singular vectors, stochastic optimals, pseudospectra. To build a Generalized Stability Analysis (GSA) platform: eigenmodes, optimal perturbations / singular vectors, forcing singular vectors, stochastic optimals, pseudospectra. To build an ensemble prediction platform. To build an ensemble prediction platform. To build 4D variational assimilation platforms. To build 4D variational assimilation platforms.

12. Overview Let’s represent NL ROMS as:Let’s represent NL ROMS as: The TL ROMS is derived by considering a small perturbation s to S. A first-order Taylor expansion yields:The TL ROMS is derived by considering a small perturbation s to S. A first-order Taylor expansion yields: A is real, non-symmetric Propagator Matrix The AD ROMS is derived by taking the inner-product with an arbitrary vector, where the inner-product defines an appropriate norm (L2-norm):The AD ROMS is derived by taking the inner-product with an arbitrary vector, where the inner-product defines an appropriate norm (L2-norm):

13. How To Build an Adjoint The ADM can be derived from: The ADM can be derived from:  Continuous equations  Discrete equations (A is symmetric; exact)  Hand-coded  Automatic differentiation adjoint compilers (TAMC) The ADM operator relative to L2-norm can be computed by multiplying each line of the TLM code by the corresponding adjoint variable, and then differentiating with respect the TLM variable. The ADM operator relative to L2-norm can be computed by multiplying each line of the TLM code by the corresponding adjoint variable, and then differentiating with respect the TLM variable. Use Geiring and Kaminski (1998) transpose TLM operators and recipes. Use Geiring and Kaminski (1998) transpose TLM operators and recipes. Non-differentiable algorithms (vertical mixing). Non-differentiable algorithms (vertical mixing).

14. Nonlinear Model DO k=1,N DO i=Istr,Iend+1 FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)* & (Hz(i,k)+Hz(i-1,k))* & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc)) END DO DO i=Istr,Iend t(i,k,nnew,itrc)=t(i,k,nnew,itrc)+ & dt*pm(i)*pn(i)*(FX(i+1)-FX(i)) END DO

15. Tangent Linear Model DO k=1,N DO i=Istr,Iend+1 !! FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)* !! & (Hz(i,k)+Hz(i-1,k))* !! & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc)) tl_FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)* & ((tl_Hz(i,k)+tl_Hz(i-1,k))* & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc))+ & (Hz(i,k)+Hz(i-1,k))* & (tl_t(i,k,nrhs,itrc) -tl_t(i-1,k,nrhs,itrc))) END DO DO i=Istr,Iend !! t(i,k,nnew,itrc)=t(i,k,nnew,itrc)+ !! & dt*pm(i)*pn(i)*(FX(i+1)-FX(i)) tl_t(i,k,nnew,itrc)=tl_t(i,k,nnew,itrc)+ & dt*pm(i)*pn(i)*(tl_FX(i+1)-tl_FX(i)) END DO

16. Adjoint Model DO k=1,N DO i=Istr,Iend !! tl_t(i,k,nnew,itrc)=tl_t(i,k,nnew,itrc) + !! & dt*pm(i)*pn(i)*(tl_FX(i+1)-tl_FX(i)) adfac=dt*pm(i)*pn(i)*ad_t(i,k,nnew,itrc) ad_FX(i+1)=ad_FX(i+1)+adfac ad_FX(i )=ad_FX(i ) -adfac END DO DO i=Istr,Iend+1 !! tl_FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))* pmon_u(i)* !! & ((tl_Hz(i,k)+tl_Hz(i-1,k))* !! & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc))+ !! & (Hz(i,k)+Hz(i-1,k))* !! & (tl_t(i,k,nrhs,itrc) -tl_t(i-1,k,nrhs,itrc))) adfac =0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)*ad_FX(i) adfac1=adfac*(t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc)) adfac2=adfac*(Hz(i,k)+Hz(i-1,k)) ad_Hz(i,k)=ad_Hz(i,k)+adfac1 ad_Hz(i-1,k)=ad_Hz(i-1,k)+adfac1 ad_t(i,k,nrhs,itrc)=ad_t(i,k,nrhs,itrc)+adfac2 ad_t(i-1,k,nrhs,itrc)=ad_t(i-1,k,nrhs,itrc) -adfac2 ad_FX(i) =0.0_r8 END DO

17. Ensemble Prediction Optimal perturbations / singular vectors and stochastic optimal can also be used to generate ensemble forecasts.Optimal perturbations / singular vectors and stochastic optimal can also be used to generate ensemble forecasts. Perturbing the system along the most unstable directions of the state space yields information about the first and second moments of the probability density function (PDF):Perturbing the system along the most unstable directions of the state space yields information about the first and second moments of the probability density function (PDF):  ensemble mean  ensemble spread Adjoint based perturbations excite the full spectrumAdjoint based perturbations excite the full spectrum

18. Ensemble Prediction For an appropriate forecast skill measure, s

19. Data Assimilation Overview Cost Function: Cost Function: wheremodel,background,observations, background error covariance, inverse background error covariance, inverse observations error covariance Model solution depends on initial conditions ( ), boundary conditions, and model parameters Model solution depends on initial conditions ( ), boundary conditions, and model parameters Minimize J to produce a best fit between model and observations by adjusting initial conditions, and/or boundary conditions, and/or model parameters. Minimize J to produce a best fit between model and observations by adjusting initial conditions, and/or boundary conditions, and/or model parameters.

20. Minimization Perfect model constrained minimization (Lagrange function): Perfect model constrained minimization (Lagrange function): We require the minimum of at which:,,,,,, yielding A T is the transpose of A, often called the adjoint operator. It can be shown that:A T is the transpose of A, often called the adjoint operator. It can be shown that: The adjoint equation solution provides gradient information

21. 4D Variational Data Assimilation Platforms (4DVAR) Strong Constraint (S4DVAR) drivers: Strong Constraint (S4DVAR) drivers:  Conventional S4DVAR: outer loop, NL, AD  Incremental S4DVAR: inner and outer loops, NL, TL, AD (Courtier et al., 1994)  Efficient Incremental S4DVAR (Weaver et al., 2003) Weak Constraint (W4DVAR) - IOM Weak Constraint (W4DVAR) - IOM  Indirect Representer Method: inner and outer loops, NL, TL, RP, AD (Egbert et al., 1994; Bennett et al, 1997) RP:

22. “Conventional” S4DVAR NLM: compute model-observations misfit and cost function ADM: compute cost function gradients Compute NLM initial conditions using first guess conjugate gradient step size NLM: compute change in cost function Compute NLM initial conditions using refined conjugate gradient step size CALL initial CALL main3d CALL ad_initial CALL ad_main3d CALL initial CALL main3d CALL descent CALL wrt_ini CALL descent CALL wrt_ini Outer Loop Ipass=1 Ipass=2

23. Incremental S4DVAR CALL initial CALL main3d Outer Loop CALL tl_initial CALL tl_main3d CALL ad_initial CALL ad_main3d CALL tl_initial CALL tl_main3d CALL descent CALL tl_wrt_ini CALL descent CALL tl_wrt_ini Inner Loop Ipass=1 Ipass=2 CALL ini_adjust CALL wrt_ini NLM: compute basic state trajectory and extract model at observations locations TLM: compute misfit cost function between model (NLM+TLM) and observations ADM: compute cost function gradients Compute TLM initial conditions using first guess conjugate gradient step size TLM: compute change in cost function Compute TLM initial conditions using refined conjugate gradient step size Compute NLM new initial conditions (NLM+TLM)

24. Efficient Incremental S4DVAR NLM: compute basic state trajectory and extract model at observations locations ADM: compute initial estimate of the gradient Initialize conjugate direction as the negative of the gradient (adjoint) solution RPM: compute misfit cost function between model (NLM+TLM) and observations ADM: compute cost function gradients Compute TLM initial conditions using conjugate gradient step size Compute NLM new initial conditions (NLM+TLM) CALL initial CALL main3d Outer Loop CALL tl_initial CALL tl_main3d CALL ad_initial CALL ad_main3d CALL descent CALL tl_wrt_ini Inner Loop CALL ini_adjust CALL wrt_ini CALL ad_initial CALL ad_main3d CALL ini_descent

25. W4DVAR, IOM iom_roms: compute first guess and misfit between observation and model nl_roms: compute basic state trajectory Inner loop, backward (ad_roms) and forward (tl_roms) integrations to compute ad_roms: backward integration to compute iom_roms: compute nl_roms < nl_roms.in ad_roms < ad_roms.in tl_roms < tl_roms.in IOM components iom_roms < iom_roms.in ad_roms < ad_roms.in iom_roms < iom_roms.in Inner Loop Outer Loop

26. Twin Experiments Spin-up an idealized, wind-forced double-gyre for 50 years. Spin-up an idealized, wind-forced double-gyre for 50 years. Basin dimensions: 1000x2000 km 2 Basin dimensions: 1000x2000 km 2 Grid resolution: dx=dy=18.518 km (54x108x4) Grid resolution: dx=dy=18.518 km (54x108x4) Run equilibrium solution for another 5 days and extract observations (true state) daily for each state variable at every spatial grid point. Run equilibrium solution for another 5 days and extract observations (true state) daily for each state variable at every spatial grid point. Initialize 4DVAR algorithms from rest and assimilate observations at day 1. Initialize 4DVAR algorithms from rest and assimilate observations at day 1. Force only with the adjoint misfit (model minus observations) terms. Force only with the adjoint misfit (model minus observations) terms.

27. Free-surface and Currents Final Adjusted Initial ConditionsAdjusted Minus Truth Solution Free-surface Difference RMS = 1.568e-5 Ubar Difference RMS = 1.690e-5 Vbar Difference RMS = 7.995e-6 S4DVAR

28. S4DVAR 3D Double Gyre Final Adjusted Initial Conditions Free-surface and Currents Model-Observation Misfit Cost Function Iteration Free-surface Difference Adjusted Minus Truth Solution Vbar Difference Potential Temperature Difference

29. IOM Final Adjusted Initial Conditions Free-surface and Currents Adjusted Minus Truth Solution Free-surface Difference RMS = 2.136e-3 Ubar Difference RMS = 2.960e-2 Vbar Difference RMS = 5.085e-2 True Solution

30. Ongoing 4DVAR Applications Southern California Bight (Cornuelle, Di Lorenzo, Miller)Southern California Bight (Cornuelle, Di Lorenzo, Miller) U.S. East coast (Arango, Moore, Wilkin)U.S. East coast (Arango, Moore, Wilkin) Intra-Americas Sea (Moore, Sheinbaum)Intra-Americas Sea (Moore, Sheinbaum) Gulf of Mexico (Moore, Sheinbaum)Gulf of Mexico (Moore, Sheinbaum) East Australia Current (Arango, Wilkin)East Australia Current (Arango, Wilkin) Oregon coast (Durski)Oregon coast (Durski)

31. Observation Types plus satellite data (SSH, SST) and radar

32. Timing considerations SCB – 6 CPU minutes per simulation day per TLM/ADM call on a 833MHz Alpha (78x118x30).SCB – 6 CPU minutes per simulation day per TLM/ADM call on a 833MHz Alpha (78x118x30). GoM – 17 CPU minutes per simulation day per TLM/ADM call on a 833 MHz Alpha.GoM – 17 CPU minutes per simulation day per TLM/ADM call on a 833 MHz Alpha. IAS – 15 CPU minutes per simulation day per TLM/ADM call on a 833 MHz Alpha.IAS – 15 CPU minutes per simulation day per TLM/ADM call on a 833 MHz Alpha. NENA – 60 CPU minutes per simulation day per TLM/ADM call on a 833 MHz Alpha (384x128x30).NENA – 60 CPU minutes per simulation day per TLM/ADM call on a 833 MHz Alpha (384x128x30). Data assimilation scaling factors:Data assimilation scaling factors:  S4DVAR = 2  IS4DVAR = 3 inner, 0.5 outer  EIS4DVAR = 2 inner, 0.5 outer  W4DVAR = 2 inner, 2.5 outer.

33. Final Remarks Maintenance of TLM, RPM, and ADM models.Maintenance of TLM, RPM, and ADM models. Parallelization of TLM, RPM, and ADM models.Parallelization of TLM, RPM, and ADM models. Modeling background error covariance.Modeling background error covariance. Training and documentation.Training and documentation.

34. Publications Moore, A.M., H.G Arango, E. Di Lorenzo, B.D. Cornuelle, A.J. Miller and D. Neilson, 2004: A comprehensive ocean prediction and analysis system based on the tangent linear and adjoint of a regional ocean model, Ocean Modelling, 7, 227-258. Moore, A.M., H.G Arango, E. Di Lorenzo, B.D. Cornuelle, A.J. Miller and D. Neilson, 2004: A comprehensive ocean prediction and analysis system based on the tangent linear and adjoint of a regional ocean model, Ocean Modelling, 7, 227-258. http://marine.rutgers.edu/po/Papers/Moore_2004_om.pdf http://marine.rutgers.edu/po/Papers/Moore_2004_om.pdf Arango, H.G., Moore, A.M., E. Di Lorenzo, B.D. Cornuelle, A.J. Miller and D. Neilson, 2003: The ROMS Tangent Linear and Adjoint Models: A comprehensive ocean prediction and analysis system, Rutgers Tech. Report. Arango, H.G., Moore, A.M., E. Di Lorenzo, B.D. Cornuelle, A.J. Miller and D. Neilson, 2003: The ROMS Tangent Linear and Adjoint Models: A comprehensive ocean prediction and analysis system, Rutgers Tech. Report. http://marine.rutgers.edu/po/Papers/roms_adjoint.pdf http://marine.rutgers.edu/po/Papers/roms_adjoint.pdf