1. Slide 1 NEMOVAR-LEFE Workshop 22/3 2007. Slide 1 Current status of NEMOVAR Kristian Mogensen

2. Slide 2 NEMOVAR-LEFE Workshop 22/3 2007. Slide 2 Outline of talk: Why do we need NEMOVAR? -Isn’t OPAVAR good enough? Goals of the NEMOVAR project Implementation plan Where are we now Outstanding issues

3. Slide 3 NEMOVAR-LEFE Workshop 22/3 2007. Slide 3 Why do we need NEMOVAR? Anthony Weaver et al from CERFACS has been developing a variational data assimilation system for OPA version 8.2 (OPAVAR). -Incremental approach -Supports 3D-VAR (FGAT) and 4D-VAR -Has been used with the ORCA2 grid and the TDH tropical pacific area grid -Written mostly in the OPA 8.2 coding style (Fortran-77) with a few extensions  Dynamic memory in a few routines -No distributed memory (MPI) parallelization, only shared memory (OpenMP) parallelization  The OpenMP scaling is not spectacular  Scaling to higher resolution than ORCA2 problematic due to memory constraints (>26 GB needed for ORCA1 needed) OPA 8.2 is not actively developed anymore All work within the OPA developers team are focussed on the new NEMO version of the OPA model

4. Slide 4 NEMOVAR-LEFE Workshop 22/3 2007. Slide 4 ECMWF/CERFACS goals for the NEMOVAR project Short term (in ~2 years) goal -To have a 3D-VAR system based on NEMO -Support distributed memory parallelization  Possible also support shared memory parallelization -Support for different ORCA configurations  We do not worry about limited area versions of NEMO -Support for profiles and altimeter observations and SST products  It should easy to add a new observation type -Multi-incremental with different resolution in the inner loops compared to the outer loops  Not a trivial task Long term goal -A full 4D-VAR system with all of the above properties

5. Slide 5 NEMOVAR-LEFE Workshop 22/3 2007. Slide 5 Splitting the variational problem into outer and inner loops: 1.In the initial outer loop the following is done: 1.Compute misfit of observations relative to background 2.Store initial trajectory for the inner loop 2.In the inner loops the following is done: 1.Minimize the incremental cost function using an iterative procedure to produce an increment 3.In the subsequent outer loop the following is done: 1.Update trajectory with the increment 2.Update misfit of observations 3.Update misfit of background 4.IF ( iloop < noutloop ) GOTO 2

6. Slide 6 NEMOVAR-LEFE Workshop 22/3 2007. Slide 6 Implementation plan: overview We have defined the following plan: -Phase 1: Split the existing OPAVAR Fortran code into separate executables for inner and outer loops -Phase 2: Develop an MPP implementation of the observation operators in the outer loop using NEMO -Phase 3: Develop a hybrid system with NEMO in the outer loop and OPAVAR in the inner loop -Phase 4: Develop an MPP implementation of the 3D-VAR with NEMO in both inner and outer loops -Phase 5: Develop an MPP implementation of the full 4D-VAR with NEMO in both inner and outer loops Phase 1 and 2 can done in parallel By phase 4 we will have archived our short term goal By phase 5 we will have archived our long term goal

7. Slide 7 NEMOVAR-LEFE Workshop 22/3 2007. Slide 7 Implementation plan: phase 1 Split the existing OPAVAR Fortran code into separate executables for inner and outer loops -This is needed for phase 3 -Will allow scientific developments to use OPAVAR to continue work in parallel with the NEMOVAR work  We don’t want this to stop while we do the technical work  Not all options of OPAVAR will be migrated to NEMOVAR -Supports both 3D-VAR and 4D-VAR and combinations of both for multiple inner loops -Will verify the approach of using different executables in the inner and the outer loop  Similar to what is done for the IFS 4D-VAR system

8. Slide 8 NEMOVAR-LEFE Workshop 22/3 2007. Slide 8 Implementation plan: phase 2 Adding observation operators to NEMO -Import interpolation routines from OPAVAR -Distribute the observations according to NEMO domain decomposition initially  Load imbalance in the observation operators? -Initially we will focus on the following observations:  Profiles (XBT’s, Argos etc.)  SLA data  SST data -Easy extendable to other observations Can be used for other assimilation schemes The observation operators can be used for diagnostics to compare a model run with observations

9. Slide 9 NEMOVAR-LEFE Workshop 22/3 2007. Slide 9 Implementation plan: phase 3 Develop a hybrid system with NEMO in the outer loop and OPAVAR in the inner loop -Useful to verify the implementation of the NEMO outer loop -Since the NEMO inner loop is the “hard” part of the migration to NEMO this system will be useful for scientific developments for some time Writing of model trajectories in NEMO for input to the OPA inner loop will have to be coded Reading of increments for applying the them to the non-linear states will have to be coded as well The later 2 items are relevant for phase 4 We will probably not worry too much about MPP for phase 3 Can in principle run both 3D-VAR and 4D-VAR

10. Slide 10 NEMOVAR-LEFE Workshop 22/3 2007. Slide 10 Implementation plan: phase 4 Develop an MPP implementation of the 3D-VAR with NEMO in both inner and outer loops -Parallelization of the control vector will be done based on ECMWF IFS experiences  A Fortran-90 derived type (control_vectors) is used to defined the distributed memory layout of the control vector  Fortran-90 operations like assignment and dot product are overloaded for this type and all message parsing is done in the overloaded functions  Flexible code for non MPP developers -The minimization of the cost function is going to be based on work done at CERFACS -Initially we will assume the same resolution in the outer and inner loop  Later we will consider different resolution in the outer and the inner loop

11. Slide 11 NEMOVAR-LEFE Workshop 22/3 2007. Slide 11 Implementation plan: phase 5 The full 4D-VAR system is dependent on the existence of the tangent linear and adjoint of the NEMO model We will get the status for this in the next talk We aim to have the code of phase 4 flexible enough that adding 4D-VAR as an option is easy once the TL/AD is available

12. Slide 12 NEMOVAR-LEFE Workshop 22/3 2007. Slide 12 Where are we now Phase 1 (the splitting of OPAVAR) has been completed. Phase 2 (observation operators in NEMO) is well under way -It has been done for profile observations -Work on including altimeter is in progress -Work SST data will start shortly Work on phase 3 (hybrid system) is just about to start Work on phase 4 (3D-VAR NEMO) is being discussed and is like to start Q2 2007

13. Slide 13 NEMOVAR-LEFE Workshop 22/3 2007. Slide 13 Some outstanding issues for discussion We have not yet considered quality control of the observations: -Some checks could be done between the first outer loop and the first minimization -Other checks could be done before the first outer loop How to best implement the change of resolutions between outer and inner loops: -Straight interpolation? -Something more advanced which better preserves the physical quantities of the ocean fields?