1. Migration to Rose and High Resolution Modelling Jean-Christophe Rioual, CRUM, Met Office 09/04/2015

2. Migration to Rose

3. GC2 coupled model Basis for Hadgem3-AOIL CMIP6 physical model Glosea5 operational seasonal forecast

4. UMUI ( Before Rose ) 15+ years Well liked By scientists But Complex to maintain and develop Ensemble runs complex

5. Transition to Rose A framework for managing and running suites Initially developed as a replacement for the operational NWP suite infrastructure Developed by Met Office Modelling Infrastructure team (Dave Matthews) UM9.0 retired UMUI and transitioned to Rose Need for GC2 to follow

6. Rose A framework for managing and running suites A workflow scheduler (Cylc) Automatically generated GUIs Version control systems FCM (subversion) A workflow scheduler (Cylc) Automatically generated GUIs Version control systems FCM (subversion) Rose is A set of tools for developing coupled earth systems A set of tools for developing coupled earth systems

8. Cylc Graph FCM MAKE NEMO FCM MAKE UM FCM MAKE UM COUPLED year1 COUPLED year2 POST PROC POST PROC POST PROC POST PROC Cylc control file Suite.rc

9. Rose GUI – Automatically generated from text inputs

10. Met Office Science Repository Service Collaborative development Rosie repository for suite discovery and version control

11. Incremental development of suites

12. A programmer’s view Suite directorysuite.rc App1 Apps App2 metadata App control file App metadata file App control file App metadata file Easy to develop Easy to control Easy to share

15. Climate production runs Specific requirements Long cycles Restartability 360days calendars Archiving apps Portability (Monsoon) Good adoption from community CMIP6 dev&runs to be under Rose Good adoption from community CMIP6 dev&runs to be under Rose

16. High Resolution Coupled Models

17. Complexity vs Resolution Scientific Complexity Resolution N96 AMIP GC2 (n216/60kms-orca25) UKESM N768-ORCA12 (~17kms) N1024-ORCA12 (~12kms) HPC Costs

18. Why higher resolution ? Climate studies ( Primavera / Horizons 2020 ) Seasonal forecasting research ( Charisma project ) NWP - Ocean Forecasting

19. High Resolution Technical Challenges o Transition to Rose o Unstability of components at higher resolutions  Co-development of science and infrastructure  Require strong version control ( Rose ) o Access to computational resources for development o IO ( servers )

20. IO servers UM JULES MCT CICE NEMO XIOS UM IOS Seconds Cores

21. Technical difficulties Project required computational expertise  Complex MPMD layer for MPI split  C++ errors in XIOS  Memory management in XIOS and UM IOS  Parallel Netcdf/HDF5 builds and performance

22. What we have achieved ( so far ) UM N512(25kms) – ORCA12 – OASIS-MCT - XIOS (detached) Rose suite operational UM N512(25kms) – ORCA12 – OASIS-MCT - XIOS (detached) Rose suite operational Total Resources80 IBM P7 Nodes NEMO ORCA121574 tasks (incl. Land suppr. ) XIOS servers48 servers XIOS modeDetached/Multiple Files UM N512~25kms (no IOS)930 tasks CouplerOASIS3-MCT Performance3.2 months simulation/day Simulation lenght12 years Good initial scientific results

23. But still a lot of work Many technical problems to sort out  Parallel Netcdf/HDF5 misbehaving  XIOS / UM IOS intercompatibility  Porting to Cray  Performance and scalability poorly understood  Upgrade to N1024 (~12kms) 100 nodes+ (per ensemble member)

24. High Resolution Coupled Regional NWP system for environmental prediction UKEP system ~2017 UM REGIONAL UK (~1.5km) UM REGIONAL UK (~1.5km) NEMO SHELF SEA MODEL (1/60) NEMO SHELF SEA MODEL (1/60) WAVE Wave Watch3 (~1km) WAVE Wave Watch3 (~1km) OASIS LBCs from operational high resolution global coupled N1024-ORCA12