© Crown copyright Met Office Atmosphere-Chemistry Model Coupling in the UK Earth System Model (UKESM) Richard Hill, April 2015 Credits: Marc Stringer, Colin Johnson, Mohit Dalvi and many more including several in this room.

© Crown copyright Met Office Contents 1.UKESM Overview 2.Background on performance issues 3.3D coupling and OASIS3-MCT 4.Possible approaches 5.Progress to date Questions

© Crown copyright Met Office UKESM – UK Earth System Model A Joint Met Office/NERC Collaboration Pooling expertise and resources from various centres The UK community contribution to CMIP6 Aims to include various component models: UM: atmosphere JULES: land surface NEMO: ocean CICE: sea-ice BICICLES: Ice shelf MEDUSA: ocean biogeochemistry UKCA: UK community atmospheric chemistry-aerosol Wave model – one day e.g. WAVEWATCH III?

JULES Land surf UM Atmosphere NEMO Ocean CICE Sea Ice OASIS3 or OASIS3-MCT Current Met Office climate models

JULES Land surf UM Atmosphere UKCA Chemistry NEMO Ocean Ice Shelf Wave? CICE Sea Ice MEDUSA Ocean Biogeochem OASIS3-MCT UKESM potential component models

© Crown copyright Met Office But: Chemistry is a BIG cost! UM+UKCA profiling: UKCA increases cost by ~400%

© Crown copyright Met Office Chemistry Costs! UKCA features heavy dynamic load imbalance Typically ~25%-30% of cost is dead time as work follows the sun! – Cray XC40 task positioning might help load balance but won't get us to where we need to be. Cost is prohibitive for intended global resolution (N216 = 432x324x85) Optimisation work on UKCA code over the years has not delivered more than a few percent improvement We need more like ~80% Tinkering with the code has no chance of delivering that. We need another way! We need another way!..

© Crown copyright Met Office How about this? Run UKCA at a REDUCED resolution (somehow) and couple to higher resolution atmosphere? Maybe.... but... – UKCA is embedded in UM atmosphere – UKCA is tied to UM atmosphere resolution – UKCA is not currently a viable stand-alone code! – We'd need to exchange (potentially) lots of 3D fields somehow Can we even do that?

© Crown copyright Met Office Is 3D coupling viable? OASIS3-MCT offers hope with 3D fields. Two possible approaches: One allows us to deal with each 3D field as a single entity The other allows us to bundle multiple 2D fields together in order to act as a 3D field. Both components have the same number of vertical levels (typically 85). This appears not to be an issue as far as the science is concerned. i.e. We only need HORIZONTAL regridding.

© Crown copyright Met Office 3D Field Coupling Option 1): define 3D fields as a true 3D record. Requires special 3D remapping weights files which repeat the same weights 85 times in the same record! (assuming no vertical interpolation) Requires special program and operations to generate such a file Such files are huge. Separate files required for fields of differing numbers of vertical levels i.e. You can't just re-use weights files in the same way that you can with 2D equivalents. Generation, management and debugging of such files is much more difficult than the 2D case. But you CAN define and put an entire 3D field in one go!

© Crown copyright Met Office Level 8 Level 7 Level 6 Level 5 Level 4 Level 3 Level 2 Level Coupling using full 3D regridding weights files Source 3D fild Source 3D field Target 3D field Full set of 3D weights in 1 file

© Crown copyright Met Office 3D Field Coupling Option 2): Treat each vertical level as a 2D field Use functional features of OASIS3-MCT to allow definition of multiple 2D fields in a single namcouple record – thereby defining a 3D entity. OASIS3-MCT effectively bundles multiple fields together. So the “put” operation only actually takes place once the final level is supplied to the “put” call. Allows use of a single 2D remapping weights file by each “2D” field at each vertical level. 2D weights files are much smaller and easier to create, manage and debug than anything which might constitute a 3D weights file. But we have to define and put lots of separate 2D fields.

© Crown copyright Met Office Coupling using 2D regridding weights files Source 3D fild Source 3D field Target 3D field Single level 2D weights file

© Crown copyright Met Office 3D coupling is viable! OASIS3-MCT offers hope with 3D fields. Tests with toy model systems and iteration with OASIS3-MCT developers: We can successfully exchange ~200 3D fields (85 vertical levels) between N216 and N96 toy models – i.e. Equivalent to 17,000 2D fields!!! (atmos- ocean uses ~53 2D fields in total !) Cost appears acceptable – 5-10% of projected model run times, massively offset by potential reduction in elapsed time and overall resources.

© Crown copyright Met Office 3D Field Coupling So 3D field coupling looks viable Option 2) “multi-2D” approach looks preferable Even if we end up having to define and exchange large numbers of fields... (~85 x option 1)... long term maintenance and usability is overriding factor. OASIS namcouple file generation and field naming is an issue. – Potentially 1000's of different field names – OASIS EXPOUT gets “interesting”...but we can probably cope So what about the component models?.....

UKCA Component Options Create a stand alone UKCA component? But remember.... UKCA is embedded in UM atmosphere UKCA is tied to UM atmosphere resolution UKCA is not currently a viable stand-alone code!

© Crown copyright Met Office UKCA Component Options Unraveling UKCA from UM is like.... Unraveling two of these

© Crown copyright Met Office UKCA Component Options WITHOUT one of these

UKCA Component Options Create a stand alone UKCA component? Very involved Requires restart and diagnostic handling system, MPP decomposition, launching mechanism, etc, etc. In fact a whole new independent supporting infrastructure. ALL BEFORE YOU START WORRYING ABOUT THE COUPLING Also requires coupling of huge numbers of 3D, 2D, 4D (and 1D) fields, mostly chemical tracers, some integers and even some logicals! Some good news: Tests with toy models and OASIS3-MCT suggest volume of coupling data would not be a limiting factor.Some good news: Tests with toy models and OASIS3-MCT suggest volume of coupling data would not be a limiting factor.

JULES Land surf (High res) UM Atmosphere (High res) UKCA Chemistry (Any res) OASIS3-MCT So we can’t do this… (at least not easily)

JULES Land surf (High res) UM Atmosphere (High res) UKCA Chemistry (Low res) OASIS3-MCT …or this… (at least not easily)

© Crown copyright Met Office UKCA Component Options Alternative: Simply run an entire “junior” UM+UKCA at the lower res coupled to a “senior” UM at the higher res. ● Pass key coupling fields from higher resolution senior component, to keep junior UM evolution in line. ● Nudging/assimilation or complete overwriting of fields. ● Simpler than creating a brand new stand-alone UKCA component. ● Requires fewer coupling fields than a true stand-alone UKCA - 3D fields only and no awkward integers, etc.

Senior – Junior coupled models proposed set-up © Crown copyright Met Office Senior UM N216 Junior UM N96 UKCA NEMO Ocean CICE Sea-ice M E D U S A OBGC OASIS3-MCT Coupling

© Crown copyright Met Office UKCA Component Options Running a “junior” UM-UKCA needs... ● Modifications to user interface to allow TWO UM components to be specified and submitted (along with existing NEMO-CICE and XIOS capability) ● Modify system control to allow two UM components to run concurrently... naming conventions of each component. ● Archiving support. ● Ability for both UM components to start from “the same” initial conditions... the UM reconfiguration and coupling exchanges at timestep zero can facilitate this.

© Crown copyright Met Office UKCA Component Options Running a “junior” UM-UKCA features... ● Tracer advection is performed in junior. ● Extra cost in terms of duplicated operations in junior is only about 20% of total junior cost (remember earlier performance chart.) ● Ready-made IO, decomposition, diagnostic, etc system. ● Generation of namcouple files on-the-fly is desirable due to flexible scientific configuration needs. ● Which in turn demands more code and data handling developments to enable this.

Current position Routinely running a prototype system coupling UM N216 with UM N96 + Chemistry. Currently exchanging ~10 3D fields using “option 2)” mentioned previously in the N216 -> N96 direction. The main purpose of these is to ensure the “junior” model evolves in the same way as the “senior”. e.g. U, V, W, temperature, moisture fields Currently setting up the N96 -> N216 coupling to pass back critical fields from chemistry which will affect atmos evolution. – e.g. Aerosols, dust etc. © Crown copyright Met Office

JULES Land surf UM Atmosphere (High res) UKCA Chemistry (Low res) NEMO Ocean CICE Sea Ice MEDUSA Ocean Biogeochem OASIS3-MCT UKESM current prototype JULES Land surf (Low res) UM Atmosphere (Low res)

Current position Investigations concentrated on: – Most effective interpolation methods – Nudging/assimilation of “junior” with coupled fields versus complete overwriting? – Likely resource requirements Bottom line(s)... – Technical viability of 3D coupling is not a major concern! – Performance of 3D coupling is not a major concern! At least for now!..... At least for now!..... © Crown copyright Met Office

Summary ● UKESM aims to include chemistry in climate model ● Chemistry is very expensive – BUT SCALABLE! ● Aim to run chemistry at reduced resolution cf. atmos ● OASIS3-MCT facilitates this by allowing 3D field coupling ● Toy model tests indicate technical and speed viability ● Actual model developments and tests to date still indicate viability..... still relatively early stages. ● Detailed scientific tests yet to be completed – potential issues with time stepping and position of coupling exchanges. ● Only tested on IBM power7.... Cray XC40 porting soon! © Crown copyright Met Office

Questions and answers