CSEG Update Mariana Vertenstein CCSM Software Engineering Group Mariana Vertenstein CCSM Software Engineering Group

22 Improvements to CCSM scripts  Addition of timing tool that provides automated information to help determine load balance, throughput and cost of a run.  Improvements made to CCSM script infrastructure  Provide the ability to define new “ component sets/modes ”. and as command line options. Makes it easier to run and test new CCSM science (such as addition of component biogeochemistry and atmospheric chemistry).  Simplified user specification of non-default task/thread settings.  Changes improve extensibility and robustness of scripts.  Addition of timing tool that provides automated information to help determine load balance, throughput and cost of a run.  Improvements made to CCSM script infrastructure  Provide the ability to define new “ component sets/modes ”. and as command line options. Makes it easier to run and test new CCSM science (such as addition of component biogeochemistry and atmospheric chemistry).  Simplified user specification of non-default task/thread settings.  Changes improve extensibility and robustness of scripts.

33 Improvements to CCSM scripts (cont)  CCSM test framework rewritten using new functionality.  Introduced new testing functionality (e.g. auto-promotion test).  Introduced new “ test suites ” that now permit different testing levels:  development, pre-tag, post-tag, monthly  currently in process of defining contributions to each test suite  Benefits:  Regression testing will be easier to perform.  Straightforward addition of tests for new CCSM scenarios.  Easier testing of new scenarios (e.g. CLM-CN with CO2 exchange).  More frequent testing of new science.  More robust CCSM code base.  CCSM test framework rewritten using new functionality.  Introduced new testing functionality (e.g. auto-promotion test).  Introduced new “ test suites ” that now permit different testing levels:  development, pre-tag, post-tag, monthly  currently in process of defining contributions to each test suite  Benefits:  Regression testing will be easier to perform.  Straightforward addition of tests for new CCSM scenarios.  Easier testing of new scenarios (e.g. CLM-CN with CO2 exchange).  More frequent testing of new science.  More robust CCSM code base.

44 Porting and Performance  ORNL Cray X1 (phoenix)  Both release and development code base have been ported. CAM FV dycore ported in development code base. Porting this dycore in CCSM proved to be very challenging.  30 years/day on 200 procs (ignorning performance fluctuations)  13 years/day on 108 procs on NCAR bluevista  400 year FV control simulation carried out  ORNL Cray XT3 (jaguar)  Port of development code base is starting now. Release code base is running - will require a validation.  NCAR IBM (bluevista)  Major new OS upgrade will permit over-subscription of nodes. Provides an opportunity for major performance improvements - but will also result in significantly larger parameter space for performance optimization.  ORNL Cray X1 (phoenix)  Both release and development code base have been ported. CAM FV dycore ported in development code base. Porting this dycore in CCSM proved to be very challenging.  30 years/day on 200 procs (ignorning performance fluctuations)  13 years/day on 108 procs on NCAR bluevista  400 year FV control simulation carried out  ORNL Cray XT3 (jaguar)  Port of development code base is starting now. Release code base is running - will require a validation.  NCAR IBM (bluevista)  Major new OS upgrade will permit over-subscription of nodes. Provides an opportunity for major performance improvements - but will also result in significantly larger parameter space for performance optimization. 44

55 Porting and Performance (future)  NCAR IBM Bluegene  Will require single executable concurrent system  Will require parallel I/O (NetCDF and binary)  NCAR IBM Bluegene  Will require single executable concurrent system  Will require parallel I/O (NetCDF and binary)

66 Data Model Project  Serial version rewrite of all data models (datm7, dlnd7, dice7, docn7) completed!  datm7 can now duplicate stand-alone CLM functionality (in serial mode).  docn7 now has both DOM and SOM functionality (SOM scientific verification is close to complete).  All data components now have same functionality.  Can perform spatial interpolation from input data resolution to model resolution.  Can data cycle over subset of years.  Serial version rewrite of all data models (datm7, dlnd7, dice7, docn7) completed!  datm7 can now duplicate stand-alone CLM functionality (in serial mode).  docn7 now has both DOM and SOM functionality (SOM scientific verification is close to complete).  All data components now have same functionality.  Can perform spatial interpolation from input data resolution to model resolution.  Can data cycle over subset of years.

77 Data Model Project (what is next)  Parallelize the new serial data models.  Incorporate parallel data models into currently developed sequential CCSM.  Replace component specific data models with CCSM data models. As examples:  Replace SOM/DOM in CAM with DOCN7  Replace forcing atm driver in CLM with DATM7  Use of only one set of data models will result in more consistent science and remove existing code duplication.  Parallelize the new serial data models.  Incorporate parallel data models into currently developed sequential CCSM.  Replace component specific data models with CCSM data models. As examples:  Replace SOM/DOM in CAM with DOCN7  Replace forcing atm driver in CLM with DATM7  Use of only one set of data models will result in more consistent science and remove existing code duplication. 77

88 Single Executable Concurrent CCSM  CSEG is leveraging the work that Helen He and Chris Ding (SciDAC) have done in order to create a single executable concurrent implementation of CCSM development code.  A somewhat different implementation will be produced to satisfy current CCSM requirements.  Aim to have development CCSM tag (including all CCSM components) by mid-July.  Aim to have a CCSM3.0 release based version released by later in the summer.  Existence of single executable system should improve CCSM portability and debugging.  CSEG is leveraging the work that Helen He and Chris Ding (SciDAC) have done in order to create a single executable concurrent implementation of CCSM development code.  A somewhat different implementation will be produced to satisfy current CCSM requirements.  Aim to have development CCSM tag (including all CCSM components) by mid-July.  Aim to have a CCSM3.0 release based version released by later in the summer.  Existence of single executable system should improve CCSM portability and debugging.

99 Single Executable Sequential CCSM  Major progress made in replacing stand-alone CAM with “ pseudo-sequential CCSM ”.  Both ESMF and MCT will be examined as coupling frameworks. We are committed to creating a sequential interoperable CCSM utilizing the ESMF framework.  ESMF superstructure coding for Stage 1 Evaluation Plan is beginning now. Superstructure design is already in place. Estimated completion date is end of summer. Plan is to incorporate ESMF coupling framework on CAM trunk upon successful completion of Stage 1 evaluation effort.  Major progress made in replacing stand-alone CAM with “ pseudo-sequential CCSM ”.  Both ESMF and MCT will be examined as coupling frameworks. We are committed to creating a sequential interoperable CCSM utilizing the ESMF framework.  ESMF superstructure coding for Stage 1 Evaluation Plan is beginning now. Superstructure design is already in place. Estimated completion date is end of summer. Plan is to incorporate ESMF coupling framework on CAM trunk upon successful completion of Stage 1 evaluation effort.

10 Original stand-alone CAM architecture (CAM3.0) CAM Driver Physics CLMCAM-ICECAM-OCN Dynamics Note that surface models are invoked from within CAM physics - not from a top level driver

Pseudo-Sequential CCSM Top level Architecture Application Driver (FD) CLMCAM-ICECAM-OCN CAM Phys Dyn Couplers (FD) ATM merger (FD) OCN merger (FD) LND merger (FD) ICE merger (FD) Thin coupling layer (FD) Thin coupling layer (FD) Thin coupling layer (FD) Thin coupling layer (FD) Introduce top level framework dependent (FD) application driver to replace CAM driver Introduce top level ESMF clock to coordinate time evolution of all components Introduce new flexible and extensible “thin” coupling layer design

12 Sequential CCSM (cont)  New top level based application driver.  Independent of CAM data structures  Time evolution based upon ESMF general time management utilities  Utilizes CCSM share code for reading input  Initial implementation: MCT  Near term implementation: ESMF  New surface coupling layer.  Initial implementation: MCT  Near term implementation: ESMF  New directory structure for sequential CCSM.  Applies to both MCT and ESMF  Inter-component domain checking in coupling layer.  Initial implementation: MCT  Near term implementation: ESMF  New top level based application driver.  Independent of CAM data structures  Time evolution based upon ESMF general time management utilities  Utilizes CCSM share code for reading input  Initial implementation: MCT  Near term implementation: ESMF  New surface coupling layer.  Initial implementation: MCT  Near term implementation: ESMF  New directory structure for sequential CCSM.  Applies to both MCT and ESMF  Inter-component domain checking in coupling layer.  Initial implementation: MCT  Near term implementation: ESMF

13 CAM Update  Implemented support for non lat-lon grids (Jim Edwards and Pat Worley). Non lat-lon support added to  Boundary data and aerosol data interpolation code.  Physics/Dyamics coupling layer.  Only aqua-planet mode is currently supported.  Significant FV dycore interface refactoring implemented:  Only data on the XY decomposition outside the portable dycore is now exposed.  Added dynamics import/export states, and dynamics component module with init, run, final methods.  CAM testing framework extended to add new platforms (e.g. phoenix) and to perform overnight regression testing.  New features added to CAM tropospheric MOZART.  Implemented support for non lat-lon grids (Jim Edwards and Pat Worley). Non lat-lon support added to  Boundary data and aerosol data interpolation code.  Physics/Dyamics coupling layer.  Only aqua-planet mode is currently supported.  Significant FV dycore interface refactoring implemented:  Only data on the XY decomposition outside the portable dycore is now exposed.  Added dynamics import/export states, and dynamics component module with init, run, final methods.  CAM testing framework extended to add new platforms (e.g. phoenix) and to perform overnight regression testing.  New features added to CAM tropospheric MOZART.

14 CAM Update (what is next)  Refactoring of CAM to run with new dycores.  Incorporation of HOMME dycore into CAM.  Refactoring CAM ’ s history module to encompass non lat-lon output.  Ability to run pseudo-sequential CCSM (not just aqua-planet) with non lat-lon CAM.  Incorporation of parallel I/O into CAM.  Replace all binary I/O with NetCDF I/O.  Creation of new tool to generate CAM namelists (tool should easily extend to other models if desired).  Refactoring of CAM to run with new dycores.  Incorporation of HOMME dycore into CAM.  Refactoring CAM ’ s history module to encompass non lat-lon output.  Ability to run pseudo-sequential CCSM (not just aqua-planet) with non lat-lon CAM.  Incorporation of parallel I/O into CAM.  Replace all binary I/O with NetCDF I/O.  Creation of new tool to generate CAM namelists (tool should easily extend to other models if desired).

15 CAM Update (what is next)  SCAM (Single Column Atmosphere Model) refactoring  Goal is to produce a more flexible and maintainable interfaces.,  Adoption of new components  Incorporation of new ice sheet model (GLIMMER).  Primary work will be done by Bill Lipscomb.  GLIMMER has already been incorporated into CAM build system.  Replacement of CAM-CSIM with CICE and CAM-SOM/DOM with DOCN7.  SCAM (Single Column Atmosphere Model) refactoring  Goal is to produce a more flexible and maintainable interfaces.,  Adoption of new components  Incorporation of new ice sheet model (GLIMMER).  Primary work will be done by Bill Lipscomb.  GLIMMER has already been incorporated into CAM build system.  Replacement of CAM-CSIM with CICE and CAM-SOM/DOM with DOCN7.

16 CLM Update  Finemesh grids have been implemented in CLM. This gives CLM the new capability to run on its own independent grid. The implementation follows the scheme of Hahmann and Dickinson.  CLM still couples to CAM or CCSM via the CAM coarse grid. Mapping is done within the CLM code base.  The implementation does not change answers when the CLM fine grid is identical to the CAM grid.  Stand-alone CAM runs have been run with a T42 coarse grid and a half degree finemesh grid. Results are encouraging.  New downscaling and upscaling interactions are being implemented.  Finemesh grids have been implemented in CLM. This gives CLM the new capability to run on its own independent grid. The implementation follows the scheme of Hahmann and Dickinson.  CLM still couples to CAM or CCSM via the CAM coarse grid. Mapping is done within the CLM code base.  The implementation does not change answers when the CLM fine grid is identical to the CAM grid.  Stand-alone CAM runs have been run with a T42 coarse grid and a half degree finemesh grid. Results are encouraging.  New downscaling and upscaling interactions are being implemented.

17 CLM Update (what is next)  A nested grid capability is being added to CLM, so the model can be run at spatially varied resolution to optimize cost.  A new prognostic canopy air space scheme is being implemented.  Scientific formulation is being finalized.  Software implementation (Forrest Hoffman) will start as soon as the scientific formulation is completed.  CLM offline testing framework will be rewritten in order to test the numerous new features that are being introduced into the system.  Urban code will be incorporated into the main development line of development.  A nested grid capability is being added to CLM, so the model can be run at spatially varied resolution to optimize cost.  A new prognostic canopy air space scheme is being implemented.  Scientific formulation is being finalized.  Software implementation (Forrest Hoffman) will start as soon as the scientific formulation is completed.  CLM offline testing framework will be rewritten in order to test the numerous new features that are being introduced into the system.  Urban code will be incorporated into the main development line of development.

18 POP2 Update  Completed incorporation of major CCSM features into the LANL pop2.1.alpha code. CCSM POP2 code is now in CCSM SVN repository and current CCSM scripts.  Added  New tracer advection scheme (1D Lax-Wendroff with 1D flux limiters).  Near-surface eddy flux, enhanced deep mixing.  Support for new 1-degree grid with Galapagos Islands topography (gx21v4).  Completed incorporation of major CCSM features into the LANL pop2.1.alpha code. CCSM POP2 code is now in CCSM SVN repository and current CCSM scripts.  Added  New tracer advection scheme (1D Lax-Wendroff with 1D flux limiters).  Near-surface eddy flux, enhanced deep mixing.  Support for new 1-degree grid with Galapagos Islands topography (gx21v4).

19 POP2 Update (what is next)  Inclusion of a more efficient barotropic solver designed by John Dennis.  Reduced equatorial viscosity.  A new vertical grid.  Systematic exploration of the above features.  Plans for incorporating the ecosystem model into POP2 will be developed during the CCSM workshop.  Inclusion of a more efficient barotropic solver designed by John Dennis.  Reduced equatorial viscosity.  A new vertical grid.  Systematic exploration of the above features.  Plans for incorporating the ecosystem model into POP2 will be developed during the CCSM workshop.

20 CICE Update  PCWG has decided to incorporate CICE as the new standard CCSM ice model. New name will be “ Community Ice CodE ”.  Testing has been done using CICE 3.1 to establish main differences with CSIM 5.0.  CICE 3.1 changes answers significantly, but still within the realm of the same climate.  CICE 3.14 has additional bug fixes and a few new physical parameterizations. This is the version that will be adopt after more testing is completed.  Goal is to incorporate CICE 4 (with new data structures) into CCSM and into pseudo-sequential CCSM.  PCWG has decided to incorporate CICE as the new standard CCSM ice model. New name will be “ Community Ice CodE ”.  Testing has been done using CICE 3.1 to establish main differences with CSIM 5.0.  CICE 3.1 changes answers significantly, but still within the realm of the same climate.  CICE 3.14 has additional bug fixes and a few new physical parameterizations. This is the version that will be adopt after more testing is completed.  Goal is to incorporate CICE 4 (with new data structures) into CCSM and into pseudo-sequential CCSM.