1. Mesoscale & Microscale Meteorological Division / NCAR WRF ESMF Development Tom Henderson, John Michalakes National Center for Atmospheric Research Mesoscale and Microscale Meteorology hender@ucar.edu 4 th ESMF Community Meeting -- 21 July 2005

2. Mesoscale & Microscale Meteorological Division / NCAR Acknowledgements Significant support –Air Force Weather Agency, esp. Jerry Wegiel –DoD HPCMO –USWRP (NOAA and NSF) Many collaborators/contributors –NCAR, NOAA, DoD, universities, vendors

3. Mesoscale & Microscale Meteorological Division / NCAR WRF Software Framework Overview Implementation of WRF Architecture –Hierarchical organization –Multiple dynamical cores –Plug compatible physics –Abstract interfaces (APIs) to external packages –Performance-portable –Metaprogramming “Registry” for managing model state ARW core Physics Interfaces Plug-compatible physics NMM core Top-level Control, Memory Management, Nesting, Parallelism, External APIs mediation driver model

4. Mesoscale & Microscale Meteorological Division / NCAR WRF Registry Specify model state arrays, I/O, coupling, interprocess communication (when needed), nesting interpolation, etc. in ASCII text file called “Registry” WRF build reads Registry file and automatically generates source code for: –Data members for WRF “domain” objects –I/O, coupling, and communication calls –Actual and dummy arguments in mediation-layer calls

5. Mesoscale & Microscale Meteorological Division / NCAR New Feature: Moving Nests John Michalakes augmented 2-way interactive nesting in WRF to allow reorientation of nested domain with respect to parent –Automatic movement algorithms such as U. Miami vortex following scheme –Automatic ingest of nested-resolution terrain and other lower boundary data to initialize leading edge of moving nest –HyCOM coupling (UNDER DEVELOPMENT) –Supports 2 or more nest levels in telescoping configuration –Parallel and efficient: small additional overhead (~2%) on top of 5- 8% overhead for non-moving 2-way nesting. With S. Chen, J. Cangialosi, W. Zhao (RSMAS, U. Miami) and S. Gopal at NCEP. Software infrastructure development supported by NOAA/NCEP (for use with NMM core). Fully implemented in ARW Core for use in RAINEX and real- time hurricane forecasting this coming season… –Following animations done by John Michalakes

6. Five-day Hurricane Ivan 12km/4km Moving Nest Run time: 8.6 hours on 64p IBM Power 4 (AFWA), including 20 minutes I/O Two-way interacting nest with high-resolution terrain ingest at leading edge 400 x 301 x 35, dt = 72 sec 331 x 352 x 35, dt = 24 sec Best track

7. Five-day Hurricane Ivan 12km/4km Moving Nest Run time: 8.6 hours on 64p IBM Power 4 (AFWA), including 20 minutes I/O Two-way interacting nest with high-resolution terrain ingest at leading edge 400 x 301 x 35, dt = 72 sec 331 x 352 x 35, dt = 24 sec

8. Five-day Hurricane Ivan 12km/4km Moving Nest Run time: 8.6 hours on 64p IBM Power 4 (AFWA), including 20 minutes I/O Two-way interacting nest with high-resolution terrain ingest at leading edge 400 x 301 x 35, dt = 72 sec 331 x 352 x 35, dt = 24 sec

9. Mesoscale & Microscale Meteorological Division / NCAR Goals Generic interoperability with other ESMF components... WRF + “ocean model” ( + “wave model”) –Hurricane applications – HWRF WRF + 3DVAR / 4DVAR –Decouple requirements WRF + “land model” for regional climate modeling –CLM == CCSM's land model First cut: call CLM as WRF subroutine (CAM-like) –Jimy Dudhia (MMM), Mariana Vertenstein (CGD) –Prototype complete Second cut: couple two ESMF gridded components –Need CLM ESMF component – early 2006 Compare performance “apples-to-apples”

10. Mesoscale & Microscale Meteorological Division / NCAR Work Completed Completed draft of WRF-ESMF Integration Plan –Reviewed by WRF SE working group (WG-2) Completed “stand-alone” component –Finished “init”, “run”, “final”, “set services” –Built trivial “AppDriver” Tested vs. standard WRF –Included in WRF 2.1 release Useful for testing, not useful otherwise…

11. Mesoscale & Microscale Meteorological Division / NCAR Work In Progress Create and populate ESMF “import” and “export” states –Attach references to ESMF_State objects to WRF “domain” objects at top level –Implement operations on ESMF_State objects as new external I/O package via WRF I/O & Coupling API No changes to WRF software framework ESMF-specific code only in new top-level driver and in external I/O package –Customize contents of ESMF_State objects via WRF Registry and WRF namelist –Prototype code complete, testing in progress

12. Mesoscale & Microscale Meteorological Division / NCAR Work In Progress Couple WRF with simple component via simple coupler –“Dummy ocean model” Read SST from file and send to WRF Receive SST from WRF and compare to file data for self-test Coding under way Test to validate prototype code

13. Mesoscale & Microscale Meteorological Division / NCAR Future Plans Couple to other ESMF components –HYCOM, CLM, 3D/4DVAR, LIS (NASA), … Currently limited to “trivial” cases due to lack of support for general curvilinear coordinates in ESMF –Now a high priority for the core team –Will need “moving nests” for coupled hurricane applications –Also, need for “single executable” makes use of ESMF more difficult than existing WRF coupling methods for some models Extension to “PSMF” –Planetary WRF NASA, CalTech, etc.

14. Mesoscale & Microscale Meteorological Division / NCAR Future Plans Maximize coupler re-use by minimizing dependencies on gridded components –Climate and Forecast (CF) metadata conventions www.cgd.ucar.edu/cms/eaton/cf-metadata/index.html Use for all WRF I/O and coupling Specify CF “standard_name” via WRF Registry Use Balaji’s solution for Arakawa grids –Model metadata Time, etc. GO-ESSP: go-essp.gfdl.noaa.gov

15. Mesoscale & Microscale Meteorological Division / NCAR A Few Issues Orthodox Canonical Form Time management Attaching ESMF objects to pre-existing component objects Destruction of ESMF objects

16. Mesoscale & Microscale Meteorological Division / NCAR Orthodox Canonical Form Key best practice for statically-typed object-oriented languages that do not support native garbage collection like C++ (and Fortran90) –Coplien: Advanced C++ Programming Styles and Idioms –Meyers: Effective C++ and More Effective C++ Objects must have an assignment operator and copy constructor ESMF “deep” objects do not have assignment operators –See ESMF Reference Manual Section 8.2 –Some do not have copy constructors, but will eventually

17. Mesoscale & Microscale Meteorological Division / NCAR Orthodox Canonical Form Lack of assignment operators causes confusion even in ESMF demo code! –CoupledFlowDemo.F90 ! Make our own local copy of the clock localclock = clock –Thanks to Nancy for fixing this after I whined… All ESMF deep objects must have assignment operators with uniformly well-defined behavior!

18. Mesoscale & Microscale Meteorological Division / NCAR Options for Deep Object Assignment Option 1: Assignment behaves like standard Fortran assignment – it looks like a deep copy –Easy to explain and use Difference between “deep” and “shallow” is performance Future changes from shallow to deep will not break user code –Difficult to implement efficiently Optimize using copy-on-write idiom via reference counting (Meyers, and many others too) Is it possible to re-use an existing implementation?

19. Mesoscale & Microscale Meteorological Division / NCAR Options for Deep Object Assignment Option 2: Assignment behaves like reference-copy (assignment of Fortran logical unit numbers) –Easier to implement –Not very helpful to users – this can already be done via Fortran pointers! Option 3: Assignment fails –Unless the assignment is a return value from a “Create” call Insane for sure! But still better than nothing

20. Mesoscale & Microscale Meteorological Division / NCAR Initialization of “master clock” Gridded components may already have expertise to initialize their own clocks –Dependencies Time step: horizontal resolution, physical processes, etc. Start and stop time: input and boundary data sets, time step, … Avoid duplicating expertise in couplers/drivers –No “empty” ESMF_Clock (like “empty” ESMF_State) –Instead, pass time information back to driver/coupler through “init” interfaces as exportState metadata –Driver/coupler resolves inconsistencies between components and creates top-level clock (or aborts) –“Model metadata” conventions useful here

21. Mesoscale & Microscale Meteorological Division / NCAR A Few More Issues Attaching ESMF objects to pre-existing component objects –ESMF_VMGetCurrent() – extend to other types –Attach pointers, not “copies” Proposal: pass pointers into “init”, “run”, “final” and let user decide via dummy argument declaration Destruction of ESMF objects –Need inquiry ability: who must delete this? –Need ability to say “it’s OK to let ESMF delete this” (delegation) Must pass in pointers to “Create” or “Set”

22. Mesoscale & Microscale Meteorological Division / NCAR Questions?