1. Bob Yantosca Software Engineer Atmospheric Chemistry Modeling Group School of Engineering & Applied Sciences Harvard University GEOS–Chem Update and a look ahead to new things Jacob Group Meeting Wednesday, January 23, 2008

2. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 2 1.Model updates since my last group meeting (Dec 2006) 2.Current met field availability for GEOS–Chem 3.GEOS–Chem benchmarking updates 4.GEOS–5 validation and outstanding issues 5.Adoption of Earth System Model Framework and porting of GEOS–Chem to IBM BlueGene architecture 6.Stuff in the pipeline Table of Contents

3. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 3 Model updates since my last group meeting (Dec 2006)

4. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 4 Internal version v7–04–11 was the development version in Dec 2006 Improvements and additions to chemistry –Ability to do chemistry up to the location of the actual tropopause (Philippe, Brendan) --DONE! –Added 2 tracers (SOG4, SOA4) to track the SOA prod from isoprene + OH reaction (Daven Henze) -- DONE! –HO2 uptake by aerosols is now turned off in the SMVGEAR mechanism (Bastien Sauvage) – DONE! Improvements and additions to emissions –Option to use GFED2 biomass emissions for gas phase + aerosol species -- DONE! –Option to use David Streets' regional emissions for China and SE Asia -- DONE! –Option to use EDGAR global NOx, CO, SO2 emissions -- DONE! –Option to use BRAVO NOx, CO, SO2 emissions over Northern Mexico -- DONE! –New module for IPCC future emissions scenarios (primarily for GCAP) -- DONE! –Near-land lightning NOx emissions module – DONE (NOW REPLACED IN v7-04-13!)‏ Updated Hg simulation –More recent Hg simulation updates still need to be standardized – READY FOR INCLUSION! Technical updates –Support for Sun 4100 platform (with AMD/Opteron chipset) – DONE! –Removal of obsolete GEOS-1 and GEOS-STRAT met fields – DONE! –Removal of support for obsolete LINUX_IFC and LINUX_EFC compilers – DONE! –Various minor bug fixes & updates for the GCAP simulation – DONE! Review: state of the model at last grp mtg

5. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 5 v7–04–13 is latest benchmarked internal release TPCORE advection now works with GEOS–5 met! –Yuxuan Wang & Dan Chen are very close to a working TPCORE for GEOS–5 nested grids v8–01–01 (public release w/ GEOS–5) is very close –A 1-year benchmark was done –A couple of things still need attention Model updates since my last group meeting!

6. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 6 GEOS–Chem v7–04–13 is latest internal release –H2 / HD offline simulation (Jaegle group @ UW) –Major bug fix in SMVGEAR! (May & Philippe) v7–04–13 is latest benchmarked internal release

7. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 7 Basic unit of SMVGEAR work = the “block” = 24 grid boxes –Each processor gets 24 G–C grid boxes to work on simultaneously –Abs & rel error tolerance checks are applied to the whole block Internal iterations –SMVGEAR iterates over all of the grid boxes in the block until the absolute & relative error tolerance for the block as a whole is satisfied. –If SMVGEAR does not converge to a solution, it reduces the timestep adaptively and then starts re-iterating over the block once again The problem –Some G–C users reported that SMVGEAR was giving them extremely high Ox and extremely low CO in a particular grid box –This caused the G–C simulation to die w/ an error SMVGEAR Bug in more detail

8. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 8 The cause –It was possible to have a single grid box in the block be very negative, while the overall local & global error tolerance was satisfied –SMVGEAR would then reset the negative value to “zero” –The “zero” concentration value caused problems later on in places where it showed up in the denominator or in a logarithm. The solution (Thanks May and Philippe! )‏ –If any of the 24 grid boxes in the block are negative, then: Revert to the conc's at the end of the last successful internal iteration We reduce the internal iteration timestep and begin again Repeat until convergence If it still does not converge, then stop with an error SMVGEAR Bug in more detail

9. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 9 GEOS–Chem v7–04–13 is latest internal release –H2 / HD offline simulation (Jaegle group @ UW) –Major bug fix in SMVGEAR! (May & Philippe) –Improved lightning NOx emissions algorithm (Lee Murray) Eliminate NOx emissions per path length Scale tropics to 260 moles N/flash (cf. Randall Martin) Eliminate top-down scaling New OTD/LIS local redistribution (cf. Murray et al 2007 in prep) –Bug fix in variable tropopause Cap tropopause at 200 hPa poleward of 60S and 60N v7–04–13 is latest benchmarked internal release

10. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 10 Getting TPCORE to work w/ GEOS–5 met Pressure at grid edges: Pe(IJL) = A(L) + [ B(L) * Psurf(IJ) ] Pressure at grid centers: Pc(IJL) = [ Pe(IJL) + Pe(IJL+1) ] / 2 Near the surface, the hybrid grid has a “sigma- like” terrain-following coordinate. Near the model top, the hybrid grid has fixed- pressure levels. The pressure at each level is the same for all latitudes & longitudes. Pros: Results in much smoother winds near the top of the model, better STE.

11. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 11 From the GEOS–5 file spec document (p.14) –“In the GEOS–4 eta files, one could compute the pressure on the edges by using the “ak” and “bk” values and the surface pressure; once the edge pressures were known, they could be used to compute the average pressure in the layer.” –“In GEOS–5, the full 3-dimensional pressure variables are explicitly provided at both layer centers (PLijl) and layer edges (PLEijl). As of this writing the pressures reported are on a hybrid-sigma coordinate, and could be obtained from the “ak-bk” relationship. But this may change in the future and so users should rely on the reported 3- dimensional pressures and not attempt to compute them from “ak” and “bk”. Getting TPCORE to work w/ GEOS–5 met

12. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 12 Our first attempts to modify the TPCORE transport code for GEOS–5 were not very successful. This tagged Ox simulation indicates very little mixing even after 7 years.

13. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 13 As a result of a telecon between Harvard and GMAO on 10/30/07 – GMAO agreed to give us the “A’s” and “B’s” which define the GEOS-5 grid. In this way, we can use the existing TPCORE code without having to modify it. Everyone agreed that this was the fastest solution (particularly since Harvard needs to get a working GEOS-5 code ASAP for ARCTAS support). – NASA/SIVO has a newer version of the TPCORE transport code that takes mass fluxes instead of winds. However, this version is fully ESMF-ized. To use it in the current GEOS-Chem version (aka “GEOS-Chem classic”) would require more time than we have before ARCTAS. – However, the newer transport code will be useful as we create the new, ESMF- ized GEOS-Chem (for porting to MPI and eventually to the BlueGene).

14. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 14 TPCORE now works with GEOS–5 met fields! GMAO provided us with the A’s and B’s that are used by TPCORE to compute the pressure at altitude from the surface pressure. Ox tends to steady state after 10 years! NOTE: This Ox spinup was used to create the initial conditions for the 1-year benchmark simulations with GEOS–5.

15. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 15 GEOS–Chem v8–01–01 is very close –This will be the first version that is compatible with the GEOS–5 met fields (hence the version # change). We will also update our web site for the public release –New GEOS–Chem user manual –Wiki will be publicly visible and editable It’s currently only internally visible at Harvard I will speak as to some of the ongoing GEOS–5 issues in a bit.. v8–01–01 will be next public release

16. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 16 Current met field availability for GEOS–Chem

17. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 17 GMAO Met Field Comparison Chart Current met field availability for GEOS–Chem 2004 – onward + 30 yr reanalysis (MERRA)‏ 1985 – present (ending in Feb 2007)‏ 2000 – 20021996 – 19981985 – 1995Temporal Coverage 2 x 2.5 4 x 5 + native 2 x 2.5 4 x 5 1 x 1 nested 2 x 2.5 4 x 5 2 x 2.5 4 x 5 Regridded to HDF4—EOSHDF4–EOS Binary Native File Format 0.5 x 0.6671 x 1.251 x 12 x 2.5 Native grid 72 hybrid55 hybrid48 sigma26 sigma20 sigma# of levels GEOS–5GEOS–4GEOS–3GEOS– STRAT GEOS–1 Past Pres, Fut. O B S O L E T E !! GEOS–3 and GEOS–4 are both still supported in GEOS–Chem. GEOS–4 is currently still heavily used in GEOS–Chem research. However, GMAO has now turned off the GEOS–4 data stream. GEOS–5 is now the only dataset currently being produced by GMAO. We also have GCAP met available for several years but these are not shown on the tables below.

18. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 18 Extra Slides Current met field availability for GEOS–Chem 2007 2006 2005 GEOS5 4x5 GEOS5 2 x2.5 GEOS5 0.5 x 0.667 nested China GEOS4 4x5 GEOS5 0.5 x 0.667 nested NA GEOS 4 2x25 NOTE: This plot is now on the GEOS–Chem web page under “Simulation Years” Met field date GEOS–4 ends on 3/31/07. We have late-look data thru 2006.

19. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 19 Current met field availability for GEOS–Chem 2007 2006 2005 GEOS5 4x5 GEOS5 2 x2.5GEOS5 China GEOS4 4x5 GEOS5 NA GEOS 4 2x25 2004 2003 2002 GEOS4 4x5 GEOS4 2x25 GEOS3 4x5 GEOS3 2x25 GEOS4 1 x 1.25 NOTE: GEOS-5 data exists for 2004, we just haven’t processed it yet GEOS-3 ends 10/31/02 Met field date

20. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 20 Current met field availability for GEOS–Chem GEOS5 4x5 GEOS5 2 x2.5 GEOS4 4x5 GEOS5 NA GEOS4 4x5 GEOS3 4x5 GEOS3 2x25 GEOS4 4x5 GEOS4 2x25 GEOS3 4x5 GEOS3 2x25 2002 2001 2000 GEOS3 1x1 nested NA GEOS3 1x1 nested China GEOS-4 continues back to 1985GEOS–3 begins Jan 2000 (some non-std data for 1998 exists)‏ Met field date

21. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 21 GEOS ‒ 5 has more levels in the PBL Vertical levels up to 4 km – PBL GEOS–3 GEOS–4 GEOS–5 GEOS–5 has much finer resolution in the PBL!

22. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 22 GEOS–5 also has more levels in the lower FT GEOS–3 GEOS–4 GEOS–5 Vertical levels – up to 30 km

23. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 23 GEOS–Chem benchmarking updates

24. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 24 Example: Cold Bay, Alaska CO (ppbv)‏ Even updates that seem “minor” at first glance can have a major impact on GEOS–Chem results! 20 ppbv cf. Rynda Hudman Upshot: We have to keep track of what changes go into G–C. For changes that affect the full-chemistry simulation, we need to perform 1-year benchmark simulations. We didn’t do such a good job of that in the recent past. Note spread of different G-C versions w/r/t the observations!

25. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 25 GEOS–Chem Version History (1-yr benchmarks) Slide from Rynda Hudman Changes are w/r/t the previous version

26. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 26 A hard object lesson was learned: –We cannot rush revisions into the code quickly and then “pray to God” that it will all work the first time thru … We now must be more diligent in benchmarking G–C –Especially for versions that contain revisions which could have a major impact on the full-chemistry simulation We ask for your understanding –“It’ll be ready when it’s ready …” –We know that delays are sometimes frustrating –Sometimes we encounter “open-ended” bugs or problems that are hard to solve quickly (cf. SMVGEAR bug fix in v7–04–13)‏ –However, It’s better to take more time and ensure the validity of the G–C version than to find out a major flaw with it later on Updates for GEOS–Chem benchmarking

27. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 27 1-month benchmarks will be done for each internal & public version (GEOS–4 and GEOS–5)‏ When do we do a new 1-yr benchmark simulation? –For updates to emissions Anthro, biofuels, biomass, lightning, etc. –For changes to chemical mechanism Reaction rates Photolysis –For dynamical changes New met fields (e.g. GEOS–4 vs. GEOS–5) New convection and/or advection schemes Updates for GEOS–Chem benchmarking

28. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 28 IDL codes for creating plots from 1-month and 1-year benchmarks have been overhauled (L. Zhang, Bob Y.)‏ 1-yr benchmarks now include the following plots (3 models vs. observations)‏ –Profiles of differences along 15S and 42N –Difference maps at surface and 500 hPa –Seasonal cycle –Model ozone profiles vs. sondes –Seasonal cycles of model ozone and CO vs. observations –Model CO vs CMDL and ship track observations –Ozone & CO vs. MOZAIC observations –Ozone & CO vs. observations from various aircraft missions Updates for GEOS–Chem benchmarking

29. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 29 1-month benchmarks now include the following plots (this version vs. previous version)‏ –Frequency distribution –J-Value ratios at surface and 500 hPa –Tracer ratios at surface and 500 hPa –Tracer absolute difference at surface and 500 hPa –Profiles of differences along 15S and 42N –Tracer concentration maps –Budget –Emissions totals Benchmark plots will be accessible from our FTP & web site Updates for GEOS–Chem benchmarking

30. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 30 GEOS–5 validation and outstanding issues

31. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 31 A 1-yr benchmark simulation was done with GEOS–Chem v8–01–01 driven with GEOS–5 met –A few areas still need attention: Convection –GEOS–5 cloud convection is weaker than GEOS–4? Lightning –Less LNOx produced w/ GEOS–5 –Lack of lightning over certain areas (e.g Central Asia) Wet deposition (rainout & washout) –Differences in precip fields between GEOS–4 and GEOS–5 GEOS–5 Outstanding Issues

32. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 32 Strong convection; wet season v7-04-12-Run2 (geos4); v7-04-13 (geos4); v8-01-01 (geos5); data Dry season at these stations Note that the GEOS–5 Ox profiles are very similar to the simulations done with GEOS–4 up to 200 hPa, but diverge higher up. This can be an indication that cloud convection in GEOS–5 is weaker than in GEOS–4. 200 hPa Plot by Inna Megretskaia

33. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 33 GEOS–4 vs. GEOS–5 cloud mass fluxes NOTE: original unit of GEOS–4 cloud mass flux is Pa/s, we have converted to kg/m2/s to compare w/ GEOS–5. No cloud mass flux below ~1 km in GEOS–5! Altitude (km) 0.8 km

34. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 34 GEOS–4 puts lightning in Central Asia, but GEOS–5 doesn’t. Figure from Lee Murray GEOS–4 LNOx (CTH)GEOS–5 LNOx (CTH) We are currently looking into this! This is most likely due to lower cloud-top heights in GEOS–5. Global LNOx total w/ GEOS–5 met: ~ 5.4 Tg N/yr Global LNOx total w/ GEOS–4 met: ~ 6 Tg N/yr 0 1.2e–04 flash/min/km2

35. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 35 v7-04-12-Run2 (geos4); v7-04-13 (geos4); v8-01-01 (geos5); data This plot shows the effect that the lower lightning NOx total in GEOS–5 has on the concentrations of Ox at midlatitudes. Ozone seasonal cycle at 4 stations and 3 altitudes Note: GEOS–5 (blue line) is lower than GEOS–4 (red, green lines) in summer! Ox (ppbv) Plot by Inna Megretskaia

36. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 36 v7-04-12-Run2 (geos4); v7-04-13 (geos4); v8-01-01 (geos5); data Plot by Inna Megretskaia Ozone seasonal cycle at 4 stations and 4 altitudes Ox (ppbv)

37. QQ: GEOS–Chem Precipitation Fields (in wet dep module) NOTE: The GEOS–5 precipitation is much higher at about 1km than in GEOS–4! Also note GEOS– 5 has more conv precip over tropics! Hongyu and Bob Y. are looking at the wet deposition in G–C with Rn-Pb-Be simulations. This is a work in progress.

38. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 38 Adoption of ESMF & MPI and Porting of GEOS–Chem to IBM BlueGene/L architecture

39. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 39 What is ESMF? –It’s a software framework for earth science models that is being implemented at NASA and elsewhere –http://www.esmf.ucar.edu What ESMF will do: –It will make it easier to connect “pieces” of the same model together –It will make it easier to take “pieces” of one model and mix them with “pieces” from other models. –It connects “pieces” of models together regardless of which language the models were written in (Fortran, C, C++, etc.)‏ –It will probably eventually reduce model development time (however, some up-front development will be needed)‏ What ESMF will not do: –It will NOT improve the quality of the science of a model ESMF Overview Summary

40. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 40 Diagnostics With ESMF, you can reduce a CTM to a collection of “pluggable” black box “Components” with standard inputs & outputs In the future, GEOS–Chem will be a configuration of ESMF-compliant components instead of a standalone model NOTE: All of the data that flows into a component goes in through the “front door”, I.e. through argument lists. No more global common blocks! ESMF library functions pass data into & out of components!

41. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 41 Harvard/SEAS has recently purchased an IBM BlueGene/L –Several research groups in SEAS are moving to this platform Earthquake fault & thrust modeling Protein folding Quantum chemistry –Advantage: provides a high-quality computational platform to several research groups –Advantage: users outside Harvard would also be able to take advantage of this platform We were asked by the Dean of IT to consider porting GEOS–Chem to the BlueGene –Philippe & I have sat in on a couple of BlueGene workshops Porting GEOS–Chem to BlueGene

42. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 42 BlueGene/L is a radically new supercomputer architecture –4 of the Top 10 fastest computers are BlueGene’s (top500.org)‏ –LLNL & DOE use BG’s to “have a blast”, among other things Why BlueGene? –Need lots of computing power to solve today’s complex problems in the interdisciplinary sciences –Advantage: massively parallel machine Think big: you have the opportunity to submit a job using hundreds or thousands of cores! (core = processor)‏ Each small task is assigned to its own core I/O can be done in parallel –BG architecture minimizes power consumption Porting GEOS–Chem to BlueGene

43. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 43 IBM BlueGene/L Architecture K. Jordan, IBM Take notice! Users log in to the “front end” (which is a more traditional Unix environment) and submit jobs to the BG/L via the LoadLeveler queuing software.

44. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 44 IBM BlueGene/L Architecture NOTE: One must break down arrays into chunks that are less than 256 MB! Each core has 256 MB of memory and does not see the memory of the other cores. Cannot use OpenMP, must use MPI! Harvard SEAS has a 2-rack BG/L machine K. Jordan, IBM

45. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 45 IBM BlueGene/L Architecture K. Jordan, IBM Very complex design!

46. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 46 Radical rewriting of GEOS–Chem will be necessary –Cannot use much of existing G–C due to BG/L memory per node Existing G–C requires 2+ GB of shared memory –Parallel file system implies file format will have to be switched from binary to netCDF or HDF Cannot read binary sequential files on BlueGene –Because we will want to interchange codes w/ NASA, we will have to pass data around via calls to ESMF/MAPL libraries Implies familiarity with ESMF programming interface But all is not lost … –We can start with existing routines from NASA/SIVO and/or GMI –We can rely on the expertise of the folks at NASA Challenges in porting GEOS–Chem to B/G

47. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 47 Technical things –ESMF and MAPL libraries must be installed on BG/L –Learning curve! Bob & Philippe will have to learn many new things, including MPI programming and ESMF & MAPL API’s –We must also communicate these new things to the G–C user base –Existing ESMF / MPI etc. documentation is not very good Human Resources –Perhaps up to ½ of Bob's or Philippe's time will be spent on porting –More time will be spent on strictly software issues –Significant up-front work may be required before we get results Challenges in porting GEOS–Chem

48. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 48 Testing concerns –We cannot move to the IBM BlueGene/L in one jump –We must port the model to a more traditional Unix environment first –On the BlueGene/L, you cannot run the model on one processor. You have to use at least 32. This makes single vs. multi-processor testing impossible on the BlueGene/L. Scalability –At this point we don’t know how scalable a new G–C code would be on the BlueGene –If the code does not scale well on BlueGene then there is no point in porting there –Better to develop the ESMF-ized version of G–C on a more traditional computational environment first. If there are scalability problems then they will more than likely show up at this stage. Challenges in porting GEOS–Chem

49. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 49 Stuff in the pipeline

50. Jan 23, 2008Bob Y – Jacob Group Meeting – Page 50 List of revisions, in no particular order Add glyoxal etc. rxns to chemistry mechanism (May Fu, Hong Kong Polytechnic U.) Updated Hg simulation (Noelle Selin, now @ MIT) Interface CASA model with mercury simulation (Nicole Smith-Downey, Harvard)‏ Emissions scale factors (A. van Donkelaar, Dalhousie) GFED2 8-day biomass emissions (P, Kasibhatla, Duke) – now in testing Aerosol phase transition (J. Wang, U. Nebraska @ Lincoln) Updated ozone quantum yields for photolysis from JPL Updated EPA/NEI99 emissions w/ California transport fix (available for use now but not in std code) CAIR (EPA 2020) ship emissions, accounting for offshore limits (P. LeSager) For ARCTAS Construction of near-real-time simulation (Bob Y.) Things to be added into GEOS–Chem