1. So you want to use GCHP? IGC9 GEOS-Chem High Performance Model Clinic
Sebastian Eastham and Lizzie Lundgren
May 6, 2019

2. A note about this presentation
Check the notes! Anything said during this presentation which isn’t on the slides should be in the presentation notes.
A glossary of terms is included at the end!
This presentation assumes use of GCHP Always check GCHP documentation for the most up-to-date information.

3. GCHP works now!
This presentation will give you information on where we are, where we are going, what you need, and how to get started.
But remember: you can use GCHP today:
GCHP undergoes the same version control as GCC (stable)
GCHP is benchmarked by the GCST (every X.Y.0 version)

4. GCHP Today is GCC but with...
Grid resolution flexibility, without recompiling
Cubed-sphere transport
Improved scalability/speed
Takes advantage of:
FV3 offline advection
ESMF for core infrastructure
MAPL layer between GEOS-Chem and ESMF
Eastham et al. (2018, GMD)

5. Stretched (“nested”) grids
GCHP Tomorrow
Now
Mid-2019
2020
2021?
TBD
New MAPL:
gfortran compatibility
Faster I/O
Lower memory footprint
Cloud GCHP
CMake
Improved error handling
Containers
Stretched (“nested”) grids
Flux-based transport
Ongoing integration of GMAO improvements throughout...

6. What we will cover in this clinic
What GCHP needs
Hardware
Software
Workflow: GCHP versus GCC
Download and create run directory
Compile
Configure run
Run
Develop
Analyze
Resources

7. What we will NOT cover in this clinic
Software library installation but...
Environment setup
Basic information covered in the GCHP wiki tutorial (
For any of this information: Stop by the GCST Help Desk during IGC9 M/Tu/W 1-3 pm
Geological Museum 103A

8. What GCHP needs: hardware concepts
Memory requirements grow with resolution*
All requirements can be distributed across all nodes..
..but bigger simulations will want better network fabric
Think carefully about data needs - the closer your I/O, the better
Node A 24 cores 128 GB mem.
Infiniband 56+ Gb/s
TCP-IP 1 Gb/s 10 Gb/s
GCHP can be run across a network, using multiple nodes
There are two main considerations:
Do I have enough cores and memory for this resolution? In current version of GCHP, memory requirements grow with both resolution and the number of cores. You might find yourself “memory limited” rather than “core limited” - asking for more cores than you need so that you can get access to more memory. This is expected to be improved with the update to the new version of MAPL.
How fast is my network? Your network fabric may be limiting you. If you are thinking about upgrading your cluster or buying a new one, consider a high-speed network fabric This could be a higher-performance ethernet (TCP-IP here), or one of the Infiniband versions. You really do want at least 10 Gb/s.
Note that faster networks are generally also good for GC-Classic! If your compute nodes can’t talk to your data nodes because of slow network speeds, your simulation speed could be affected. However, you are more likely to fall foul of slow read/write at the disk level.
Node B 24 cores 128 GB mem.

9. What GCHP needs: hardware concepts
GCHP can be run across a network, using multiple nodes
Two main considerations:
Do I have enough cores and memory for this resolution? In current version of GCHP, memory requirements grow with both resolution and the number of cores. You might find yourself “memory limited” rather than “core limited” - asking for more cores than you need so that you can get access to more memory. This is expected to be improved with the update to the new version of MAPL.
How fast is my network? Your network fabric may be limiting you. If you are thinking about upgrading your cluster or buying a new one, consider a high-speed network fabric This could be a higher-performance ethernet (TCP-IP here), or one of the Infiniband versions. You really do want at least 10 Gb/s.
Note that faster networks are generally also good for GC-Classic! If your compute nodes can’t talk to your data nodes because of slow network speeds, your simulation speed could be affected. However, you are more likely to fall foul of slow read/write at the disk level.

10. What GCHP needs: hardware examples
Resolution
Requirements
Minimum
Recommended*
C24 (~4°)
6 cores (1 node) 60 GB memory
24 cores (1 node) 80 GB memory
...
C90 (~1°)
48 cores (2 nodes) 256 GB memory
96 cores (4 nodes) 512 GB memory
C360 (~0.25°)
cores (10+) 2048 GB memory
The number in parentheses is the number of nodes
These numbers are recommendations only
*Recommended values based on where memory requirements grow with core count

11. What GCHP needs: software
Requirement
...for GC-Classic?
...for GCHP?
GNU or Intel Fortran compiler
Yes
HDF-5 and zlib
NetCDF-C
NetCDF-Fortran
MPI implementation No
EMSF v7+, MAPL, FV3
Included in GCHP

12. Workflow Comparison Step 1: Download and create run directory
Two nested source code repositories
Create GCHP run directory from source code, not Unit Tester
Use interactive script, not edit a text file
GCC will do this in a future update
geos-chem
gchp
(includes MAPL, ESMF, FV3)
See slides at end of presentation for more detailed explanations of steps.

13. Workflow Comparison Step 2: Compile
Like GCC, build from run directory
Multiple build options
Initial full build required; subsequent build is change-dependent
Type make or make help to display options and use cases
ESMF
MAPL
FV3
GEOS-Chem Core
make build_all ✓
make build_mapl
make build_core

14. Workflow Comparison Step 3: Configure run
More config files in GCHP than GCC
Set common run-time settings in driver file runConfig.sh
Simulation start/end time
Cubed-sphere grid resolution
Diagnostics collection frequency, duration, mode
# cores
Configure emissions in HEMCO_Config.rc, but also ExtData.rc
Unlike GCC diagnostics:
GCHP HISTORY.rc includes emissions
Diagnostic frequency is HHmmSS, not YYYYMMDD HHmmSS

15. Workflow Comparison Step 4: Run
Assess resource needs - # cores, # nodes, memory
Adapt run script from runScriptSamples/
Non-system-specific in gchp.local.run
SLURM-specific in gchp.run
Advanced multi-run option for monthly diagnostics
Resource management tips:
# cores used (runConfig.sh) can be less than # cores requested (gchp.run) to maximize memory per core

16. Workflow Comparison Step 5: Develop
Develop source code same as GCC
Key exceptions:
main.F: driver files in GCHP (Chem_GridCompMod, gigc_chunk_mod)
HEMCO I/O and History/: I/O handling in MAPL (ExtData, History)
tpcore: advection in FVdycoreCubed_GridComp/
Key concepts:
Same GEOS-Chem states as GCC (State_Chm/Met/Diag)
State array grids different per core (regional subset)
GCHP also has MAPL states (Imports, Exports, Internal State)
GEOS-Chem and MAPL states exchange data via pointers

17. Workflow Comparison Step 6: Analyze
Diagnostics data in OutputDir/
Restart and emissions output are vertically flipped
Use gcpy benchmark code for data comparisons
Same functions for GCC and GCHP
Plot on cubed-sphere, or regrid to any grid/resolution
See gcpy/examples/compare_diagnostics.ipynb
Use NASA GISS Panoply netcdf viewer for raw data
Format not compatible until after upcoming MAPL version update

18. Relax, you are not alone!
Create issues on GitHub to get help from GCST and beyond
Join the GCHP Working Group and post your project
Join the Slack workspace to chat with users
Read and contribute to the GCHP wiki
Contact the GEOS-Chem Support Team

19. Questions/Comments/Discussion

20. Top 10 list of time-saving tips
Only recompile code you changed (use make help)
Changing simulations does not require recompilation (copy geos)
runConfig.sh overwrites settings in other config files
Do not simply run the executable; use or adapt run scripts provided
Run-time MAPL/ESMF errors are nearly always from bad config
CAP error is usually bad simulation dates; check runConfig.sh
ExtData error is input; set MAPL_DEBUG_LEVEL to 20 in runConfig.sh
MAPL History error is problem with diagnostics; check HISTORY.rc
Search for “making install” in compile.log to find MAPL build errors
Create GCHP GitHub issue if you think you need to edit MAPL or ESMF

21. Glossary of terms GCHP GEOS-Chem High Performance GCC
GEOS-Chem Classic
FV3
Finite Volume for Cubed Sphere Transport. Performs transport calculations on a cubed-sphere grid. From NASA GMAO.
MAPL
Model Analysis and Prediction Layer; connects components (e.g. transport, I/O, chemistry) together. From NASA GMAO.
MPI
Message Passing Interface. System underlying high-speed communication between computational nodes.
ESMF
Earth System Modeling Framework. The code framework on which MAPL is based. Maintained independently.
SLURM
Open-source job scheduler for Linux and Unix-like kernels
Network fabric
The hardware on which your network is based. Can be ethernet (typically 1-10 Gb/s) or Infiniband (typically 56 Gb/s or higher).
gcpy
Python package for GEOS-Chem data analysis.

22. Detailed Workflow Comparison Step 1: Download and create run directory
GCC
GCHP
clone geoschem/geos-chem
clone geoschem/geos-chem-unittest
clone geoschem/gchp (as subdirectory ‘GCHP’)
go to UT/perl directory
go to GCHP directory
edit CopyRunDirs.input: source path, target, etc
type ./createRunDir.sh for interactive rundir creation
type ./gcCopyRunDir
go to run directory
type ./setEnvironment path/to/envfile

23. Detailed Workflow Comparison Step 2: Compile
GCC
GCHP
Go to run directory
source path/to/envfile
type ‘make build_all’
type make command, e.g. ‘make –j4 mpbuild’

24. Detailed Workflow Comparison Step 3: Configure run
GCC
GCHP
go to run directory
edit input.geos
edit runConfig.sh
-> when sourced at run-time, updates many files,
including input.geos
edit HEMCO_Config.rc
edit HEMCO_Config.rc and ExtData.rc
edit HISTORY.rc for non-emissions diagnostics
-> collections, fields, frequency, duration, etc
edit HISTORY.rc for ALL diagnostics
-> collections and fields only!
(freq/dur/mode set from runConfig.sh!)
Edit HEMCO_Diagn.rc for emissions diagnostics
-> all uncommented entries are output
-> only entries in HISTORY.rc are output

25. Detailed Workflow Comparison Step 4: Run
GCC
GCHP
Go to run directory
Check that config files are correct
Option to run interactively:
source environment file
type ./geos
copy runScriptSamples/gchp.local.run to rundir
type ./gchp.local.run
NOTE: You must have enough memory and cores available!
Option to submit job:
Copy an existing run script to rundir
Edit as needed to request resources
Sumit to your local cluster job resource manager, e.g. sbatch gcc.run
Copy an existing run script to rundir, or adapt an example from runScriptSamples/
Submit to your local cluster job resource manager, e.g. sbatch gchp.run

26. Detailed Workflow Comparison Step 5: Develop
GCC
GCHP
High-level calls and order of components in:
main.F
GCHP/Chem_GridCompMod.F90
GCHP/gigc_chunk_mod.F90
Edit state variables and non-transport components in GEOS-Chem repository
Same as GCC
Edit transport in GEOS-Chem repository
Edit transport in GCHP/FVdycoreCubed_GridComp, main file AdvCore_GridCompMod.F90
Edit core I/O handling in GEOS-Chem repository
Edit core I/O handling in GCHP/Shared/MAPL_Base

27. Detailed Workflow Comparison Step 6: Analyze
GCC
GCHP
Verify output present:
All output in run directory
Restart: GEOSChem.Restart.*.nc4
Emissions: HEMCO_diagnostics.*.nc
bpch: trac_avg.*
netcdf: GEOSChem.collection.*.nc4
Restart in run directory: gcchem*.nc
All diagnostic output in OutputDir/
Emissions: GCHP.Emissions.*.nc4
Other diagnostics: GCHP.collection.*.nc4
Analyze GCC output data (bpch or nc) with python;
IDL GAMAP no longer supported
Analyze GCHP output data with python;
IDL GAMAP does not support cubed-sphere
Use gcpy benchmark comparison tools for GCC:
Regrid lat-lon to any resolution
Level maps and zonal mean
Use gcpy benchmark comparison tools for GCHP:
Regrid cubed-sphere to any resolution
Regrid cubed-sphere to lat-lon
Many python libraries available for data analysis.
The same libraries can be used for GCHP data.