1. Jinwon Kim and Paul Ramirez and RCMES Science and IT Teams led by Duane Waliser (JPL), Science Leader Chris Mattmann (JPL), IT Leader Regional Climate Model Evaluation System (RCMES): Combining Observations & IT to Establish Core Climate Model Assessment Capabilities NCPP Workshop, Boulder, Colorado, August 2013

2. RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities Model evaluation is a key step in assessing climate model fidelity and in turn the (un)certainty of climate change impacts. Systematic experimentation and evaluation of GCMs have been undertaken for some time (e.g., AMIP, CMIP, CFMIP), while that for RCMs (e.g., NARCCAP, CORDEX) being more recent and less mature. NASA can provide critical and unique observational data to facilitate RCM evaluations and thus make key contributions to impact assessment process, e.g., the National Climate Assessment (NCA). RCMES was developed to facilitate model evaluations via easy access to observational data, especially from satellite remote sensing, and software tools for calculating and visualizing evaluation metrics.

3. Raw Data RCMED (Regional Climate Model Evaluation Database) RCMET (Regional Climate Model Evaluation Tool) Metadata Data Table Common Format, Native grid, Efficient architecture Common Format, Native grid, Efficient architecture Cloud Database Extractor for various data formats TRMM MODIS AIRS CERES ETC Soil moisture Extract OBS data Extract model data User input Regridder (Put the OBS & model data on the same time/space grid) Regridder (Put the OBS & model data on the same time/space grid) Analyzer Calculate evaluation metrics & assessment model input data Analyzer Calculate evaluation metrics & assessment model input data Visualizer (Plot the metrics) Visualizer (Plot the metrics) URL Users' own software for specific evaluation and analysis Data extractor (Binary or netCDF) Model data Other Data Centers (ESGF, DAAC, ExArch Network) Other Data Centers (ESGF, DAAC, ExArch Network) Assessment modeling RCMES is flexible and open source software Can utilize multiple (distributed) data sources. Easily transferrable to local platforms (Linux, Mac-OS) Open source (via Apache)

4. RCMED Datasets: Satellite retrievals, Surface analysis, Reanalysis, Assimilations * MODIS (satellite cloud fraction): [daily 2000 – 2010] * TRMM (satellite precipitation): 3B42 [daily 1998– 2010]‏ * AIRS (satellite surface + T & q profiles) [daily 2002 – 2010] * CERES and GEWEX-SRB radiation – Surface and Top of the atmosphere * NCEP CPC Rain Gauge analysis (gridded precipitation): [daily 1948 – 2010] * CRU TS 3.1: precipitation, Tavg, Tmax, Tmin [monthly means, 1901 – 2006] * University of Delaware precipitation and temperature analysis * Snow Water Equivalent over Sierra Nevada Mts [monthly 2000-2010] * NASA MERRA Land Surface Assimilation & pressure-level data [daily, 1979-2011] * ERA-Interim (reanalysis): [daily 1989 – 2010] * AVISO sea-level height [1992-2010] * (In progress) CloudSat atmospheric ice and liquid, Satellite-based snow (Himalayas), ISCCP cloud fraction, Fine-scale SST, etc. RCMET Metrics: * Bias (e.g. seasonal means or variance) * RMS error (e.g. interannual variability) * Anomaly Correlation (spatial patterns of variability) * PDFs (likelihoods, extremes and their changes) * Taylor Plots & Portrait Diagrams (overall model performance) * Statistical Tests * User-defined regions (e.g. water shed, desert, sea, political) Datasets and metrics are continuously updated.

5. Not Illustrated Here Arctic & Antarctic Domains RCMES: Combining Observations & IT to Establish Core Climate Model Assessment Capabilities RCMES is being (will be) used for model evaluation in a number of regional climate experiment regions via worldwide collaboration N. America – NARCCAP WCRP CORDEX Regions Ongoing/tested: Africa, South Asia, Middle East – N. Africa, Australia Under arrangement: Caribbean, South America, East Asia, Arctic

6. NARCCAP Multi-decadal Hindcast Evaluation: Surface Insolation Considerable biases with large-scale spatial structures exist in surface insolation fields. Figure. Surface insolation biases against the GEWEX SRB data Model performance varies according to regions and seasons. Figure. Evaluation of the mean & interannual variability in various regions within the conterminous US. Bias (% SRB) Interannual variability (% SRB) Kim et al., 2013, J. Climate, 26, 5698-5715.

7. RCMES' capability to handle multiple model- and/or/both observation datasets allows to visualize model performance as well as uncertainties in the model evaluation due to observational data. Figure. Evaluation of the annual-mean climatology from multiple RCMs and observations against the equal-weight multi- observation ensemble. The figure shows that there exist noticeable uncertainties among widely used observations. Model Evaluation and Observational Uncertainties Kim et al., 2013, J. Climate, 26, 5698-5715. Observational Uncertainty TRMM, CPC, CRU, UDEL, GPCP

8. Construct watershed-mean met data for hydrology model * Transferring the gridded climate model data onto a watershed is the first step in assessing water resources using a bulk hydrology model that runs on watershed-mean met data. 1. Overlay model grid over the watershed area. 2. Calculate the percentage of each grid box contained within the watershed area. This is the weighting factor for calculating the area-mean meteorological data. 3. Calculate the watershed-mean value of a variable P using the weights The shaded area is the 0.5 o - resoln RCM grid boxes that are entirely or partially included in the basin. The Sacramento River basin Map the watershed area onto the RCM domain (Kim et al. 2000, J. Hydromet.) Area mapping Figure. Calculation of area-mean data for an irregularly-shaped watershed from gridded climate model data.

9. Evaluation of extreme events Analysis of long-tailed temperature PDFs in NARR and NARCCAP RCMs: Loikith et al., 2013: Geophys. Res. Lett., in press. Variance/skewness of daily temperature are related with weather extremes. These PDF properties can also be related with regional processes e.g., Positive tail in the LA basin is related with the Santa Ana winds Utilize these PDF characteristics to evaluate RCMs in simulating extreme events

10. * RCMES can be also used to evaluate multiple GCMs for various regions using multiple observation data. * Can be useful, for example, in identifying GCMs most suitable for downscaling. Fig. July precipitation bias: CMIP5 present-day Ens & NARCCAP RCM Ens CMIP5 GCMs - ObservationNARCCAP RCMs - Observation Regional Evaluation of GCMs

11. Summary The Regional Climate Model Evaluation System (RCMES) was developed to facilitate model evaluation via easy access to key observational datasets especially from satellite remote sensing data sets. RCMES is being used in evaluation studies for several regional climate experiments including NARCCAP and multiple CORDEX domains. RCMES includes calculation of a number of metrics for model evaluation. RCMES is continuously improved with additional datasets, more efficient database scheme, and evaluation metrics. RCMES is useful for a wide range of users: Can be obtained and installed as a Virtual Machine image http://rcmes.jpl.nasa.gov/training/downloads Deployable to user laptops and server machines with ease Source Code at Apache Open Climate Workbench http://climate.incubator.apache.org/ Command line interface Browser based interface API to construct your own evaluations

12. Ongoing and Future Efforts Links to ESGF Enable users to import data directly from ESGF for processing. Made-to-order system Distribution of RCMES in a virtual machine package as specified by users. Link climate model data with assessment models Preparation of met forcing data from climate model data, bias correction of met forcing data for specific assessment models. Performance-based variable weighting model ensemble construction Data processing and metrics Handling of very large datasets. Develop metrics for evaluating PDF characteristics (e.g., variance, skewness) Cluster analysis Collaborative Development through Apache Open Climate Workbench Open source effort based on a well known open source software community and team expertise at Apache

13. RCMES: Evaluation of Multi-model Hindcast in the CORDEX Africa jkim@atmos.ucla.edu; duane.e.waliser@jpl.nasa.gov; chris.a.mattmann@jpl.nasa.gov Terrain of the evaluation domain and the annual precipitation climatology Evaluation of the spatial variability of the annual mean precipitation RMSE (% Annual mean)Correlation coefficients Evaluation of the simulated precipitation annual cycle Kim, J., D. Waliser, C. Mattmann, C. Goodale, A. Hart, P. Zimdars, D. Crichton, C. Jones, G. Nikulin, B. Hewitson, C. Jack, C. Lennard, and A. Favre, 2013: Evaluation of the CORDEX-Africa multi-RCM hindcast: systematic model errors. Clim. Dyns, DOI 10.1007/s00382-013-1751-7. Project supported by: NASA: NCA (11-NCA11-0028), AIST (AIST-QRS-12-0002), American Recovery and Reinvestment Act. NSF: ExArch (1125798), EaSM (2011-67004-30224)

14. RCMES: Precipitation Evaluation of Multi-model Hindcast in the CORDEX South Asia – Indian subcontinent Indian subdomain The hindcast and evaluation domain Hindcast domain % of Observation Ensemble Uncertainties in the Observations mm/day Observations RCMs Model biases in simulating spatial variations in the annual-mean climatology R01 R04 R05 R06 R01 R02 R03 R04 Annual cycle evaluation for the northern mountainous region Supported by: NASA: NCA (11-NCA11-0028), AIST (AIST-QRS-12-0002); NSF: ExArch (1125798)

15. Global Climate Projections Regional Downscaling Decision Support RCMES – Regional Climate Model Evaluation System Sponsor Acknowledgements NSF G8 Initiative: ExArch Project NASA: National Climate Assessment (NCA) NASA: Adv. Information. Systems Technology (AIST) NASA: Comp. Modeling Alg. & Cyberinfras. (CMAC) Selected Publications Crichton et al., IEEE Software, 29, 63-71. Hart et al., ICSE 2011 Workshop Software Engineering for Cloud Computing - SECLOUD, Honolulu, HI, May 2011, 43-49, ISBN: 978-1-4503-0582-2, doi: 10.1145/1985500.1985508. Kim et al., 2013a, J. Climate, 26, 5698-5715. Kim et al., 2013b, Climate Dynamics, DOI 10.1007/s00382- 013-1751-7. Loikith et al.,2013, Geophys. Res. Lett., in press. Mattmann et al., 2013, Earth Sci. Informatics, doi 10.1007/s12145-013-0126-2. Waliser et al., 2012, Tech. input for consideration to the 2013 Us National Climate Assessment, 19pp. Whitehall et al., 2012, WMO Bulletin, 61, 29-34. plus a number of IT papers and conference papers http://rcmes.jpl.nasa.gov/