Keith Dixon 1, Katharine Hayhoe 2, John Lanzante 1, Anne Stoner 2, Aparna Radhakrishnan 3, V. Balaji 4, Carlos Gaitán 5 “Perfect Model” Experiments: Testing Stationarity in Statistical Downscaling (NCPP Protocol 2) IS PAST PERFORMANCE AN INDICATION OF FUTURE RESULTS ? DRC Inc. 4 Princeton Univ. 5 Univ. of Oklahoma 2013 Workshop on Quantitative Evaluation of Downscaled Data

Goals of this presentation 1. Define the ‘stationarity assumption’ inherent to statistical downscaling of future climate projections. 2. Present our ‘perfect model’ (aka ‘big brother’) approach to quantitatively assess the extent to which the stationarity assumption holds. 3. Illustrate with a few examples the kind of results one can generate using this evaluation framework (Anne Stoner will show more detail next…) 4. Introduce options to extend and supplement the method (setting the hurdle at different heights). 5. Invite statistical downscalers to consider testing their methods within the perfect model framework. 6. Garner feedback from workshop participants.

This project does not produce any data files that one would use in real world applications. The aim at GFDL is to gain knowledge about aspects of commonly used SD methods (& GCMS), and to communicate that knowledge so that better informed decisions can be made.

REAL WORLD APPLICATION: Start with 3 types of data sets … lacking observations of the futureCompute transform functions in training step (1 example below)Produce downscaled refinements of GCM (historical and/or future)Skill: Compare Obs to SD historical output (e.g., cross validation)Cannot evaluate future skill -- left assuming transform functions apply equally well to past & future -- “The Stationarity Assumption” TARGET PREDICTORS

“PERFECT MODEL” EXPERIMENTAL DESIGN: Start with 4 types of data sets – Hi-res GCM output as proxy for obs & coarsened version of Hi-res GCM output as proxy for usual GCM Proxy for observations in real-world application Proxy for GCM output in real-world application

Daily time resolution ~25km grid spacing 194 x 114 grid 22k pts

~200km grid spacing 64:1

~64 of the smaller 25km resolution grid cells fit within the coarser cell

Daily time resolution ~25km grid spacing 194 x 114 grid

Follow the same interpolation/regridding sequence to produce coarsened data sets for the future climate projections

“PERFECT MODEL” EXPERIMENTAL DESIGN: Compute transform functions in training step (1 example below)Produce downscaled output for historical and future time periods Can directly evaluate skill both for the historical period and the future using the Hi Res GCM output as “truth” – Test Stationarity QuantitativeTests of Stationarity: The extent to which SKILL computed for the FUTURE CLIMATE PROJECTIONS is diminished relative to the SKILL computed for the HISTORICAL PERIOD

GFDL-HiRAM-C360 experiments All data used in the perfect model tests were derived from GFDL-HiRAM-C360 (C360) model simulations conducted following CMIP5 high resolution time slice experiment protocols. We consider a pair of 30-year long “historical” experiments ( ), labeled H2 and H2 here… For the period , we make use of a pair of 3 member ensembles -- a total of six 10-year long experiments, identified as the “C” and “E” ensembles to the right (C warms more than E)...

Same GCM used to generate 2 sets of future projections (3 members each) model climatology projection “C” (3x10yr) projection “E” (3x10yr)

Q: How High a Hurdle does this Perfect Model approach present? A: Varies geographically, by variable of interest, time period of interest, etc.

Coarsened data is slightly cooler with Slightly smaller std dev. … not too challenging August Daily Max Temperatures for a point in Oklahoma (2.5C histogram bins)

approx. +7 C mean warming August Daily Max Temperatures for a point in Oklahoma (2.5C histogram bins)

August Daily Max Temperatures for a point just NE of San Fancisco Coarsened data is more ‘maritime’ & HiRes target is more ‘continental’. Presents more of a challenge during the historical period than the Oklahoma example. (2.5C histogram bins)

August Daily Max Temperatures for a point just NE of San Fancisco (2.5C histogram bins)

Note that in several locations there is a tendency for ARRM cross-validated downscaled output to have too low standard deviations just offshore and too high std. dev. over coastal land Map produced by NCPP Evaluation Environment ‘Machinery’

NEXT: A sampling of results…  Intended to be illustrative… not exhaustive, nor systematic.  Anne Stoner (TTU) will show more.

Area Mean Time Mean Absolute Downscaling Errors Σ | (Downscaled Estimate – HiRes GCM) | (NumDays) Start with a summary: ARRM downscaling errors are larger for daily max temp at end of 21 st C than for

Mean Absolute downscaling Error (MAE) during Σ | (Downscaled Estimate – HiRes GCM) | (60 * 365) Geographic Variations: Downscaling MAE for

Mean absolute downscaling error during “E” Projections (3 member ensemble) Geographic Variations: MAE pattern for “E” projections (+5,4C)

Mean absolute downscaling error during “C” Projections (3 member ensemble) Geographic Variations: MAE pattern for “C” projections (+7,6C)

Mean Climate Change Signal Difference (ARRM minus Target) “C” Projection Geographic Variations: Bias pattern for “C” projections (+7,6C)

J F M A M J J A S O N D Red = “C” Blue=“E” J F M A M J J A S O N D Entire US48 Domain Looking at how well the stationarity assumption holds in different seasons Where and when ratio=1.0, the stationarity assumption fully holds ( i.e., no degradation in mean absolute downscaling error during vs. the period using in training.) tasmax

…comparing near-coastal land vs. interior Looking at how well the stationarity assumption holds in different seasons

J F M A M J J A S O N D Blue= coastal SC-CSC Black = full US48+ domain Green = interior SC-CSC Looking at how well the stationarity assumption holds in different seasons “C” ensemble

tasmax A clear intra-month MAE trend in some but not all months Looking at how well the stationarity assumption holds in different seasons

“sawtooth” has lower values in cooler part of month, higher in warmer part of month “C” ensemble tasmax

Options for extending this ‘perfect model-based’ exploration of statistical downscaling stationarity More SD Methods More Climate Variables & Indices Different HiRes Climate Models Different Emissions Scenarios & Times Use of Synthetic Time Series Alter GCM-based data in known ways to ‘Raise the Bar’ Mix & Match GCMs’ Target & Predictors ?? More Ideas ??

GFDL-HiRAM-C360 experiments Data sets used in the perfect model tests are being made available to the community. This enables others to use our exp design to test the stationarity assumption for their own SD method. Also, we invite SD developers to consider collaborating with us, so that their techniques & perspectives may be more fully incorporated into this evolving perfect model-based research & assessment system. For more info, visit &

In other words, by accessing these types of files, folks can generate their own SD results and test how well the stationarity assumption holds for their favorite SD method. from us

In other words, by accessing these types of files, folks can generate their own SD results and test how well the stationarity assumption holds for their favorite SD method. from us SD method of choice from us

Obviously, one does not need to perform tests using all 22,000+ grid points. Below we indicate the locations of a set of 16 points we’ve used for some development work. (color areas are 3x3 with grid point of interest in the center)

 Anne Stoner and Katharine Hayhoe with more ‘perfect model’ analyses…

Keith Dixon 1, Katharine Hayhoe 2, John Lanzante 1, Anne Stoner 2, Aparna Radhakrishnan 3, V. Balaji 4, Carlos Gaitán 5 “Perfect Model” Experiments: Testing Stationarity in Statistical Downscaling (NCPP Protocol 2) IS PAST PERFORMANCE AN INDICATION OF FUTURE RESULTS ? DRC Inc. 4 Princeton Univ. 5 Univ. of Oklahoma 2013 Workshop on Quantitative Evaluation of Downscaled Data

“PERFECT MODEL” EXPERIMENTAL DESIGN: Compute transform functions in training step (1 example below)Produce downscaled output for historical and future time periods Can directly evaluate skill both for the historical period and the future using the Hi Res GCM output as “truth” – Test Stationarity

Options for extending this ‘perfect model-based’ exploration of statistical downscaling stationarity More SD Methods More Climate Variables & Indices Different HiRes Climate Models Different Emissions Scenarios & Times Use of Synthetic Time Series Alter GCM-based data in known ways to ‘Raise the Bar’ Mix & Match GCMs’ Target & Predictors ?? More Ideas ??

This project does not produce any data files that one would use in real world applications. The aim at GFDL is to gain knowledge about aspects of commonly used SD methods (& GCMS), and to communicate that knowledge so that better informed decisions can be made. Odds-n-ends from day 1 * Not all SD codes run on a cell phone * Not All GCMs have 200km atmos grid (50km) * GCMs are developed primarily as research tools – not prediction tools

Goals of this presentation 1. Define the ‘stationarity assumption’ inherent to statistical downscaling of future climate projections. 2. Present our ‘perfect model’ (aka ‘big brother’) approach to quantitatively assess the extent to which the stationarity assumption holds. 3. Illustrate with a few examples the kind of results one can generate using this evaluation framework (Anne Stoner will show more detail next…) 4. Introduce options to extend and supplement the method (setting the hurdle at different heights). 5. Invite statistical downscalers to consider testing their methods within the perfect model framework. 6. Garner feedback from workshop participants.