1. Keith Dixon 1, Katharine Hayhoe 2, John Lanzante 1, Anne Stoner 2, Aparna Radhakrishnan 3, V. Balaji 4, Carlos Gaitán 5 “Perfect Model” Experiments: Testing Stationarity in Statistical Downscaling (NCPP Protocol 2) IS PAST PERFORMANCE AN INDICATION OF FUTURE RESULTS ? 1 2 3 DRC Inc. 4 Princeton Univ. 5 Univ. of Oklahoma 2013 Workshop on Quantitative Evaluation of Downscaled Data

2. Goals of this presentation 1. Define the ‘stationarity assumption’ inherent to statistical downscaling of future climate projections. 2. Present our ‘perfect model’ (aka ‘big brother’) approach to quantitatively assess the extent to which the stationarity assumption holds. 3. Illustrate with a few examples the kind of results one can generate using this evaluation framework (Anne Stoner will show more detail next…) 4. Introduce options to extend and supplement the method (setting the hurdle at different heights). 5. Invite statistical downscalers to consider testing their methods within the perfect model framework. 6. Garner feedback from workshop participants.

3. This project does not produce any data files that one would use in real world applications. The aim at GFDL is to gain knowledge about aspects of commonly used SD methods (& GCMS), and to communicate that knowledge so that better informed decisions can be made.

4. REAL WORLD APPLICATION: Start with 3 types of data sets … lacking observations of the futureCompute transform functions in training step (1 example below)Produce downscaled refinements of GCM (historical and/or future)Skill: Compare Obs to SD historical output (e.g., cross validation)Cannot evaluate future skill -- left assuming transform functions apply equally well to past & future -- “The Stationarity Assumption” TARGET PREDICTORS

5. “PERFECT MODEL” EXPERIMENTAL DESIGN: Start with 4 types of data sets – Hi-res GCM output as proxy for obs & coarsened version of Hi-res GCM output as proxy for usual GCM Proxy for observations in real-world application Proxy for GCM output in real-world application

6. Daily time resolution ~25km grid spacing 194 x 114 grid 22k pts

7. ~200km grid spacing 64:1

8. ~64 of the smaller 25km resolution grid cells fit within the coarser cell

10. Daily time resolution ~25km grid spacing 194 x 114 grid

12. Follow the same interpolation/regridding sequence to produce coarsened data sets for the future climate projections

13. “PERFECT MODEL” EXPERIMENTAL DESIGN: Compute transform functions in training step (1 example below)Produce downscaled output for historical and future time periods Can directly evaluate skill both for the historical period and the future using the Hi Res GCM output as “truth” – Test Stationarity QuantitativeTests of Stationarity: The extent to which SKILL computed for the FUTURE CLIMATE PROJECTIONS is diminished relative to the SKILL computed for the HISTORICAL PERIOD

14. GFDL-HiRAM-C360 experiments All data used in the perfect model tests were derived from GFDL-HiRAM-C360 (C360) model simulations conducted following CMIP5 high resolution time slice experiment protocols. We consider a pair of 30-year long “historical” experiments (1979-2008), labeled H2 and H2 here… For the period 2086-2095, we make use of a pair of 3 member ensembles -- a total of six 10-year long experiments, identified as the “C” and “E” ensembles to the right (C warms more than E)...

15. Same GCM used to generate 2 sets of future projections (3 members each) 1979-2008 model climatology 2086-2095 projection “C” (3x10yr) 2086-2095 projection “E” (3x10yr)

16. Q: How High a Hurdle does this Perfect Model approach present? A: Varies geographically, by variable of interest, time period of interest, etc.

17. Coarsened data is slightly cooler with Slightly smaller std dev. … not too challenging August Daily Max Temperatures for a point in Oklahoma (2.5C histogram bins)

18. approx. +7 C mean warming August Daily Max Temperatures for a point in Oklahoma (2.5C histogram bins)

19. August Daily Max Temperatures for a point just NE of San Fancisco Coarsened data is more ‘maritime’ & HiRes target is more ‘continental’. Presents more of a challenge during the historical period than the Oklahoma example. (2.5C histogram bins)

20. August Daily Max Temperatures for a point just NE of San Fancisco (2.5C histogram bins)

21. Note that in several locations there is a tendency for ARRM cross-validated downscaled output to have too low standard deviations just offshore and too high std. dev. over coastal land Map produced by NCPP Evaluation Environment ‘Machinery’

22. NEXT: A sampling of results…  Intended to be illustrative… not exhaustive, nor systematic.  Anne Stoner (TTU) will show more.

23. Area Mean Time Mean Absolute Downscaling Errors Σ | (Downscaled Estimate – HiRes GCM) | (NumDays) Start with a summary: ARRM downscaling errors are larger for daily max temp at end of 21 st C than for 1979-2008

24. Mean Absolute downscaling Error (MAE) during 1979-2008 Σ | (Downscaled Estimate – HiRes GCM) | (60 * 365) Geographic Variations: Downscaling MAE for 1979-2008

25. Mean absolute downscaling error during 2086-2095 “E” Projections (3 member ensemble) Geographic Variations: MAE pattern for “E” projections (+5,4C)

26. Mean absolute downscaling error during 2086-2095 “C” Projections (3 member ensemble) Geographic Variations: MAE pattern for “C” projections (+7,6C)

27. Mean Climate Change Signal Difference (ARRM minus Target) “C” Projection Geographic Variations: Bias pattern for “C” projections (+7,6C)

28. J F M A M J J A S O N D Red = “C” Blue=“E” J F M A M J J A S O N D Entire US48 Domain Looking at how well the stationarity assumption holds in different seasons Where and when ratio=1.0, the stationarity assumption fully holds ( i.e., no degradation in mean absolute downscaling error during 2085-2095 vs. the 1979-2008 period using in training.) 43214321 tasmax

29. …comparing near-coastal land vs. interior Looking at how well the stationarity assumption holds in different seasons

30. J F M A M J J A S O N D Blue= coastal SC-CSC Black = full US48+ domain Green = interior SC-CSC Looking at how well the stationarity assumption holds in different seasons “C” ensemble 43214321

31. tasmax A clear intra-month MAE trend in some but not all months Looking at how well the stationarity assumption holds in different seasons

32. “sawtooth” has lower values in cooler part of month, higher in warmer part of month “C” ensemble tasmax

33. Options for extending this ‘perfect model-based’ exploration of statistical downscaling stationarity More SD Methods More Climate Variables & Indices Different HiRes Climate Models Different Emissions Scenarios & Times Use of Synthetic Time Series Alter GCM-based data in known ways to ‘Raise the Bar’ Mix & Match GCMs’ Target & Predictors ?? More Ideas ??

34. GFDL-HiRAM-C360 experiments Data sets used in the perfect model tests are being made available to the community. This enables others to use our exp design to test the stationarity assumption for their own SD method. Also, we invite SD developers to consider collaborating with us, so that their techniques & perspectives may be more fully incorporated into this evolving perfect model-based research & assessment system. For more info, visit www.gfdl.noaa.gov/esd_eval & www.earthsystemcog.org/projects/gfdl-perfectmodel

35. In other words, by accessing these types of files, folks can generate their own SD results and test how well the stationarity assumption holds for their favorite SD method. from us

36. In other words, by accessing these types of files, folks can generate their own SD results and test how well the stationarity assumption holds for their favorite SD method. from us SD method of choice from us

37. Obviously, one does not need to perform tests using all 22,000+ grid points. Below we indicate the locations of a set of 16 points we’ve used for some development work. (color areas are 3x3 with grid point of interest in the center)

38.  Anne Stoner and Katharine Hayhoe with more ‘perfect model’ analyses…

39. Keith Dixon 1, Katharine Hayhoe 2, John Lanzante 1, Anne Stoner 2, Aparna Radhakrishnan 3, V. Balaji 4, Carlos Gaitán 5 “Perfect Model” Experiments: Testing Stationarity in Statistical Downscaling (NCPP Protocol 2) IS PAST PERFORMANCE AN INDICATION OF FUTURE RESULTS ? 1 2 3 DRC Inc. 4 Princeton Univ. 5 Univ. of Oklahoma 2013 Workshop on Quantitative Evaluation of Downscaled Data

40. “PERFECT MODEL” EXPERIMENTAL DESIGN: Compute transform functions in training step (1 example below)Produce downscaled output for historical and future time periods Can directly evaluate skill both for the historical period and the future using the Hi Res GCM output as “truth” – Test Stationarity

41. Options for extending this ‘perfect model-based’ exploration of statistical downscaling stationarity More SD Methods More Climate Variables & Indices Different HiRes Climate Models Different Emissions Scenarios & Times Use of Synthetic Time Series Alter GCM-based data in known ways to ‘Raise the Bar’ Mix & Match GCMs’ Target & Predictors ?? More Ideas ??

42. This project does not produce any data files that one would use in real world applications. The aim at GFDL is to gain knowledge about aspects of commonly used SD methods (& GCMS), and to communicate that knowledge so that better informed decisions can be made. Odds-n-ends from day 1 * Not all SD codes run on a cell phone * Not All GCMs have 200km atmos grid (50km) * GCMs are developed primarily as research tools – not prediction tools

43. Goals of this presentation 1. Define the ‘stationarity assumption’ inherent to statistical downscaling of future climate projections. 2. Present our ‘perfect model’ (aka ‘big brother’) approach to quantitatively assess the extent to which the stationarity assumption holds. 3. Illustrate with a few examples the kind of results one can generate using this evaluation framework (Anne Stoner will show more detail next…) 4. Introduce options to extend and supplement the method (setting the hurdle at different heights). 5. Invite statistical downscalers to consider testing their methods within the perfect model framework. 6. Garner feedback from workshop participants.