Model dependence and an idea for post- processing multi-model ensembles Craig H. Bishop Naval Research Laboratory, Monterey, CA, USA Gab Abramowitz Climate.

Slides:



Advertisements
Similar presentations
Multi-model ensemble post-processing and the replicate Earth paradigm (Manuscript available on-line in Climate Dynamics) Craig H. Bishop Naval Research.
Advertisements

Regression and correlation methods
FTP Biostatistics II Model parameter estimations: Confronting models with measurements.
Estimation  Samples are collected to estimate characteristics of the population of particular interest. Parameter – numerical characteristic of the population.
3.2 OLS Fitted Values and Residuals -after obtaining OLS estimates, we can then obtain fitted or predicted values for y: -given our actual and predicted.
Regression, Correlation. Research Theoretical empirical Usually combination of the two.
Prediction, Correlation, and Lack of Fit in Regression (§11. 4, 11
Errors in Error Variance Prediction and Ensemble Post-Processing Elizabeth Satterfield 1, Craig Bishop 2 1 National Research Council, Monterey, CA, USA;
1 Introduction to Inference Confidence Intervals William P. Wattles, Ph.D. Psychology 302.
Initialization Issues of Coupled Ocean-atmosphere Prediction System Climate and Environment System Research Center Seoul National University, Korea In-Sik.
Objectives (BPS chapter 24)
Introduction to Probability and Probabilistic Forecasting L i n k i n g S c i e n c e t o S o c i e t y Simon Mason International Research Institute for.
Basic geostatistics Austin Troy.
The Simple Linear Regression Model: Specification and Estimation
Correlation and Autocorrelation
Resampling techniques Why resampling? Jacknife Cross-validation Bootstrap Examples of application of bootstrap.
Outline Further Reading: Detailed Notes Posted on Class Web Sites Natural Environments: The Atmosphere GE 101 – Spring 2007 Boston University Myneni L29:
Regression Analysis. Unscheduled Maintenance Issue: l 36 flight squadrons l Each experiences unscheduled maintenance actions (UMAs) l UMAs costs $1000.
Statistics II: An Overview of Statistics. Outline for Statistics II Lecture: SPSS Syntax – Some examples. Normal Distribution Curve. Sampling Distribution.
Statistics for Business and Economics
SYSTEMS Identification
EG1204: Earth Systems: an introduction Meteorology and Climate Lecture 7 Climate: prediction & change.
1 Simple Linear Regression Chapter Introduction In this chapter we examine the relationship among interval variables via a mathematical equation.
Ensemble Post-Processing and it’s Potential Benefits for the Operational Forecaster Michael Erickson and Brian A. Colle School of Marine and Atmospheric.
Statistical Methods for long-range forecast By Syunji Takahashi Climate Prediction Division JMA.
Introduction to Regression Analysis, Chapter 13,
Relationships Among Variables
Chapter 8: Bivariate Regression and Correlation
Lecture 16 Correlation and Coefficient of Correlation
Regression and Correlation Methods Judy Zhong Ph.D.
1 FORECASTING Regression Analysis Aslı Sencer Graduate Program in Business Information Systems.
Lecture 12 Statistical Inference (Estimation) Point and Interval estimation By Aziza Munir.
Montecarlo Simulation LAB NOV ECON Montecarlo Simulations Monte Carlo simulation is a method of analysis based on artificially recreating.
Geo479/579: Geostatistics Ch12. Ordinary Kriging (1)
Portfolio Theory Chapter 7
© 2001 Prentice-Hall, Inc. Statistics for Business and Economics Simple Linear Regression Chapter 10.
VI. Evaluate Model Fit Basic questions that modelers must address are: How well does the model fit the data? Do changes to a model, such as reparameterization,
Tests and Measurements Intersession 2006.
Model validation Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015.
Properties of OLS How Reliable is OLS?. Learning Objectives 1.Review of the idea that the OLS estimator is a random variable 2.How do we judge the quality.
Maximum Likelihood Estimation Methods of Economic Investigation Lecture 17.
Craig H. Bishop Elizabeth A Satterfield Kevin T. Shanley, David Kuhl, Tom Rosmond, Justin McLay and Nancy Baker Naval Research Laboratory Monterey CA November.
Chapter 7 Point Estimation of Parameters. Learning Objectives Explain the general concepts of estimating Explain important properties of point estimators.
ECE-7000: Nonlinear Dynamical Systems Overfitting and model costs Overfitting  The more free parameters a model has, the better it can be adapted.
Chapter 20 Classification and Estimation Classification – Feature selection Good feature have four characteristics: –Discrimination. Features.
Marshall University School of Medicine Department of Biochemistry and Microbiology BMS 617 Lecture 11: Models Marshall University Genomics Core Facility.
BASIC STATISTICAL CONCEPTS Statistical Moments & Probability Density Functions Ocean is not “stationary” “Stationary” - statistical properties remain constant.
Linear Regression Linear Regression. Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc. Purpose Understand Linear Regression. Use R functions.
Of what use is a statistician in climate modeling? Peter Guttorp University of Washington Norwegian Computing Center
26134 Business Statistics Week 4 Tutorial Simple Linear Regression Key concepts in this tutorial are listed below 1. Detecting.
A Random Subgrouping Scheme for Ensemble Kalman Filters Yun Liu Dept. of Atmospheric and Oceanic Science, University of Maryland Atmospheric and oceanic.
Chapter 13 Understanding research results: statistical inference.
Forecast 2 Linear trend Forecast error Seasonal demand.
Expected Return and Risk. Explain how expected return and risk for securities are determined. Explain how expected return and risk for portfolios are.
Shortwave and longwave contributions to global warming under increased CO 2 Aaron Donohoe, University of Washington CLIVAR CONCEPT HEAT Meeting Exeter,
1/39 Seasonal Prediction of Asian Monsoon: Predictability Issues and Limitations Arun Kumar Climate Prediction Center
Central Bank of Egypt Basic statistics. Central Bank of Egypt 2 Index I.Measures of Central Tendency II.Measures of variability of distribution III.Covariance.
26134 Business Statistics Week 4 Tutorial Simple Linear Regression Key concepts in this tutorial are listed below 1. Detecting.
Why Model? Make predictions or forecasts where we don’t have data.
Probability and the Normal Curve
Ch. 2: The Simple Regression Model
Probability Theory and Parameter Estimation I
Verifying and interpreting ensemble products
Nathalie Voisin, Andy W. Wood and Dennis P. Lettenmaier
Statistical Methods For Engineers
Statistics II: An Overview of Statistics
The Simple Regression Model
Parametric Methods Berlin Chen, 2005 References:
Measuring the performance of climate predictions
ARCCSS & Climate Change Research Centre, UNSW, Sydney
Presentation transcript:

Model dependence and an idea for post- processing multi-model ensembles Craig H. Bishop Naval Research Laboratory, Monterey, CA, USA Gab Abramowitz Climate Change Research Centre, UNSW, Australia

Outline What model independence is and why it matters Error covariances and model dependence The replicate Earth conceptual framework for allowing chaotic climate behaviour in model evaluation A replicate Earth ensemble transformation that improves the relationship between the frequency of modelled events and the likelihood of their actual occurrence

Outline What model independence is and why it matters Error covariances and model dependence The replicate Earth conceptual framework for allowing chaotic climate behaviour in model evaluation A replicate Earth ensemble transformation that improves the relationship between the frequency of modelled events and the likelihood of their actual occurrence

Outline What model independence is and why it matters Error covariances and model dependence The replicate Earth conceptual framework for allowing chaotic climate behaviour in model evaluation A replicate Earth ensemble transformation that improves the relationship between the frequency of modelled events and the likelihood of their actual occurrence

Error covariances and model dependence At any location and time step, let y be the observed value and x k be the (bias corrected) modelled value from the k th model. Now find the minimum error variance linear combination of models:

Error covariances and model dependence

How do error correlations affect multi-model mean?

How does error correlation affect multi-model mean? Mean square error of the optimal linear combination of ensemble members as a function of the error correlation parameter  for an idealized 5 member ensemble having an error covariance matrix given by Equation (8). Error of multi-model mean decreases rapidly with decreasing error correlation  When inter-model error correlation is zero, error variance of multi- model mean is 1/K of the error variance of a single model. When errors are perfectly anti- correlated (  = -1), error variance of multi-model mean is zero.

Weighting for model dependence - results HadCRUT3 5°×5°observed monthly surface temperature CMIP3 models interpolated to 5°×5° White grid cells => >20% missing obs data

Weighting for model dependence - results Apply weights at each grid cell Apply weights globally Multi- model mean 30 out-of-sample tests: 29 years to define weights, 1 to test – repeated for all 30 possible testing years 24 CMIP3 models HadCRUT3 5°×5°observed monthly surface temperature Performance weights obtained by assuming that

Outline What model independence is and why it matters Error covariances and model dependence The replicate Earth conceptual framework for allowing chaotic climate behaviour in model evaluation A replicate Earth ensemble transformation that improves the relationship between the frequency of modelled events and the likelihood of their actual occurrence

If zero correlation we have performance weighting: We’d expect zero correlation for independent random variables, but do we really want this for independent climate models? This would suggest a perfect, independent model would reproduce the observed data + noise: – Observed data would always be at the centre of the distribution of an ensemble of perfect models – “truth+error” paradigm => completely deterministic, predictable climate Can we be sure that climate is predictable? Do we need to be sure that it is? Error correlation, independence and determinism

CPDFs give probability of observing ranges of values of temperature, wind, rain or even particular environmental phenomena – such as tropical cyclones or floods. – The instantaneous CPDF gives the probability of outcomes of system at a point in time – Any observation of our Earth is a single random draw from such a CPDF If the climate system were static, the CPDF would be well approximated by historical data Climate forcing green-house gases rapidly increasing It is impossible to empirically determine the CPDF during climate change Climate change impact assessment requires the CPDF, not just the mean Climatic Probability Density Function (CPDF)

The Replicate Earth Paradigm Imagine a very large number of Earth replicates that experience immeasurably similar orbital / solar / GHG forcing Each Earth has a different atmosphere / ocean state as a result of chaotic processes Behaviour across replicate Earths defines the CPDF in presence of climate change; e.g. frequency of weather categories Climate models can be viewed as attempts to create replicate Earths conditioned on the observations used for model development and initialization A perfect model’s prediction would be a random draw from the CPDF of replicate Earths

The Replicate Earth paradigm Global mean surface temperature over the last century, expressed as an anomaly. An ensemble of climate models is represented by the yellow lines, their multi-model mean in red, and the observational record in black (originally Figure TS.23.a in IPCC AR4 Working Group 1 Technical Summary).

“Truth+error” vs “Replicate Earth” paradigms “Truth+error” => perfect model is obs + noise; multi-model mean of an ensemble of perfect models should converge to zero error as number of models increases – Assumption that all processes are predictable “Replicate Earth” => perfect model is drawn from the same distribution as observations; multi-model mean of ensemble of perfect models will converge to mean of CPDF To what extent do climate models approximate replicate Earths?

Outline What model independence is and why it matters Error covariances and model dependence The replicate Earth conceptual framework for allowing chaotic climate behaviour in model evaluation A replicate Earth ensemble transformation that improves the relationship between the frequency of modelled events and the likelihood of their actual occurrence

Two key properties of replicate Earths 1.Mean of the distribution of replicate Earths (blue line) is the linear combination of replicate Earths that minimises distance from our Earth’s observations. 2.Time average of the variance of replicate Earths is approximately equal to the mean square error of the observations about the CPDF mean (i.e. MSE of replicate Earth mean w.r.t observations)

CMIP3 climate models do not look like replicate Earths Biases and shared differences in response to, e.g., greenhouse gases, mean models are unlike replicate Earths. We can show: 1.The mean of the AR4 ensemble is not the minimum error variance estimate

CMIP3 climate models do not look like replicate Earths Biases and shared differences in response to, e.g., greenhouse gases, mean models are unlike replicate Earths. We can show: 1.The mean of the AR4 ensemble, is not the minimum error variance estimate, and 2.The time average of the variance of the AR4 ensemble is not equal to the mean square error of the mean of the AR4 ensemble [nor is it equal to the error variance of the true minimum error variance estimate].

Ensemble Transformation 1.Find linear combination of models that minimizes error variance – Coefficients not necessarily positive 2.Rescale weights and models’ time series so that: a)Weights are positive (interpret a weight as probability model is an Earth replicate) b)Weighted mean gives the minimum error variance estimate in 1. c)The time average of the CPDF variance estimate equals the MSE of the weighted mean Transformations require solving for parameters α and β: Ensemble created by sampling with frequency is like a replicate Earth ensemble in that (a) its sample mean is the minimum error variance estimate, and (b) its variance equals the error variance of the sample mean.

Rank Frequency Histograms for M=6 Raw and bias corrected ensembles were found not to give reliable probabilistic forecasts for any ensemble size The most accurate forecast (orange line) – which is based on differing weights for each grid cell – was found to give approximately reliable probabilistic forecasts for M=6 and M=5. For smaller values of M, the extreme ranks were under-populated by the verifying observation. For larger values of M, the extreme ranks were under-populated. Does this mean the effective local ensemble size is about 5.5? The ensemble forecast based on one set of global weights gave its flattest RFH for M=9. Flat line (zero slope) indicates that ensemble frequencies give reliable probabilistic forecasts 01 At each grid cell and time, take n samples from a uniform distribution [0,1] to select an n- model ensemble Vary size of ensemble to achieve flat histograms

Rank Frequency Histograms

Conclusions Model dependence crucial to climate prediction accuracy – Dependence of zero error correlation implies climate is entirely predictable Replicate Earth paradigm: – Observations not at the centre of the distribution – Perfect, independent model drawn from the same distribution as observations – CPDF is formally unobservable in the presence of a changing climate – models provide the only estimate of the replicate Earth ensemble – Provides a framework for understanding role of chaos in climate prediction Ensemble post-processing to give replicate-Earth-like models: – Marked reduction in RMSE of prediction – Flatter rank frequency histograms – needed to climate change impacts assessments Future work: Use framework to make climate predictions: – Following suggestion from Kevin Bowman (JPL) perhaps focus on error measure more directly related to climate sensitivity: OLR, difference between Spring and Autumn snow cover, any suggestions?

The effective number of climate models Case 1: truth + error paradigm: Consider 24 independent random time series with mean = 0 and variance = 1 : – Mean of time series will have variance 1/24 Now consider 24 time series of model error – Zero correlation (‘independent’) => multi-model mean error variance should drop to ~ 1/24 of average ensemble member error variance – Ratio of actual error variance to this expected value gives effective number of models

The effective number of climate models Case 2: replicate Earth paradigm, based on perturbed weights: If models were replicate Earths, minimum error variance linear combination would be obtained using weights = 1/24 Model dependence => significant error correlations, this is evident in weights: – Some weights nearly zero – Some weights > 1/24

Rank frequency histogram (out of sample) How can we tell if the transformation works? For a single grid cell, at a single time step, what is the rank of the observed value in the observed + model set? Perturbed models behave more like replicate Earths than raw or bias corrected models (flatter histogram)