Designing Ensembles for Climate Prediction

Slides:



Advertisements
Similar presentations
Uncertainties in Predictions of Arctic Climate Peter Challenor, Bablu Sinha (NOC) Myles Allen (Oxford), Robin Tokmakian (NPS)
Advertisements

Pattern Recognition and Machine Learning
Rachel T. Johnson Douglas C. Montgomery Bradley Jones
1 -Classification: Internal Uncertainty in petroleum reservoirs.
On Sequential Experimental Design for Empirical Model-Building under Interval Error Sergei Zhilin, Altai State University, Barnaul, Russia.
Insert Date HereSlide 1 Using Derivative and Integral Information in the Statistical Analysis of Computer Models Gemma Stephenson March 2007.
Getting started with GEM-SA Marc Kennedy. This talk  Starting GEM-SA program  Creating input and output files  Explanation of the menus, toolbars,
Introduction to Design of Experiments by Dr Brad Morantz
Xuhua Xia Multiple regression Xuhua Xia
Multiple Comparisons in Factorial Experiments
some observations on the
Split Questionnaire Designs for Consumer Expenditure Survey Trivellore Raghunathan (Raghu) University of Michigan BLS Workshop December 8-9, 2010.
Basic Experimental Design
Use of Kalman filters in time and frequency analysis John Davis 1st May 2011.
Experimental Design, Response Surface Analysis, and Optimization
Neuroinformatics 1: review of statistics Kenneth D. Harris UCL, 28/1/15.
Validating uncertain predictions Tony O’Hagan, Leo Bastos, Jeremy Oakley, University of Sheffield.
Factor Analysis Purpose of Factor Analysis
Discrete Event Simulation How to generate RV according to a specified distribution? geometric Poisson etc. Example of a DEVS: repair problem.
CRASH UQ Program: Overview & Results James Paul Holloway CRASH Annual Review Fall 2010.
An Optimal Learning Approach to Finding an Outbreak of a Disease Warren Scott Warren Powell
A Two Level Monte Carlo Approach To Calculating
Active Appearance Models Suppose we have a statistical appearance model –Trained from sets of examples How do we use it to interpret new images? Use an.
Lecture 10 Comparison and Evaluation of Alternative System Designs.
Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from.
Basics of regression analysis
Introduction to the design (and analysis) of experiments James M. Curran Department of Statistics, University of Auckland
Determining the Significance of Item Order In Randomized Problem Sets Zachary A. Pardos, Neil T. Heffernan Worcester Polytechnic Institute Department of.
Statistical Methods For Engineers ChE 477 (UO Lab) Larry Baxter & Stan Harding Brigham Young University.
Random Number Generators CISC/QCSE 810. What is random? Flip 10 coins: how many do you expect will be heads? Measure 100 people: how are their heights.
Gaussian process modelling
Experimental Design Tutorial Presented By Michael W. Totaro Wireless Research Group Center for Advanced Computer Studies University of Louisiana at Lafayette.
CHAPTER 17 O PTIMAL D ESIGN FOR E XPERIMENTAL I NPUTS Organization of chapter in ISSO* –Background Motivation Finite sample and asymptotic (continuous)
Applications of Bayesian sensitivity and uncertainty analysis to the statistical analysis of computer simulators for carbon dynamics Marc Kennedy Clive.
MML Inference of RBFs Enes Makalic Lloyd Allison Andrew Paplinski.
1 Advances in the Construction of Efficient Stated Choice Experimental Designs John Rose 1 Michiel Bliemer 1,2 1 The University of Sydney, Australia 2.
Chapter 11Design & Analysis of Experiments 8E 2012 Montgomery 1.
Statistics and the Verification Validation & Testing of Adaptive Systems Roman D. Fresnedo M&CT, Phantom Works The Boeing Company.
Getting started with GEM-SA. GEM-SA course - session 32 This talk Starting GEM-SA program Creating input and output files Explanation of the menus, toolbars,
17 May 2007RSS Kent Local Group1 Quantifying uncertainty in the UK carbon flux Tony O’Hagan CTCD, Sheffield.
Center for Radiative Shock Hydrodynamics Fall 2011 Review Assessment of predictive capability Derek Bingham 1.
PROCESS MODELLING AND MODEL ANALYSIS © CAPE Centre, The University of Queensland Hungarian Academy of Sciences Statistical Model Calibration and Validation.
July 11, 2006Bayesian Inference and Maximum Entropy Probing the covariance matrix Kenneth M. Hanson T-16, Nuclear Physics; Theoretical Division Los.
CHAPTER 17 O PTIMAL D ESIGN FOR E XPERIMENTAL I NPUTS Organization of chapter in ISSO –Background Motivation Finite sample and asymptotic (continuous)
Emulation and Sensitivity Analysis of the CMAQ Model During a UK Ozone Pollution Episode Andrew Beddows Environmental Research Group King’s College London.
- 1 - Overall procedure of validation Calibration Validation Figure 12.4 Validation, calibration, and prediction (Oberkampf and Barone, 2004 ). Model accuracy.
Efficient Gaussian Process Regression for large Data Sets ANJISHNU BANERJEE, DAVID DUNSON, SURYA TOKDAR Biometrika, 2008.
How Good is a Model? How much information does AIC give us? –Model 1: 3124 –Model 2: 2932 –Model 3: 2968 –Model 4: 3204 –Model 5: 5436.
Designing Factorial Experiments with Binary Response Tel-Aviv University Faculty of Exact Sciences Department of Statistics and Operations Research Hovav.
Options and generalisations. Outline Dimensionality Many inputs and/or many outputs GP structure Mean and variance functions Prior information Multi-output,
INTRO TO OPTIMIZATION MATH-415 Numerical Analysis 1.
University of Colorado Boulder ASEN 5070 Statistical Orbit determination I Fall 2012 Professor George H. Born Professor Jeffrey S. Parker Lecture 9: Least.
Colorado Center for Astrodynamics Research The University of Colorado 1 STATISTICAL ORBIT DETERMINATION Kalman Filter with Process Noise Gauss- Markov.
1 Design of experiment for computer simulations Let X = (X 1,…,X p )  R p denote the vector of input values chosen for the computer program Each X j is.
Computacion Inteligente Least-Square Methods for System Identification.
Introduction to emulators Tony O’Hagan University of Sheffield.
A Kriging or Gaussian Process emulator has: an unadjusted mean (frequently a least squares fit: ), a correction / adjustment to the mean based on data,
8 Sept 2006, DEMA2006Slide 1 An Introduction to Computer Experiments and their Design Problems Tony O’Hagan University of Sheffield.
Kriging - Introduction Method invented in the 1950s by South African geologist Daniel Krige (1919-) for predicting distribution of minerals. Became very.
A Primer on Running Deterministic Experiments
PSY 626: Bayesian Statistics for Psychological Science
How Good is a Model? How much information does AIC give us?
Rutgers Intelligent Transportation Systems (RITS) Laboratory
Robust Multipath Channel Identification
PSY 626: Bayesian Statistics for Psychological Science
Discrete Event Simulation - 4
Network Inference Chris Holmes Oxford Centre for Gene Function, &,
Lecture 4 - Monte Carlo improvements via variance reduction techniques: antithetic sampling Antithetic variates: for any one path obtained by a gaussian.
Experimental Design All experiments consist of two basic structures:
Sampling Plans.
Presentation transcript:

Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre

Why Ensembles for Climate Prediction? Not just a point estimate Uncertainty estimates as well Calibration of models against data Sensitivity analysis

Overview What is experimental design? Why should we be interested? Perturbed physics ensembles Space filling designs Some recent results Multimodel ensembles Conclusions

What is experimental design? Developed from agricultural experiments in the 1920’s How should you apply treatments to experimental plots in a field?

R.A. Fisher Greatest Statistician of the 20th Century Randomisation Block designs Latin squares Split plots …

Clinical Trials Randomisation Blind and Double Blind Trials Sequential Designs

Why should we worry about designing our experiments? Would you take medication that hadn’t been through a properly designed clinical trial? Would you set climate policy without a properly designed climate model experiment?

Computer Experiments (Climate model ensembles) Computer experiments are very different from either clinical trials or field experiments. In general we are using them to explore the properties of some computer simulator (model). This is usually the numerical solution of a system of PDE’s or ODE’s

Computer Experiments Mathematically we can write our computer simulator as where y is the output, x is the input and η(.) is the unknown mathematical function represented by the simulator y and x are often very high dimension y=\eta(x)

Computer Experiments Normally the purpose of our computer experiment is to make some inference about the model Estimate what the model does at inputs we haven’t run it at Optimise the model parameters w.r.to some data Make predictions

Types of ensemble Perturbed physics ensembles Change inputs (parameters, initial conditions, …) to a single model Multimodel ensembles Look at multiple models

What is a good experimental design? Make our inferences to the highest accuracy with the minimum cost

Optimal Design Fisher Information ≃ inverse of variance Maximise the information = minimising the variance D-optimal designs Minimise the determinant of the variance matrix A-optimal designs Minimise the trace of the variance matrix There are others (I, V, G, E optimality)

Bayesian Design Set up a D-optimal design by maximising the utility U(S)=\int_\theta log (det[\mathcal{I}(\theta|S)])p(\theta)d\theta \mathcal{I}(\theta|S) is the variance matrix of θ in design S

Fisher Information Matrix The Fisher Information matrix is an approximation to the inverse of the variance matrix. \mathcal{F}_{i,j}=\frac{\partial^2 L}{\partial x_i \partial x_j}

An Possible Design ‘Star’ design

No aphorism is more frequently repeated in connection with field trials experiments, than that we must ask Nature few questions or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken. Nature, he suggests, will best respond to a logical and carefully thought out questionnaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed. R.A. Fisher, 1926

What we need from the design of a climate model ensemble We want to Span the whole input space Observe interactions Minimise the number of simulator evaluations

Space Filling Designs Factorial Designs Latin Hypercubes Pseudo random sequences Sobol sequence

The Full Factorial We set each input (factor) at a set number of levels All combinations are included in the design n levels of m factors needs nm points This gets large quickly

An Example 52 factorial

Fractional Factorial Full factorials are expensive For large number of factors only 2 or 3 levels Can use fractional factorials (2 levels)

The Latin Hypercube Decide how many simulator runs you can afford Divide each input range into that number of intervals Allocate a point to each interval Randomly permute across each input

The Latin Hypercube

The Latin Hypercube We don’t have an algorithm for the optimal Latin hypercube What is a good Latin hypercube? Maximin Orthogonal designs Pragmatic designs

A Latin Hypercube

A maximin LHC

Are Factorials better than Latin Hypercubes

Low Discrepancy Sequences Alternative to Latin hypercubes Designed for multi-dimensional integrals Examples include Halton sequences, Niederreiter nets and Sobol sequences \delta^*(D)=\sup_B \left | \frac{I_B(D)}{N}-|B| \right | B= \left \{ \prod_{i=1}^{d} [0,u_i) : u_i \in (0,1] \right \}

Warning: Hard Maths

Discrepancy Discrepancy is defined as where is the number of points in B and \delta^*(D)=\sup_B \left | \frac{I_B(D)}{N}-|B| \right | B= \left \{ \prod_{i=1}^{d} [0,u_i) : u_i \in (0,1] \right \} I_B(D) |B| = \int_B dx

Low Discrepancy Sequences It is believed, but not proved, that minimum discrepancy sequences have the property Look for low values of Examples include Halton sequences, Niederreiter nets and Sobol sequences \delta^*(D) \leqslant C_d(\log n)^d + O(\log n)^{d-1} C_d

End of Hard Maths

Sobol Sequences A low discrepancy sequence A 2n-1 Sobol sequence is a Latin hypercube Some projections of multi-dimensional Sobol sequences are not ‘good’

Sobol Sequences

Sobol Sequences

Sequential Designs So far our designs have been one off We make a design and that dictates how we run the simulator We do not learn from the early runs An idea from clinical trials is to learn as we carry out the experiment

Sequential Design for Computer Experiments Perform an initial experiment (usually space filling) Add additional points to satisfy some criteria We might add additional points where our predictions of simulator output are most uncertain We might add additional points for optimisation

A D-optimal design for smoothness I’m fitting an emulator to a computer experiment Can we design an experiment to estimate the ‘smoothness’ parameters of the emulator optimally?

Emulators δ is a zero mean Gaussian process \eta(\theta) = \mu(\theta) + \delta(\theta) \mu(\theta) = A(\theta) \beta’ δ is a zero mean Gaussian process This is defined in terms of a variance (σ2 and a correlation function (C(x1,x2))

An Example of an Emulator

Zhu and Stein (2004) In the geostatistical context Zhu and Stein show that the Fisher information is approximately given by U(S)=\int_B log (det[\mathcal{I}(B|S)])p(B)dB \mathcal{I}_{i,j}(B|S) = \frac{1}{2} tr\left( \Sigma^{-1} \Sigma_i \Sigma^{-1} \Sigma_j\right) \Sigma_k(i,j) = \frac{\partial \Sigma(B)}{\partial B_k}-(x_{i,k} - x_{j,k})^2 \Sigma(x_i,x_j)

Bayesian Design Approximate the inverse of the covariance matrix by the Fisher information matrix Set up a D-optimal design by maximising the utility

5-point Sobol

10-point Sobol

Sobol 10 +5 One at time (5) Five at a time

Designing for Multiple Climate Models So far we have considered designs for single simulators How might we design for multiple models? The IPCC problem ‘Ensemble of opportunity’?

So What’s the Problem Common outputs between simulators Not common inputs An important area for research

Conclusions Designing model ensembles can make them more efficient make the experimenter think about the problem There are a variety of designs around Consult a statistician before you design the experiment Design of computer experiments is an active area of research (not only in climate/environmental sciences)