Furthermore… References Katz, R.W. and A.H. Murphy (eds), 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, Cambridge.

Slides:



Advertisements
Similar presentations
Measuring the performance of climate predictions Chris Ferro, Tom Fricker, David Stephenson Mathematics Research Institute University of Exeter, UK IMA.
Advertisements

What is a good ensemble forecast? Chris Ferro University of Exeter, UK With thanks to Tom Fricker, Keith Mitchell, Stefan Siegert, David Stephenson, Robin.
What is a good ensemble forecast? Chris Ferro University of Exeter, UK With thanks to Tom Fricker, Keith Mitchell, Stefan Siegert, David Stephenson, Robin.
Hypothesis testing and confidence intervals by resampling by J. Kárász.
14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic May 2001.
Introduction to Probability and Probabilistic Forecasting L i n k i n g S c i e n c e t o S o c i e t y Simon Mason International Research Institute for.
Barbara Casati June 2009 FMI Verification of continuous predictands
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. Chapter 12 Analyzing the Association Between Quantitative Variables: Regression Analysis Section.
Verification of extreme events Barbara Casati (Environment Canada) D.B. Stephenson (University of Reading) ENVIRONMENT CANADA ENVIRONNEMENT CANADA.
1 Design of Engineering Experiments Part 2 – Basic Statistical Concepts Simple comparative experiments –The hypothesis testing framework –The two-sample.
IRDM WS Source: Arnold O. Allen, Probability, Statistics, and Queueing Theory with Computer Science Applications, Academic Press, 1990 Reference.
Verification of ensembles Courtesy of Barbara Brown Acknowledgments: Tom Hamill, Laurence Wilson, Tressa Fowler Copyright UCAR 2012, all rights reserved.
4IWVM - Tutorial Session - June 2009 Verification of categorical predictands Anna Ghelli ECMWF.
Hypothesis Testing CSCE 587.
Statistics as a means to construct knowledge in climate and related sciences -- a discourse -- Hans von Storch Institute for Coastal Research GKSS, Germany.
Permutation Procedures, Bootstrap Methods and the Jackknife Bob Livezey Climate Services Division/OCWWS/NWS AMS Short Course on Significance Testing, Model.
Sampling Uncertainty in Verification Measures for Binary Deterministic Forecasts Ian Jolliffe and David Stephenson 1EMS September Sampling uncertainty.
Advanced Higher Statistics Data Analysis and Modelling Hypothesis Testing Statistical Inference AH.
Estimating Incremental Cost- Effectiveness Ratios from Cluster Randomized Intervention Trials M. Ashraf Chaudhary & M. Shoukri.
AP STATISTICS LESSON COMPARING TWO PROPORTIONS.
CHAPTER 17: Tests of Significance: The Basics
7.4 – Sampling Distribution Statistic: a numerical descriptive measure of a sample Parameter: a numerical descriptive measure of a population.
© Cambridge University Press 2013 Thomson_alphaem.
© Cambridge University Press 2013 Thomson_Fig
2nd SRNWP Workshop on “Short-range ensembles” – Bologna, 7-8 April Verification of ensemble systems Chiara Marsigli ARPA-SIM.
Normal Distribution.
Statistics 101 Chapter 10 Section 2. How to run a significance test Step 1: Identify the population of interest and the parameter you want to draw conclusions.
World Meteorological Organization Working together in weather, climate and water Enhanced User and Forecaster Oriented TAF Quality Assessment CAeM-XIV.
Deutscher Wetterdienst Bootstrapping – using different methods to estimate statistical differences between model errors Ulrich Damrath COSMO GM Rome 2011.
Name Buena Regional Middle School INCLUDE Math Class Year My Electronic Portfolio.
ECMWF Training Course Reading, 25 April 2006 EPS Diagnostic Tools Renate Hagedorn European Centre for Medium-Range Weather Forecasts.
: An alternative representation of level of significance. - normal distribution applies. - α level of significance (e.g. 5% in two tails) determines the.
A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC Outlooks Bob Livezey Climate Services Division/OCWWS/NWS.
Statistics as a means to construct knowledge in climate and related sciences -- a discourse -- Hans von Storch Institute for Coastal Research GKSS, Germany.
Reasoning in Psychology Using Statistics Psychology
ON NONPARAMETRIC INTERVAL ESTIMATION OF A REGRESSION FUNCTION BASED ON THE RESAMPLING ALEXANDER ANDRONOV Riga Technical University Riga, Latvia.
Verification of ensemble systems Chiara Marsigli ARPA-SIMC.
Common verification methods for ensemble forecasts
Diagnostic verification and extremes: 1 st Breakout Discussed the need for toolkit to build beyond current capabilities (e.g., NCEP) Identified (and began.
NCAR, 15 April Fuzzy verification of fake cases Beth Ebert Center for Australian Weather and Climate Research Bureau of Meteorology.
© Cambridge University Press 2013 Thomson_Fig
Project Plan Task 8 and VERSUS2 Installation problems Anatoly Myravyev and Anastasia Bundel, Hydrometcenter of Russia March 2010.
Verification methods - towards a user oriented verification The verification group.
Quantifying Uncertainty
ENVIRONMENTAL SCIENCE TEACHERS’ CONFERENCE ENVIRONMENTAL SCIENCE TEACHERS’ CONFERENCE, Borki Molo, Poland, 7-10 February 2007 Extreme Climatic and atmospheric.
User-Focused Verification Barbara Brown* NCAR July 2006
Advanced Higher Statistics
© Cambridge University Press 2011
Verifying and interpreting ensemble products
Elementary Statistics
Binary Forecasts and Observations
When we free ourselves of desire,
Quantifying uncertainty using the bootstrap
Introduction to Inference
forecasts of rare events
BOOTSTRAPPING: LEARNING FROM THE SAMPLE
Thomson_eeWWtgc © Cambridge University Press 2013.
Validation-Based Decision Making
Thomson_atlascmsEventsAlt
Comparing two Rates Farrokh Alemi Ph.D.
Differences between social & natural sciences
Evaluation challenges - Reporting
Introduction: Statistics meets corpus linguistics
CHAPTER 12 Inference for Proportions
CHAPTER 12 Inference for Proportions
Thomson_CandP © Cambridge University Press 2013.
Measuring the performance of climate predictions
the performance of weather forecasts
Thomson_AFBCartoon © Cambridge University Press 2013.
What is a good ensemble forecast?
Presentation transcript:

Furthermore… References Katz, R.W. and A.H. Murphy (eds), 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, Cambridge. Jolliffe, I.T., and D.B. Stephenson, 2003: Forecast Verification. A Practitioner's Guide in Atmospheric Science. Wiley and Sons Ltd, 240 pp. Click here to see the Table of Contents. von Storch, H. and F.W. Zwiers, 1999: Statistical Analysis in Climate Research. Cambridge University Press, Cambridge. Wilks, D.S., 2006: Statistical Methods in the Atmospheric Sciences. An Introduction. Academic Press, San Diego, 467 pp.

Challenges in Spatial Forecasts Forecasts 1-4 have POD=0; FAR=1; CSI=0 Fifth forecast has POD>0, FAR 1 OF OF OFO F F O

Verifying Spatial Forecasts

Verifying Extreme Forecasts What are extremes? –Events large in magnitude –Events rare in occurrence Traditional skill scores become unstable as the probability of the event becomes increasingly small. Extreme value statistics. Different distributions are more appropriate for use with extreme events.

Confidence Intervals Skill scores are statistics and as such it is reasonable to ask for confidence intervals. Some techniques for estimating confidence intervals –Parametric assumptions and inference –Bootstrapping (re-sampling original data) –Use of holdout data in creating model Difficulties include –Dependent observations –Huge numbers of observation – everything is significant –Small number of data