Binary Forecasts and Observations

Slides:



Advertisements
Similar presentations
ECMWF Slide 1Met Op training course – Reading, March 2004 Forecast verification: probabilistic aspects Anna Ghelli, ECMWF.
Advertisements

Robin Hogan Ewan OConnor University of Reading, UK What is the half-life of a cloud forecast?
Canadian Aviation Forecast Verification
Guidance of the WMO Commission for CIimatology on verification of operational seasonal forecasts Ernesto Rodríguez Camino AEMET (Thanks to S. Mason, C.
14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic May 2001.
1 of Introduction to Forecasts and Verification.
Validation of Satellite Precipitation Estimates for Weather and Hydrological Applications Beth Ebert BMRC, Melbourne, Australia 3 rd IPWG Workshop / 3.
NWS TAF Verification Brandi Richardson NWS Shreveport, LA.
1 Verification Introduction Holly C. Hartmann Department of Hydrology and Water Resources University of Arizona RFC Verification Workshop,
Verification of probability and ensemble forecasts
Bookmaker or Forecaster? By Philip Johnson. Jersey Meteorological Department.
Daria Kluver Independent Study From Statistical Methods in the Atmospheric Sciences By Daniel Wilks.
Introduction to Probability and Probabilistic Forecasting L i n k i n g S c i e n c e t o S o c i e t y Simon Mason International Research Institute for.
Paul Fajman NOAA/NWS/MDL September 7,  NDFD ugly string  NDFD Forecasts and encoding  Observations  Assumptions  Output, Scores and Display.
EXAMPLES Attributes of Information. Overview Information Systems help companies achieve their goals. How do they do it? 1. By processing raw data into.
True/False. False True Subject May Go Here True / False ? Type correct answer here. Type incorrect answer here.
Barbara Casati June 2009 FMI Verification of continuous predictands
Verification of extreme events Barbara Casati (Environment Canada) D.B. Stephenson (University of Reading) ENVIRONMENT CANADA ENVIRONNEMENT CANADA.
Richard (Rick)Jones Regional Training Workshop on Severe Weather Forecasting Macau, April 8 -13, 2013.
Verification of ensembles Courtesy of Barbara Brown Acknowledgments: Tom Hamill, Laurence Wilson, Tressa Fowler Copyright UCAR 2012, all rights reserved.
4IWVM - Tutorial Session - June 2009 Verification of categorical predictands Anna Ghelli ECMWF.
Verification methods - towards a user oriented verification WG5.
A Preliminary Verification of the National Hurricane Center’s Tropical Cyclone Wind Probability Forecast Product Jackie Shafer Scitor Corporation Florida.
Measuring forecast skill: is it real skill or is it the varying climatology? Tom Hamill NOAA Earth System Research Lab, Boulder, Colorado
Model validation Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015.
Latest results in verification over Poland Katarzyna Starosta, Joanna Linkowska Institute of Meteorology and Water Management, Warsaw 9th COSMO General.
Requirements from KENDA on the verification NetCDF feedback files: -produced by analysis system (LETKF) and ‘stat’ utility ((to.
World Meteorological Organization Working together in weather, climate and water Enhanced User and Forecaster Oriented TAF Quality Assessment CAeM-XIV.
Traditional Verification Scores Fake forecasts  5 geometric  7 perturbed subjective evaluation  expert scores from last year’s workshop  9 cases x.
A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC Outlooks Bob Livezey Climate Services Division/OCWWS/NWS.
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Verification and Metrics (CAWCR)
Standard Verification Strategies Proposal from NWS Verification Team NWS Verification Team Draft03/23/2009 These slides include notes, which can be expanded.
© 2009 UCAR. All rights reserved. ATEC-4DWX IPR, 21−22 April 2009 National Security Applications Program Research Applications Laboratory Ensemble-4DWX.
Probabilistic Forecasts of Extreme Precipitation Events for the U.S. Hazards Assessment Kenneth Pelman 32 nd Climate Diagnostics Workshop Tallahassee,
Two Variable Statistics Limitations of the χ 2 Test.
Furthermore… References Katz, R.W. and A.H. Murphy (eds), 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, Cambridge.
Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa Hellenic National Meteorological Service.
Common verification methods for ensemble forecasts
Operational verification system Rodica Dumitrache National Metorogical Administration ROMANIA.
Gridded warning verification Harold E. Brooks NOAA/National Severe Storms Laboratory Norman, Oklahoma
Extracting probabilistic severe weather guidance from convection-allowing model forecasts Ryan Sobash 4 December 2009 Convection/NWP Seminar Series Ryan.
Verification methods - towards a user oriented verification The verification group.
Verification of C&V Forecasts Jennifer Mahoney and Barbara Brown 19 April 2001.
Centro Nazionale di Meteorologia e Climatologia Aeronautica Common Verification Suite Zurich, Sep 2005 Alessandro GALLIANI, Patrizio EMILIANI, Adriano.
7. Performance Measurement
Tom Hopson, NCAR (among others) Satya Priya, World Bank
Uncertainty cones deduced from an ensemble prediction system
TAIWIN model verification task
Spatial Verification Intercomparison Meeting, 20 February 2007, NCAR
Addressing the environmental impact of salt use on roads
Verifying and interpreting ensemble products
Tom Hopson, Jason Knievel, Yubao Liu, Gregory Roux, Wanli Wu
General framework for features-based verification
COSMO Priority Project ”Quantitative Precipitation Forecasts”
Climate Predictability Tool (CPT)
Air Quality Forecast Verification (AFQx)
P-value Approach for Test Conclusion
Relative Operating Characteristics
Probabilistic forecasts
Validation-Based Decision Making
N. Voisin, J.C. Schaake and D.P. Lettenmaier
Quantitative verification of cloud fraction forecasts
New Developments in Aviation Forecast Guidance from the RUC
COSMO-LEPS Verification
How good (or bad) are seasonal forecasts?
Drivers Influencing Weather-related NAS Metrics
How can an improved model yield higher RMS errors?
Can we distinguish wet years from dry years?
Verification of SPE Probability Forecasts at SEPC
Short Range Ensemble Prediction System Verification over Greece
Presentation transcript:

Binary Forecasts and Observations Yes/ No; True/False; 1/0 Form the basis for many verification techniques Used in many applications of weather data Will the temperature in Phoenix exceed 90 degrees tomorrow? While quite simple, still commonly used.

2 x 2 Contingency Table Columns = obs Rows = Forecast

Finley Tornado Data (1884)

A success? Percent Correct = (28+2680)/2803 = 0.966 !!!!

Maybe not. Percent Correct = (0+2752)/2803 = 0.982 – better!! Critisism from Pierce and Gilbert. (1884) Percent Correct = (0+2752)/2803 = 0.982 – better!!

Alternative Statistics Mention – the never forecast has a POD = 0, FAR = 0, CSI = 1 Single statistics can be misleading. Threat Score = 28 / (28 + 72+ 23) = 0.228 Probability of Detection = 28 / (28 + 23) = 0.549 False Alarm Rate= 72/(28 + 72) = 0.720

Attributes of forecast quality Accuracy Bias Reliability Resolution Discrimination Sharpness

Value If Costs can be associated with false positives and false negative forecasts, one can determine if a forecast has value. The value of a forecast varies by user.

Skill Scores Single value to summarize performance. Reference forecast - best naive guess; persistence, climatology Proper skill scores – reflect forecastor true intent. A perfect forecast implies that the object can be perfectly observed

Generic Skill Score Positively oriented – Positive is good

Exercise 2 (ex2.r) Load verification library verify function frcst.type; obs.type; Documentation example()