Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Ensembles & Probability Fcsts with MET Ensemble Stat Tool.

Slides:

Advertisements

Similar presentations

ECMWF Slide 1Met Op training course – Reading, March 2004 Forecast verification: probabilistic aspects Anna Ghelli, ECMWF.

Advertisements

Severe Weather Forecasts

Model Evaluation Tools MET. What is MET Model Evaluation Tools ( MET )- a powerful and highly configurable verification package developed by DTC offering:

14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic May 2001.

Climate Predictability Tool (CPT)

Verification of probability and ensemble forecasts

Gridded OCF Probabilistic Forecasting For Australia For more information please contact © Commonwealth of Australia 2011 Shaun Cooper.

Model Evaluation Tools (MET): An Overview. MET Verification Techniques Standard verification methods –gridded model data to point-based observations.

NWP Verification with Shape- matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1, Edward Tollerud 2, Tara Jensen 1,

An Investigation of Cool Season Extratropical Cyclone Forecast Errors Within Operational Models Brian A. Colle 1 and Michael Charles 1,2 1 School of Marine.

Exploring the Use of Object- Oriented Verification at the Hydrometeorological Prediction Center Faye E. Barthold 1,2, Keith F. Brill 1, and David R. Novak.

JEFS Status Report Department of Atmospheric Sciences University of Washington Cliff Mass, Jeff Baars, David Carey JEFS Workshop, August

Univ of AZ WRF Model Verification. Method NCEP Stage IV data used for precipitation verification – Stage IV is composite of rain fall observations and.

MOS Performance MOS significantly improves on the skill of model output. National Weather Service verification statistics have shown a narrowing gap between.

Juan Ruiz 1,2, Celeste Saulo 1,2, Soledad Cardazzo 1, Eugenia Kalnay 3 1 Departamento de Cs. de la Atmósfera y los Océanos (FCEyN-UBA), 2 Centro de Investigaciones.

Ensemble Post-Processing and it’s Potential Benefits for the Operational Forecaster Michael Erickson and Brian A. Colle School of Marine and Atmospheric.

Tara Jensen for DTC Staff 1, Steve Weiss 2, Jack Kain 3, Mike Coniglio 3 1 NCAR/RAL and NOAA/GSD, Boulder Colorado, USA 2 NOAA/NWS/Storm Prediction Center,

Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex Lisa Goddard and Simon Mason International Research Institute for.

Performance of the MOGREPS Regional Ensemble

Verification has been undertaken for the 3 month Summer period (30/05/12 – 06/09/12) using forecasts and observations at all 205 UK civil and defence aerodromes.

Using the Frequencies Procedure in SPSS 9.0 for Windows © by Julia Hartman © Copyright 2000, Julia Hartman.

Verification of ensembles Courtesy of Barbara Brown Acknowledgments: Tom Hamill, Laurence Wilson, Tressa Fowler Copyright UCAR 2012, all rights reserved.

Rank Histograms – measuring the reliability of an ensemble forecast You cannot verify an ensemble forecast with a single.

Ed Tollerud, Tara Jensen, Barb Brown ( also Yuejian Zhu, Zoltan Toth, Tony Eckel, Curtis Alexander, Huiling Yuan,…) Module 6 Objective: Provide a portable,

Tutorial. Other post-processing approaches … 1) Bayesian Model Averaging (BMA) – Raftery et al (1997) 2) Analogue approaches – Hopson and Webster, J.

Verification Summit AMB verification: rapid feedback to guide model development decisions Patrick Hofmann, Bill Moninger, Steve Weygandt, Curtis Alexander,

Verification in NCEP/HPC Using VSDB-fvs Keith F. Brill November 2007.

The set of files includes : Tcl source of the POLYGON program The database (file obtained initially by P.Afonine from using phenix.model_vs_data.

Requirements from KENDA on the verification NetCDF feedback files: -produced by analysis system (LETKF) and ‘stat’ utility ((to.

ESIP Federation 2004 : L.B.Pham S. Berrick, L. Pham, G. Leptoukh, Z. Liu, H. Rui, S. Shen, W. Teng, T. Zhu NASA Goddard Earth Sciences (GES) Data & Information.

© Crown copyright Met Office Probabilistic turbulence forecasts from ensemble models and verification Philip Gill and Piers Buchanan NCAR Aviation Turbulence.

Probabilistic Forecasting. pdfs and Histograms Probability density functions (pdfs) are unobservable. They can only be estimated. They tell us the density,

Short-Range Ensemble Prediction System at INM José A. García-Moya SMNT – INM 27th EWGLAM & 12th SRNWP Meetings Ljubljana, October 2005.

Climate Predictability Tool (CPT) Ousmane Ndiaye and Simon J. Mason International Research Institute for Climate and Society The Earth.

Lecture 3 Topic - Descriptive Procedures Programs 3-4 LSB 4:1-4.4; 4:9:4:11; 8:1-8:5; 5:1-5.2.

Refinement and Evaluation of Automated High-Resolution Ensemble-Based Hazard Detection Guidance Tools for Transition to NWS Operations Kick off JNTP project.

© Crown copyright Met Office Standard Verification System for Long-range Forecasts (SVSLRF) Richard Graham, Met Office Hadley Centre. With acknowledgements.

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Verification and Metrics (CAWCR)

© 2009 UCAR. All rights reserved. ATEC-4DWX IPR, 21−22 April 2009 National Security Applications Program Research Applications Laboratory Ensemble-4DWX.

Quality Improvement Tools CHAPTER SIX SUPPLEMENT McGraw-Hill/Irwin Copyright © 2011 by the McGraw-Hill Companies, Inc. All rights reserved.

Probabilistic Forecasts of Extreme Precipitation Events for the U.S. Hazards Assessment Kenneth Pelman 32 nd Climate Diagnostics Workshop Tallahassee,

Generate ξ m,i for each direction i given H, σ 1 and m (Eq. 2) calculate X’ m,i for each direction i (Eq. 1) given ξ m,i and X m, which corresponds for.

Verification of ensemble systems Chiara Marsigli ARPA-SIMC.

Customize SAS Output Using ODS Joan Dong. The Output Delivery System (ODS) gives you greater flexibility in generating, storing, and reproducing SAS procedure.

Nathalie Voisin 1, Florian Pappenberger 2, Dennis Lettenmaier 1, Roberto Buizza 2, and John Schaake 3 1 University of Washington 2 ECMWF 3 National Weather.

Common verification methods for ensemble forecasts

Verification of ensemble precipitation forecasts using the TIGGE dataset Laurence J. Wilson Environment Canada Anna Ghelli ECMWF GIFS-TIGGE Meeting, Feb.

Extracting probabilistic severe weather guidance from convection-allowing model forecasts Ryan Sobash 4 December 2009 Convection/NWP Seminar Series Ryan.

DOWNSCALING GLOBAL MEDIUM RANGE METEOROLOGICAL PREDICTIONS FOR FLOOD PREDICTION Nathalie Voisin, Andy W. Wood, Dennis P. Lettenmaier University of Washington,

VERIFICATION OF A DOWNSCALING SEQUENCE APPLIED TO MEDIUM RANGE METEOROLOGICAL PREDICTIONS FOR GLOBAL FLOOD PREDICTION Nathalie Voisin, Andy W. Wood and.

DET Module 1 Ensemble Configuration Linda Wharton 1, Paula McCaslin 1, Tara Jensen 2 1 NOAA/GSD, Boulder, CO 2 NCAR/RAL, Boulder, CO 3/8/2016.

Overview of SPC Efforts in Objective Verification of Convection-Allowing Models and Ensembles Israel Jirak, Chris Melick, Patrick Marsh, Andy Dean and.

Probability and Statistics 12/11/2015. Statistics Review/ Excel: Objectives Be able to find the mean, median, mode and standard deviation for a set of.

Verify WRF Forecast using MET

LEPS VERIFICATION ON MAP CASES

TIGGE Archives and Access

University of Washington Ensemble Systems for Probabilistic Analysis and Forecasting Cliff Mass, Atmospheric Sciences University of Washington.

Huaqing Cai, Jim Wilson, James Pinto, Dave Albo and Cindy Mueller

Verifying and interpreting ensemble products

Tom Hopson, Jason Knievel, Yubao Liu, Gregory Roux, Wanli Wu

Dan Petersen Bruce Veenhuis Greg Carbin Mark Klein Mike Bodner

What temporal averaging period is appropriate for MM5 verification?

MOS Developed by and Run at the NWS Meteorological Development Lab (MDL) Full range of products available at:

Post Processing.

Probabilistic forecasts

COSMO-LEPS Verification

Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold

Spatial Statistics A 15 minute Tour….

Verification of Tropical Cyclone Forecasts

Advances in Rfdbk based Verification at DWD

Presentation transcript:

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Ensembles & Probability Fcsts with MET Ensemble Stat Tool Ensemble Mean Fields Probability Fields Rank Histograms Point Stat and Grid Stat Tool Brier Score + Decomposition Reliability Diagrams Receiver Operating Characteristic Diagram + Area Under the Curve Joint/Conditional factorization table Presenter: Tara Jensen

Statistics Stat Analysis MODE Analysis ASCII User Defined Display Ensemble Stat Tool User Graphics Package Analysis User Graphics Package Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Wavelet Stat MODE Grid Stat Ensemble Stat Point Stat STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF PS STAT ASCII NetCDF

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Capabilities Calculates: Ensemble Mean and Standard Deviations fields Ensemble Mean + 1 Standard Deviation fields Ensemble Min and Max fields Ensemble Range field Ensemble Valid Data Count field Ensemble Relative Frequency by threshold fields Ranked Histograms (if Obs Field Provided) Writes: Gridded field of Observation Ranks to a NetCDF file Stat file with Rank Histogram and Ensemble information Observation Rank Matched Pairs

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Usage Usage: ensemble_stat n_ens ens_file_1 \... ens_file_n | ens_file_list config_file [-grid_obs file] [-point_obs file] [-ens_valid time] [-ens_lead time] [-obs_valid_beg time] [-obs_valid_end time] [-obs_lead time] [-outdir path] [-v level] Number of Ensemble members followed by list of ensemble member names OR ens_file_list (the name of an ASCII file with names of members) Config file name Name of gridded or point observed file – Required if Rank Histograms desired (optional) YYYYMMDD[_HH[MMSS]] format sets the ensemble valid time or lead time to be used (optional) YYYYMMDD[_HH[MMSS]] format to set the beginning and end of the matching observation time window (optional) HH[MMSS] format - observation lead time (opt.) Set output directory (optional) Set level of verbosity (optional)

23 configurable parameters – set only a few: Set Ensemble Field to Precip for calculating Ensemble Mean, Std. Dev, Ens Relative Freq fields, etc.. ens_field[ ] = [“61/A24”]; Using only the following thresholds ens_thresh[ ] = [“gt0.0 ge12.7 ge25.4”]; Set ratio of valid members (data values) to total ensemble members required for processing valid_ens_thresh = 0.5; valid_data_thresh = 0.5; Set fcst_field and obs_field to Precip for calculating Rank Histograms fcst_field[ ] = [“61/A24”]; obs_field[ ] = [“ ”]; Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Configuration

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Run ensemble_stat \ 6 sample_fcst/ /*gep*/d01_ _02400.grib \ config/EnsembleStatConfig \ -grid_obs sample_obs/ST4/ST h \ -point_obs out/ascii2nc/precip24_ nc \ -outdir out/ensemble_stat -v 2 NOTE: You can pass in a path with wildcards to pull out the files you would like to process or you can pass in an ASCII filename that contains a list of ensemble members Gridded and Obs field are included for use in calculating Rank Histogram

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Run dakota:/d3/personal/jensen> test_ensemble_stat.sh *** Running Ensemble-Stat on APCP using GRIB forecasts, point observations, and gridded observations *** GSL_RNG_TYPE=mt19937 GSL_RNG_SEED= Configuration File: EnsembleStatConfig_default Ensemble Files[6]: data/sample_fcst/ /arw-fer-gep1/d01_ _02400.grib data/sample_fcst/ /arw-fer-gep5/d01_ _02400.grib data/sample_fcst/ /arw-sch-gep2/d01_ _02400.grib data/sample_fcst/ /arw-sch-gep6/d01_ _02400.grib data/sample_fcst/ /arw-tom-gep3/d01_ _02400.grib data/sample_fcst/ /arw-tom-gep7/d01_ _02400.grib Gridded Observation Files[1]: data/sample_obs/ST4/ST h Point Observation Files[1]: out/ascii2nc/precip24_ nc Processing ensemble field: APCP_24/A … Processing gridded verification APCP_24/A24 versus APCP_24/A24, for observation type MC_PCP, over region FULL, for interpolation method UW_MEAN(1), using pairs Output file: test/ensemble_stat_ _120000V.stat Output file: test/ensemble_stat_ _120000V_rhist.txt Output file: test/ensemble_stat_ _120000V_orank.txt Output file: test/ensemble_stat_ _120000V_ens.nc Output file: test/ensemble_stat_ _120000V_orank.nc

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Run Output from out/ensemble_stat/ensemble_stat_ _120000V_rhist.txt VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_RANK RANK_1 RANK_2 RANK_3 RANK_4 RANK_5 RANK_6 RANK_7 V3.0 WRF _ _ _ _ APCP_24 A24 APCP_24 A24 MC_PCP FULL UW_MEAN 1 NA NA NA NA RHIST Ensemble Mean Ensemble StdDev Prob > 5 mm Prob > 0 mm Output from out/ensemble_stat/ensemble_stat_ _120000V_ens.nc (using ncview)

Rank Histogram Copyright 2010, University Corporation for Atmospheric Research, all rights reserved

Statistics Stat Analysis MODE Analysis ASCII User Defined Display Uses for Output from Ensemble Stat Analysis Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Wavelet Stat MODE Grid Stat Ensemble Stat Point Stat STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF PS STAT ASCII NetCDF

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Probabilistic Fields and Ensemble Relative Frequency

Statistics Stat Analysis MODE Analysis ASCII User Defined Display Probability Fields in Other Tools User Graphics Package Analysis User Graphics Package Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Wavelet Stat MODE Grid Stat Ensemble Stat Point Stat STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF PS STAT ASCII NetCDF

Some of the key settings include: Set fcst_field and obs_field to Precip for calculating Rank Histograms fcst_field[] = [ “APCP/A6/PROB", “APCP/A6/PROB", “APCP/A6/PROB" ]; obs_field[] = [ "APCP/A6", "APCP/A6", "APCP/A6" ]; Set fcst_thresh to multiple “bins” for nx2 contingency table. We have 3 probabilistic fields so bins must be set for each one. fcst_thresh[] = [ "ge0.0 ge0.1 ge0.2 ge0.3 ge0.4 ge0.5 ge0.6 ge0.7 ge0.8 ge0.9 ge1.0", \ "ge0.0 ge0.1 ge0.2 ge0.3 ge0.4 ge0.5 ge0.6 ge0.7 ge0.8 ge0.9 ge1.0", \ "ge0.0 ge0.1 ge0.2 ge0.3 ge0.4 ge0.5 ge0.6 ge0.7 ge0.8 ge0.9 ge1.0"]; Set obs_thresh to appropriate observed thresholds obs_thresh[] = [ "ge12.7", "ge25.4", "ge50.8" ]; Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Grid Stat Probability Configuration

Grid Stat for Probability: Run Make sure Probability Output flags set correctly in Grid-Stat Config file (output flags 7-10) Run Grid-Stat as previously described Output written to.stat file and, if desired, to individual text files: PCT – Probability Contingency Table Counts PSTD – Probability Contingency Table Scores Brier Score, Reliability, Resolution, Uncertainty, Area Under ROC PJC – Joint/Continuous Statistics of Probabilistic Variables Calibration, Refinement, Likelihood, Base Rate PRC – ROC Curve Points for Probabilistic Variables Copyright 2012, University Corporation for Atmospheric Research, all rights reserved

Grid Stat Probability: Example Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Observed Precip Field 4 km Simple km Simple Ensemble Relative Frequency 4 km Nbrhood Method Applied Gaussian smoother applied to each member before calculating relative frequency. Relative frequency calculated from raw field

Grid Stat Probability: Examples Copyright 2012, University Corporation for Atmospheric Research, all rights reserved

A teaser… Spatial Methods Application Copyright 2012, University Corporation for Atmospheric Research, all rights reserved You can use MODE on probability fields also… Probability field threshold = 50% Observed field threshold > 12.7 mm (or 0.5”) In this case: