Presentation is loading. Please wait.

Presentation is loading. Please wait.

Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Ensembles & Probability Fcsts with MET Ensemble Stat Tool.

Published byBennett Taylor Modified over 9 years ago

Similar presentations

Presentation on theme: "Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Ensembles & Probability Fcsts with MET Ensemble Stat Tool."— Presentation transcript:

1 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Ensembles & Probability Fcsts with MET Ensemble Stat Tool Ensemble Mean Fields Probability Fields Rank Histograms Point Stat and Grid Stat Tool Brier Score + Decomposition Reliability Diagrams Receiver Operating Characteristic Diagram + Area Under the Curve Joint/Conditional factorization table Presenter: Tara Jensen

2 Statistics Stat Analysis MODE Analysis ASCII User Defined Display Ensemble Stat Tool User Graphics Package Analysis User Graphics Package Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Wavelet Stat MODE Grid Stat Ensemble Stat Point Stat STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF PS STAT ASCII NetCDF

3 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Capabilities Calculates: Ensemble Mean and Standard Deviations fields Ensemble Mean + 1 Standard Deviation fields Ensemble Min and Max fields Ensemble Range field Ensemble Valid Data Count field Ensemble Relative Frequency by threshold fields Ranked Histograms (if Obs Field Provided) Writes: Gridded field of Observation Ranks to a NetCDF file Stat file with Rank Histogram and Ensemble information Observation Rank Matched Pairs

4 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Usage Usage: ensemble_stat n_ens ens_file_1 \... ens_file_n | ens_file_list config_file [-grid_obs file] [-point_obs file] [-ens_valid time] [-ens_lead time] [-obs_valid_beg time] [-obs_valid_end time] [-obs_lead time] [-outdir path] [-v level] Number of Ensemble members followed by list of ensemble member names OR ens_file_list (the name of an ASCII file with names of members) Config file name Name of gridded or point observed file – Required if Rank Histograms desired (optional) YYYYMMDD[_HH[MMSS]] format sets the ensemble valid time or lead time to be used (optional) YYYYMMDD[_HH[MMSS]] format to set the beginning and end of the matching observation time window (optional) HH[MMSS] format - observation lead time (opt.) Set output directory (optional) Set level of verbosity (optional)

5 23 configurable parameters – set only a few: Set Ensemble Field to Precip for calculating Ensemble Mean, Std. Dev, Ens Relative Freq fields, etc.. ens_field[ ] = [“61/A24”]; Using only the following thresholds ens_thresh[ ] = [“gt0.0 ge12.7 ge25.4”]; Set ratio of valid members (data values) to total ensemble members required for processing valid_ens_thresh = 0.5; valid_data_thresh = 0.5; Set fcst_field and obs_field to Precip for calculating Rank Histograms fcst_field[ ] = [“61/A24”]; obs_field[ ] = [“ ”]; Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Configuration

6 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Run ensemble_stat \ 6 sample_fcst/2009123112/*gep*/d01_2009123112_02400.grib \ config/EnsembleStatConfig \ -grid_obs sample_obs/ST4/ST4.2010010112.24h \ -point_obs out/ascii2nc/precip24_2010010112.nc \ -outdir out/ensemble_stat -v 2 NOTE: You can pass in a path with wildcards to pull out the files you would like to process or you can pass in an ASCII filename that contains a list of ensemble members Gridded and Obs field are included for use in calculating Rank Histogram

7 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Run dakota:/d3/personal/jensen> test_ensemble_stat.sh *** Running Ensemble-Stat on APCP using GRIB forecasts, point observations, and gridded observations *** GSL_RNG_TYPE=mt19937 GSL_RNG_SEED=1814440524 Configuration File: EnsembleStatConfig_default Ensemble Files[6]: data/sample_fcst/2009123112/arw-fer-gep1/d01_2009123112_02400.grib data/sample_fcst/2009123112/arw-fer-gep5/d01_2009123112_02400.grib data/sample_fcst/2009123112/arw-sch-gep2/d01_2009123112_02400.grib data/sample_fcst/2009123112/arw-sch-gep6/d01_2009123112_02400.grib data/sample_fcst/2009123112/arw-tom-gep3/d01_2009123112_02400.grib data/sample_fcst/2009123112/arw-tom-gep7/d01_2009123112_02400.grib Gridded Observation Files[1]: data/sample_obs/ST4/ST4.2010010112.24h Point Observation Files[1]: out/ascii2nc/precip24_2010010112.nc -------------------------------------------------------------------------------- Processing ensemble field: APCP_24/A24 -------------------------------------------------------------------------------- … Processing gridded verification APCP_24/A24 versus APCP_24/A24, for observation type MC_PCP, over region FULL, for interpolation method UW_MEAN(1), using 15480 pairs. -------------------------------------------------------------------------------- Output file: test/ensemble_stat_20100101_120000V.stat Output file: test/ensemble_stat_20100101_120000V_rhist.txt Output file: test/ensemble_stat_20100101_120000V_orank.txt Output file: test/ensemble_stat_20100101_120000V_ens.nc Output file: test/ensemble_stat_20100101_120000V_orank.nc

8 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Ensemble Stat Tool: Run Output from out/ensemble_stat/ensemble_stat_20100101_120000V_rhist.txt VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_RANK RANK_1 RANK_2 RANK_3 RANK_4 RANK_5 RANK_6 RANK_7 V3.0 WRF 240000 20100101_120000 20100101_120000 240000 20100101_120000 20100101_120000 APCP_24 A24 APCP_24 A24 MC_PCP FULL UW_MEAN 1 NA NA NA NA RHIST 15480 7 4517 2133 1885 1681 1759 1519 1986 Ensemble Mean Ensemble StdDev Prob > 5 mm Prob > 0 mm Output from out/ensemble_stat/ensemble_stat_20100101_120000V_ens.nc (using ncview)

9 Rank Histogram Copyright 2010, University Corporation for Atmospheric Research, all rights reserved

10 Statistics Stat Analysis MODE Analysis ASCII User Defined Display Uses for Output from Ensemble Stat Analysis Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Wavelet Stat MODE Grid Stat Ensemble Stat Point Stat STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF PS STAT ASCII NetCDF

11 Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Probabilistic Fields and Ensemble Relative Frequency

12 Statistics Stat Analysis MODE Analysis ASCII User Defined Display Probability Fields in Other Tools User Graphics Package Analysis User Graphics Package Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Wavelet Stat MODE Grid Stat Ensemble Stat Point Stat STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF PS STAT ASCII NetCDF

13 Some of the key settings include: Set fcst_field and obs_field to Precip for calculating Rank Histograms fcst_field[] = [ “APCP/A6/PROB", “APCP/A6/PROB", “APCP/A6/PROB" ]; obs_field[] = [ "APCP/A6", "APCP/A6", "APCP/A6" ]; Set fcst_thresh to multiple “bins” for nx2 contingency table. We have 3 probabilistic fields so bins must be set for each one. fcst_thresh[] = [ "ge0.0 ge0.1 ge0.2 ge0.3 ge0.4 ge0.5 ge0.6 ge0.7 ge0.8 ge0.9 ge1.0", \ "ge0.0 ge0.1 ge0.2 ge0.3 ge0.4 ge0.5 ge0.6 ge0.7 ge0.8 ge0.9 ge1.0", \ "ge0.0 ge0.1 ge0.2 ge0.3 ge0.4 ge0.5 ge0.6 ge0.7 ge0.8 ge0.9 ge1.0"]; Set obs_thresh to appropriate observed thresholds obs_thresh[] = [ "ge12.7", "ge25.4", "ge50.8" ]; Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Grid Stat Probability Configuration

14 Grid Stat for Probability: Run Make sure Probability Output flags set correctly in Grid-Stat Config file (output flags 7-10) Run Grid-Stat as previously described Output written to.stat file and, if desired, to individual text files: PCT – Probability Contingency Table Counts PSTD – Probability Contingency Table Scores Brier Score, Reliability, Resolution, Uncertainty, Area Under ROC PJC – Joint/Continuous Statistics of Probabilistic Variables Calibration, Refinement, Likelihood, Base Rate PRC – ROC Curve Points for Probabilistic Variables Copyright 2012, University Corporation for Atmospheric Research, all rights reserved

15 Grid Stat Probability: Example Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Observed Precip Field 4 km Simple 32-35 km Simple Ensemble Relative Frequency 4 km Nbrhood Method Applied Gaussian smoother applied to each member before calculating relative frequency. Relative frequency calculated from raw field

16 Grid Stat Probability: Examples Copyright 2012, University Corporation for Atmospheric Research, all rights reserved

17 A teaser… Spatial Methods Application Copyright 2012, University Corporation for Atmospheric Research, all rights reserved You can use MODE on probability fields also… Probability field threshold = 50% Observed field threshold > 12.7 mm (or 0.5”) In this case:

Download ppt "Copyright 2012, University Corporation for Atmospheric Research, all rights reserved Verifying Ensembles & Probability Fcsts with MET Ensemble Stat Tool."

Similar presentations

ECMWF Slide 1Met Op training course – Reading, March 2004 Forecast verification: probabilistic aspects Anna Ghelli, ECMWF.

ECMWF Slide 1Met Op training course – Reading, March 2004 Forecast verification: probabilistic aspects Anna Ghelli, ECMWF.
Severe Weather Forecasts

Severe Weather Forecasts
Model Evaluation Tools MET. What is MET Model Evaluation Tools ( MET )- a powerful and highly configurable verification package developed by DTC offering:

Model Evaluation Tools MET. What is MET Model Evaluation Tools ( MET )- a powerful and highly configurable verification package developed by DTC offering:
14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic 14-16 May 2001.

14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic May 2001.
Climate Predictability Tool (CPT)

Climate Predictability Tool (CPT)
Verification of probability and ensemble forecasts

Verification of probability and ensemble forecasts
Gridded OCF Probabilistic Forecasting For Australia For more information please contact © Commonwealth of Australia 2011 Shaun Cooper.

Gridded OCF Probabilistic Forecasting For Australia For more information please contact © Commonwealth of Australia 2011 Shaun Cooper.
Model Evaluation Tools (MET): An Overview. MET Verification Techniques Standard verification methods –gridded model data to point-based observations.

Model Evaluation Tools (MET): An Overview. MET Verification Techniques Standard verification methods –gridded model data to point-based observations.
NWP Verification with Shape- matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1, Edward Tollerud 2, Tara Jensen 1,

NWP Verification with Shape- matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1, Edward Tollerud 2, Tara Jensen 1,
An Investigation of Cool Season Extratropical Cyclone Forecast Errors Within Operational Models Brian A. Colle 1 and Michael Charles 1,2 1 School of Marine.

An Investigation of Cool Season Extratropical Cyclone Forecast Errors Within Operational Models Brian A. Colle 1 and Michael Charles 1,2 1 School of Marine.
Exploring the Use of Object- Oriented Verification at the Hydrometeorological Prediction Center Faye E. Barthold 1,2, Keith F. Brill 1, and David R. Novak.

Exploring the Use of Object- Oriented Verification at the Hydrometeorological Prediction Center Faye E. Barthold 1,2, Keith F. Brill 1, and David R. Novak.
JEFS Status Report Department of Atmospheric Sciences University of Washington Cliff Mass, Jeff Baars, David Carey JEFS Workshop, August 28-29.

JEFS Status Report Department of Atmospheric Sciences University of Washington Cliff Mass, Jeff Baars, David Carey JEFS Workshop, August
Univ of AZ WRF Model Verification. Method NCEP Stage IV data used for precipitation verification – Stage IV is composite of rain fall observations and.

Univ of AZ WRF Model Verification. Method NCEP Stage IV data used for precipitation verification – Stage IV is composite of rain fall observations and.
MOS Performance MOS significantly improves on the skill of model output. National Weather Service verification statistics have shown a narrowing gap between.

MOS Performance MOS significantly improves on the skill of model output. National Weather Service verification statistics have shown a narrowing gap between.
Juan Ruiz 1,2, Celeste Saulo 1,2, Soledad Cardazzo 1, Eugenia Kalnay 3 1 Departamento de Cs. de la Atmósfera y los Océanos (FCEyN-UBA), 2 Centro de Investigaciones.

Juan Ruiz 1,2, Celeste Saulo 1,2, Soledad Cardazzo 1, Eugenia Kalnay 3 1 Departamento de Cs. de la Atmósfera y los Océanos (FCEyN-UBA), 2 Centro de Investigaciones.
Ensemble Post-Processing and it’s Potential Benefits for the Operational Forecaster Michael Erickson and Brian A. Colle School of Marine and Atmospheric.

Ensemble Post-Processing and it’s Potential Benefits for the Operational Forecaster Michael Erickson and Brian A. Colle School of Marine and Atmospheric.
Tara Jensen for DTC Staff 1, Steve Weiss 2, Jack Kain 3, Mike Coniglio 3 1 NCAR/RAL and NOAA/GSD, Boulder Colorado, USA 2 NOAA/NWS/Storm Prediction Center,

Tara Jensen for DTC Staff 1, Steve Weiss 2, Jack Kain 3, Mike Coniglio 3 1 NCAR/RAL and NOAA/GSD, Boulder Colorado, USA 2 NOAA/NWS/Storm Prediction Center,
Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex Lisa Goddard and Simon Mason International Research Institute for.

Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex Lisa Goddard and Simon Mason International Research Institute for.
Performance of the MOGREPS Regional Ensemble

Performance of the MOGREPS Regional Ensemble
Verification has been undertaken for the 3 month Summer period (30/05/12 – 06/09/12) using forecasts and observations at all 205 UK civil and defence aerodromes.

Verification has been undertaken for the 3 month Summer period (30/05/12 – 06/09/12) using forecasts and observations at all 205 UK civil and defence aerodromes.

Similar presentations

Ads by Google