Verification for the High Impact Weather Prediction Project (HIWPP) and other ESRL/GSD verification work Steve Weygandt, Bonny Strong Bill Moninger, Jeff.

Slides:

Advertisements

Similar presentations

Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF.

Advertisements

Slide 1ECMWF forecast products users meeting – Reading, June 2005 Verification of weather parameters Anna Ghelli, ECMWF.

Chapter 13 – Weather Analysis and Forecasting

Model Evaluation Tools MET. What is MET Model Evaluation Tools ( MET )- a powerful and highly configurable verification package developed by DTC offering:

5 th International Conference of Mesoscale Meteor. And Typhoons, Boulder, CO 31 October 2006 National Scale Probabilistic Storm Forecasting for Aviation.

Report of the Q2 Short Range QPF Discussion Group Jon Ahlquist Curtis Marshall John McGinley - lead Dan Petersen D. J. Seo Jean Vieux.

Adaptation of the Gridpoint Statistical Interpolation (GSI) for hourly cycled application within the Rapid Refresh Ming Hu 1,2, Stan Benjamin 1, Steve.

Rapid Refresh and RTMA. RUC: AKA-Rapid Refresh A major issue is how to assimilate and use the rapidly increasing array of off-time or continuous observations.

The 2014 Warn-on-Forecast and High-Impact Weather Workshop

Demonstration Testbed for the Evaluation of Experimental Models for Tropical Cyclone Forecasting in Support of the NOAA Hurricane Forecast Improvement.

Hourly RUC Convective Probability Forecasts using Ensembles and Radar Assimilation Steve Weygandt Stan Benjamin Forecast Systems Laboratory NOAA.

NWP Verification with Shape- matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1, Edward Tollerud 2, Tara Jensen 1,

Univ of AZ WRF Model Verification. Method NCEP Stage IV data used for precipitation verification – Stage IV is composite of rain fall observations and.

Atmospheric Modeling at RENCI Brian J. Etherton. Atmospheric Modeling at RENCI Focus of RENCI for C- STAR project is to provide modeling support/development.

Tara Jensen for DTC Staff 1, Steve Weiss 2, Jack Kain 3, Mike Coniglio 3 1 NCAR/RAL and NOAA/GSD, Boulder Colorado, USA 2 NOAA/NWS/Storm Prediction Center,

Chapter 13 – Weather Analysis and Forecasting. The National Weather Service The National Weather Service (NWS) is responsible for forecasts several times.

Transitioning CMAQ for NWS Operational AQ Forecasting Jeff McQueen*, Pius Lee*, Marina Tsildulko*, G. DiMego*, B. Katz* R. Mathur,T. Otte, J. Pleim, J.

Jamie Wolff Jeff Beck, Laurie Carson, Michelle Harrold, Tracy Hertneky 15 April 2015 Assessment of two microphysics schemes in the NOAA Environmental Modeling.

Weather Model Development for Aviation Stan Benjamin and Steve Weygandt: Assimilation and Modeling Branch, Chief/Deputy NOAA Earth System Research Laboratory,

1 Agenda Topic: Verification Strategy (Atmosphere) Presented By: Geoff DiMego (NCEP/EMC) Contributors: Stan Benjamin, Stephen Weygandt, Bonny Strong (ESRL/GSD)

1 Ensemble Reforecasts Presented By: Yuejian Zhu (NWS/NCEP) Contributors:

Forecasting and Numerical Weather Prediction (NWP) NOWcasting Description of atmospheric models Specific Models Types of variables and how to determine.

Objective Evaluation of Aviation Related Variables during 2010 Hazardous Weather Testbed (HWT) Spring Experiment Tara Jensen 1*, Steve Weiss 2, Jason J.

Verification Summit AMB verification: rapid feedback to guide model development decisions Patrick Hofmann, Bill Moninger, Steve Weygandt, Curtis Alexander,

GFS Surface Temperature (T2m) Cold Bias over CONUS East Mitigation Experiments Detailed verification at Fanglin’s websites

Assimilation of AIRS Radiance Data within the Rapid Refresh Rapid Refresh domain Haidao Lin Steve Weygandt Ming Hu Stan Benjamin Patrick Hofmann Curtis.

NOAA’s National Weather Service National Digital Forecast Database: Status Update LeRoy Spayd Chief, Meteorological Services Division Unidata Policy Committee.

VERIFICATION OF NDFD GRIDDED FORECASTS IN THE WESTERN UNITED STATES John Horel 1, David Myrick 1, Bradley Colman 2, Mark Jackson 3 1 NOAA Cooperative Institute.

Development of an EnKF/Hybrid Data Assimilation System for Mesoscale Application with the Rapid Refresh Ming Hu 1,2, Yujie Pan 3, Kefeng Zhu 3, Xuguang.

AMB Verification and Quality Control monitoring Efforts involving RAOB, Profiler, Mesonets, Aircraft Bill Moninger, Xue Wei, Susan Sahm, Brian Jamison.

Modification of GFS Land Surface Model Parameters to Mitigate the Near- Surface Cold and Wet Bias in the Midwest CONUS: Analysis of Parallel Test Results.

Verification of Global Ensemble Forecasts Fanglin Yang Yuejian Zhu, Glenn White, John Derber Environmental Modeling Center National Centers for Environmental.

WMO/WWRP Workshop Use of NWP for Nowcasting 25 October 2011 Evaluation of the 3-km High Resolution Rapid Refresh (HRRR) as Nowcast Guidance NOAA/ESRL/GSD.

LAPS / STMAS Verification Activities Steve Albers, Isidora Jankov ESRL / GSD Verification Summit September 2011 Updated Sep 7, 2011.

2006(-07)TAMDAR aircraft impact experiments for RUC humidity, temperature and wind forecasts Stan Benjamin, Bill Moninger, Tracy Lorraine Smith, Brian.

Evaluation of impact of satellite radiance data within the hybrid variational/EnKF Rapid Refresh data assimilation system Haidao Lin Steve Weygandt Ming.

Refinement and Evaluation of Automated High-Resolution Ensemble-Based Hazard Detection Guidance Tools for Transition to NWS Operations Kick off JNTP project.

Impact of Meteorological Inputs on Surface O 3 Prediction Jianping Huang 9 th CMAS Annual Conference Oct. 12, 2010, Chapel, NC.

Transitioning unique NASA data and research technologies to the NWS AIRS Profile Assimilation - Case Study results Shih-Hung Chou, Brad Zavodsky Gary Jedlovec,

Evaluation of radiance data assimilation impact on Rapid Refresh forecast skill for retrospective and real-time experiments Haidao Lin Steve Weygandt Stan.

Statistical Post Processing - Using Reforecast to Improve GEFS Forecast Yuejian Zhu Hong Guan and Bo Cui ECM/NCEP/NWS Dec. 3 rd 2013 Acknowledgements:

Renewable Energy Requirements Rapid Update Analysis/Nowcasting Workshop - June 4, 2015 Sue Ellen Haupt Director, Weather Systems & Assessment Program Research.

GSI applications within the Rapid Refresh and High Resolution Rapid Refresh 17 th IOAS-AOLS Conference 93 rd AMS Annual Meeting 9 January 2013 Patrick.

1 Aviation Forecasting – Works in Progress NCVF – Ceiling & Visibility CoSPA – Storm Prediction A Joint Effort Among: MIT Lincoln Laboratory NCAR – National.

NCEP Production Suite Review

Recent and Future Advancements in Convective-Scale Storm Prediction with the High- Resolution Rapid Refresh (HRRR) Forecast System NOAA/ESRL/GSD/AMB Curtis.

Assimilation of AIRS SFOV Profiles in the Rapid Refresh Rapid Refresh domain Haidao Lin Ming Hu Steve Weygandt Stan Benjamin Assimilation and Modeling.

Lightning data assimilation in the Rapid Refresh and evaluation of lightning diagnostics from HRRR runs Steve Weygandt, Ming Hu, Curtis Alexander, Stan.

Extracting probabilistic severe weather guidance from convection-allowing model forecasts Ryan Sobash 4 December 2009 Convection/NWP Seminar Series Ryan.

Nowcasting Convection Fusing 0-6 hour observation- and model-based probability forecasts WWRP Symposium on Nowcasting and Very Short Range Forecasting.

Satellite Data Assimilation Activities at CIMSS for FY2003 Robert M. Aune Advanced Satellite Products Team NOAA/NESDIS/ORA/ARAD Cooperative Institute for.

2004 Developments in Aviation Forecast Guidance from the RUC Stan Benjamin Steve Weygandt NOAA / Forecast Systems Lab NY Courtesy:

Global vs mesoscale ATOVS assimilation at the Met Office Global Large obs error (4 K) NESDIS 1B radiances NOAA-15 & 16 HIRS and AMSU thinned to 154 km.

WARN ON FORECAST AND HIGH IMPACT WEATHER WORKSHOP 09 February 2012 Use of Rapid Updating Meso‐ and Storm‐scale Data Assimilation to Improve Forecasts of.

1 Recent AMDAR (MDCRS/ACARS) Activities at GSD New AMDAR-RUC database that helps evaluate AMDAR data quality Optimization study that suggests data can.

1 Application of MET for the Verification of the NWP Cloud and Precipitation Products using A-Train Satellite Observations Paul A. Kucera, Courtney Weeks,

HRRR Primary FAA-CoSPA NCEPESRL/GSD/AMB RAP Dev1 RAPv2 Primary HRRR Dev1 RAPv1 NCO RAP Dev2 RAP Retro HRRR Retro Retrospective Real-Time HRRR (and RAP)

RUC Convective Probability Forecasts using Ensembles and Hourly Assimilation Steve Weygandt Stan Benjamin Forecast Systems Laboratory NOAA.

Encast Global forecasting.

A few examples of heavy precipitation forecast Ming Xue Director

Rapid Update Cycle-RUC

Aircraft weather observations: Impacts for regional NWP models

Aviation Forecast Guidance from the RUC

Aircraft-based Observations:

Rapid Update Cycle-RUC Rapid Refresh-RR High Resolution Rapid Refresh-HRRR RTMA.

WMO NWP Wokshop: Blending Breakout

University of Washington Center for Science in the Earth System

New Developments in Aviation Forecast Guidance from the RUC

Comparison of different combinations of ensemble-based and variational data assimilation approaches for deterministic NWP Mark Buehner Data Assimilation.

Short Range Ensemble Prediction System Verification over Greece

Presentation transcript:

Verification for the High Impact Weather Prediction Project (HIWPP) and other ESRL/GSD verification work Steve Weygandt, Bonny Strong Bill Moninger, Jeff Hamilton, Xue Wei, Randy Pierce AMS 23 rd NWP Conf. 2 July 2015

HIWPP Verification Goals Starting with existing capabilities, build a verification system for development / improvement of operational weather prediction models. The verification system will include traditional skill measures as well as verification of sensible weather elements Designed for model development / improvement Quick turn around, easily modified / customized Interactive, on-the-fly stratification of results  DATABASE Start from existing pieces, add advanced capabilities Collaboration including GSD and EMC Initial application for HIWPP hydrostatic models

Evolving the HIWPP Verification System Deterministic Ensemble g2o, g2g MySQL Database (interactive) Initial (EMC) GSD/EMB Advanced System Ensemble verification in database with web-interface Global precip verification (CRPSS for ensemble precip) Trop. cyclone verification Scorecard verification NGGPS Deterministic g2o, g2g DATABASE Mid- Range Traditional and novel techniques MODE Metviewer MET Add... Fully integrated verification to support NOAA operations, R2O VSDB files

EMB Verification System Motivation and design goals “Model Developer” verification -- Fast computation and display of verification results (real-time parallel and retrospective cycles, individual cases) -- Quick stratification by key parameters (metric, threshold, scale, valid time, initial time, forecast length, region)  Easily accessible web-based display of verification results  Ability to quickly examine aggregate statistics AND single-case plots in complementary manner Database with well designed user-interface essential for achieving these attributes!

Model Region Scale Averaging period Metric Threshold Forecast Length Valid time Date Range Many R/T runs and retros Web interface for “time-series” view mode (EMB reflectivity example )

ANOMALY CORRELATION COEFFICIENT time series, die-off UPPER-AIR (radiosonde) time series, vertical profile SURFACE T, Td, RH, wind (METAR, mesonet) time series, diurnal, lead-time PRECIPITATION (CPC, 6-hSYNOP) time series, diurnal, threshold REFLECTIVITY (MRMS) time series, diurnal, lead-time, threshold Sample “view modes” from EMB verification | | | | | | | | (in)

Raob - Statistics based on native files (every 10 hPa from the surface up) verified against (nearly) all RAOB data. Interactive time-series plotsInteractive time-series plots... (beta version)(beta version) Interactive vertical profilesInteractive vertical profiles... (beta version)(beta version) Find Model-RAOB ResidualsFind Model-RAOB Residuals... (beta version)(beta version) AMDAR data. Every hour, highly non-uniform coverage Anomaly correlation statistics for the FIM and GFS. GFS vs. FIM - last 31 days - 5-day 6-day 7-day5-day6-day7-day Interactive time-series plotsInteractive time-series plots... (beta version)(beta version) Dieoff (anomaly correlation vs. forecast projection) plots NCEP GFS stats, including anomaly correlation for latest 31 days NCEP GFS stats for previous months Ceiling hourly s tatistics verified against METAR data. Precipitation Type statistics verified against METARs Visibility hourly statistics verified against METAR data. Cloud and Solar Insolation verification against GSIP, SURFRAD/ISIS,METARS. Surface statistics. Mesonet stations verified against forecasts. Experimental Composite Reflectivity Statistics. NSSL vs. HRRR, RAP, RUC, NAM Hourly Time Series for various models, BETA versionHourly Time Series for various models, BETA version, Valid Time Statistics for various models, BETA version Lead Time Statistics for various models, BETA version Vertically Integrated Liquid (VIL) Statistics. NSSL vs. HRRR, NAM_NEST Echo Top Height Statistics. NSSL vs. HRRR, NAM_NEST Experimental Sub-24HR Precipitation Statistics. StageIV vs. HRRR, RAP, RUC, NAM Experimental Sub-24Hr Time Series statistics for various models, BETA versionSub-24Hr Time Series statistics for various models, BETA version, Sub-24Hr Threshold Statistics for various models, BETA version 24HR Precipitation Statistics. (CPC,StageIV) vs. HRRR, RAP, RUC, NAM Experimental 24Hr Time Series statistics for various models, BETA version24Hr Time Series statistics for various models, BETA version, 24Hr Threshold Statistics for various models, BETA version Convective Probability Statistics. NCWD vs. HCPF, RCPF, CCFP, LLProb Experimental Hourly Time Series statistics for various models, BETA version CSI vs Bias Figures for various models, BETA version Relative Operating Characteristic (ROC) Curves for various models, BETA version Reliability Diagrams for various models, BETA version Wind Profiler statistics. NPN/CAP vs. RAP, HRRR. Experimental Time Series statistics for various models, BETA version Non-java version Vertical structure, BETA version Wind Tower statistics. vs. RAP, HRRR, RRrapx. Restricted Cloud and Solar Insola tion statistics from GOES GSIP vs. HRRR, Experimental Time Series statistics and plan-view (beta) EMB interactive verification ( Total database size ~ 800 GB

HRRR-dev HRRR RR-dev w/ Pseudo-obs HRRR-dev better HRRR better Reflectivity (> 25 dBZ) CSI Eastern US on 40 km grid (3-day avg) Models Thresh “Time series” mode Metric Region Scale Sample application of “time-series” stats Difference Upgraded in RR-prim HRRR-dev Longer time-step RR-dev Added shorter vert. length-scales in RR-dev/GSI Upgraded In HRRR

HRRR (and RAP) Future MilestonesHRRR Milestones 15-min Validation of HRRR forecasts Downward shortwave flux at surface Observed (14 SURFRAD sites) HRRR mean (12z runs ) 15 minute HRRR output 1 minute SURFRAD obs Drag zoom

HRRR (and RAP) Future MilestonesHRRR Milestones 15-min Validation of HRRR forecasts Downward shortwave flux at surface 29 May 2015 Observed (individual sites) HRRR mean (12z run ) Bondville, IL Table Mtn, CO Bismarck, ND

12-h forecasts of downward shortwave flux at surface 18 May May 2014 RAP mean HRRR mean observed mean (14 SURF- RAD / ISIS stations) Time of Day (UTC) 2-m Temperature (K), CONUS 2-m Dewpoint (K), CONUS too warm… too dry… HRRR RAP SURFRAD verification guiding model physics improvements

2m Temperature RMS (K) Rapid Refresh V2 vs. V3 Surface Temperature verification CONUS 1June 31 Aug h fcst 3-h fcst 1-h fcst Analysis 2m Temperature bias (K) 6-h fcst 3-h fcst 1-h fcst Analysis 2 K 0 K RAP V2RAP V3 CONUS 1 April 31 May m Temperature RMS (K) 2m Temperature bias (K) Warm Cold

Link from HIWPP top page (under “resources”) “Initial” verification system (VSDB)

Link from HIWPP (under “resources”) “Initial” verification system (VSDB)

30 day period500 mb height +5 day fcst anom. corr. N Hemi. FIM9 GFS oper NAVGEM prod ESRL running “Initial” verification system (NCEP-VSDB) FIM9zeus NAVGEM exp (Fanglin Yang)

200 mb wind RMSE Global 00z lead-time 500 mb height Anom. Corr. North Hemi. 00z die-off “Initial” verification system (VSDB) Difference from GFS Difference from GFS

Comparison: Mid-range vs. Initial systems --- GFS ---- FIM8 500 mb height ACC + 8 day fcst Global, 00z Time series Mid-range (database) Initial (VSDB)

“Initial” verification system (VSDB)

“Mid-range” verification system (database) + 48h fcst global wind RMSE (vs. raob) --- GFS ---- FIM9 0.0 DIFF (FIM9 – GFS) Drag Zoom

“Mid-range” verification system vs. raobs (database) --- GFS ---- FIM Difference (Confidence intervals) 0.0 DIFF (FIM9 -- GFS) m/s (95% confidence)

Current capabilities 1. New web-interface (jquery, html5) 2. GLOBAL METAR-based T / Td / wind verification (database) 3. GLOBAL SYNOP-based precipitation verification (database)  Work toward CMORPH global gridded precipitation verification  Work toward global CRPSS precipitation verification of ensembles Future capabilities 1. Enhanced ensembles verification 2. Fully incorporated tropical cyclone verification 3. Fully incorporated scorecard verification “Advanced” verification system

New html5 display (Model Assessment Tool Suite) mb Temperature RMS Error Time Series --- NAVGEM-exp --- GFS ---- FIM9 (zeus) + 5 day fcst Global

mb Temperature RMS Error Time Series --- NAVGEM-exp --- GFS ---- FIM9 + 5 day fcst Global New html5 display (Model Assessment Tool Suite)

mb Temperature RMS Error Time Series FIM9 GFS NAVGEM Global + 5 day fcst New html5 display (Model Assessment Tool Suite)

Comparison: html5 vs. java display mb Temperature RMS Error Time Series + 5 day fcst Global --- NAVGEM-exp --- GFS ---- FIM9 Both plots accessing the same database

GLOBAL METAR (sfc T/Td/wind) and SYNOP (precip) Total # reports (~ 4500) METARs # of Reports Total # reports (~3500) > 0.10” / 6h > 0.25” / 6h > 0.01” / 6h GLOBAL 12-h fcst of 6h precip # of Reports ~ 3600 SYNOP precip reports every 6h ~ About 15% of the reports are non-zero

METAR surface data – where reports from?? Snapshot from 18z 20 January 2015 Temperature, Dewpoint, Wind

SYNOP precip data – where reports from?? -- Typical spatial coverage (for directly measured + inferred) Purple dots are reports of 0.00” precip, other colors are non-zero amounts Snapshot from 12z 16 December 2014 Quirk No pcp

QMORPH SYNOP +6h accumulated precipitation ending 18z 15 Oct. SYNOP precip data – matching CMORPH

Global SYNOP precipitation verification GLOBAL 6h precip 3d average.52 FIM-ex > 0.10”.47 GFS-op > 0.10”.33 GFS-op > 0.25”.32 FIM-ex > 0.25” TSS +12 forecast +36 forecast.46 FIM-ex > 0.10”.42 GFS-op > 0.10”.27 GFS-op > 0.25”.28 FIM-ex > 0.25”

Coordinating Verification Efforts ESRL GSD Bonny Strong, Steve Weygandt, Bill Moninger, Jeff Hamilton, Xue Wei, Randy Pierce NCEP EMC Binbin Zhou, Perry Shafran, Fanglin Yang, Yuejian Zhu, Ying Lin NCAR DTC Tara Jensen, John Halley-Gotway, Tatiana Burek NGGPS Ivanka Stajner Verification for operational model improvement!