1 Agenda Topic: Verification Strategy (Atmosphere) Presented By: Geoff DiMego (NCEP/EMC) Contributors: Stan Benjamin, Stephen Weygandt, Bonny Strong (ESRL/GSD) Fanglin Yang (IMSG at NCEP/EMC) Ying Lin (NCEP/EMC), Tara Jensen (NCAR/DTC) Perry Shafran, Binbin Zhou (IMSG at NCEP/EMC) Guang Ping Lou, Tracey Dorian (IMSG at NCEP/EMC) Keith Brill (IMSG at NCEP/WPC)

2 Operational System Attributes-1 System NameAcronymAreal Coverage Horz Res Cycle FreqFcst Length Grid versus observation: Verifies NAM, NAM nests, FireWx nest, HiResWindow, DGEX, RAP, HRRR, SREF, CMAQ ozone & PM2.5, and GFS plus developmental parallels of all models grid2obsVarious3-km thru 12-km Hourly for NAM/nests/H RRR, RAP, 3- hourly for all others 15 hr thru 87 hr Grid versus grid (gridded truth) including single and ensemble grid2gid Global, CONUS, Alaska Various QPF verificationpcpverif ConUS: 24h/6h FSS; 24h/3h SL1L2 and contingenc y table- based (FHO) scores. AK, HI, PR: 24h SL1L2 and FHO. Multiple horizon talgrids ranging from 4.8km to 80km 24h, 6h, 3h 84h for NAM, up to 192h for GFS and DGEX. Other models: to the extent of forecast length or available data (in case of int’l models) Cyclone track: Using model output data including analyses and forecasts. The models include, GFS, GEFS, NAM, SREF, CMC, CENS, ECMWF, EENS, NVGM, FENS, UKMET TrackerGlobal & regional Vario us 2- & 4-per day 84hr- 180hr

3 Operational System Attributes-2 System NameAcronymAreal CoverageHorz Res Cycle Freq Fcst Length Grid versus observation (for global NWP models, including GFS and ECMWF) grd2obNH, SH, Tropics, and CONUS 1.0-deg4/day168hr Grid versus grid (gridded truth) (for global NWP models, see grid2gidNH, SH, Tropics, and CONUS 2.5-deg 4/day384hr QPF verification (for global NWP models, including objected-oriented and ETS scores) MODECONUS13km4/day168hr Cyclone track (for global NWP models)hurtrackAtlantic, Western Pacific, Eastern Pacific 0.25-deg4/day126hr GFS/GDAS data assimilation monitoringDAglobalvarious4/day6hr

4 Why System(s) are Operational  Primary stakeholders and requirement drivers Developers and users both internal and external Operational system decision makers, funding / portfolio / system managers Research community (potential code contributors), private sector, universities  What products are the verification tools contributing to? Improvement of NWP systems via long-term & comparative record of performance Myriad of stats, verification statistic database (VSDB), MySQL database Web pages & display using FVS, Perl-based web, GRADS, and/or METviewer  What product aspects are you trying to improve with your development plans? Enhance, streamline & simplify 3 components of verification: 1. calculation of error/skill scores; 2. database storage & retrieval; 3. Web display (static images and interactive capability) Community code model [METviewer at EMC and flexible html viewer (MATS = Model Assessment Tool Suite) maintained within repository at ESRL/GSD/EMB for HIWPP] Rapid Update Analysis would provide gridded truth for many fields  Top 3 System Performance Strengths Flexible: deterministic & ensemble guidance, myriad of stats & metrics, portable Content for all models, long records of stats, stored in database, retrievable Accessible, locally at least, familiar to developers, interactivity  Top 3 System Performance Challenges Separate complex systems across the agency & enterprise, no common repository Access, reliability & user support is lacking, more user friendly, limitations remain Lack of reliable truth fields and need for quality control of obs

5 System Evolution Over the Next 5 Years  Major forcing factors Major dev efforts e.g. NGGPS, WOF require fast, flexible verif to make reapid progress Need a shared & unified verification system that spans global-scale to storm-scale Challenges of verifying/detecting features, svr wx, clouds, etc & their attributes  Science and development priorities Unification & sharing of common codes for all required verification capabilities Enhanced diagnostics & graphics including spatial distribution of errors & model difs Enhancements: obs impact monitoring, octahedral & cube-sphere projections, sub 1km spacings, using more sat prod, non-PrepBUFR (e.g. Aeronet) and RapUpdAnal Ens. PBS stat type to support reliability diagrams, Brier Score, decomp of Brier Score, and Brier Skill Score against the sample climatology.  What are you top challenges to evolving the system(s) to meet stakeholder requirements? Unification & sharing of common codes for all required verification capabilities Resources: personnel, compute, storage, training, common computer workspace Utilizing & QC of new data sources, and validating new techniques for verification  Potential opportunities for simplification going forward Building a unified, fast, flexible, comprehensive verification system (incorporating MET/METViewer and HIWPP/EMB elements / features) will GREATLY facilitate the most rapid skill improvement for all NCEP model systems [WPC adopting MET too] Partnering is more common e.g. Stony Brook R2O proj & hazards detection Using scripting languages (e.g. python) and workflow managers (e.g. Roccoto)

6 Top 3 Things You Need From the UMAC 1.Push for INVESTMENT in and unification of verification across the enterprise: MET/METviewer versus MATS versus in-house written systems. EMC, NGGPS, WOF, FACETS really need this.