PREPARED BY BRADLEY ZAVODSKY NASA/MSFC SHORT-TERM PREDICTION RESEARCH AND TRANSITION (SPORT) CENTER SPoRT MET Scripts Tutorial

Effectively evaluating model performance requires a combination of quantitative metrics and case studies SPoRT does not only transition data and products SPoRT values transition of capabilities that enable NWS partners to perform evaluations that support forecaster-led conference presentations and journal articles Purpose of SPoRT MET Scripts SPoRT directly interacts with NWS forecasters

Purpose of SPoRT MET Scripts Model Evaluation Tools (MET) is a software package developed by NCAR that contains a number of executable programs that will: Reformat observations Match model grid to observations Perform statistical evaluation Steep learning curve due to missing pieces Creating ASCII files from MADIS Dynamic scripts to easily run the software Open source plotting scripts to visualize stats

Tarball Contents Once unzipped and untarred, a number of directories, Perl scripts (*.pl), Perl modules (*.pm), and a namelist.met file should appear Namelist is modified by users to configure what variables and statistics are to be generated for the current run Scripts run the MET workflow Modules contain subroutines and code used by multiple scripts Directories contain documentation, configuration templates used by scripts, or are placeholders where scripts dump data Users should only modify the namelist.met file to configure What follows is an overview of the various components that make up the SPoRT MET Scripts

Scripts: runSPoRTMETScripts.pl Orange dashed circle in workflow that wraps and drives the entire set of scripts and modules Contains a series of true/false statements that are read from namelist.met to determine which parts of MET will be run Uses readNamelist.pm to extract the necessary information needed to run the scripts

Scripts: madisFormatAscii2Nc.pl Orange box, ASCII Point Obs under Input, and ASCII2NC under Reformat in workflow Creates temporary MET-formatted ASCII file needed to run ASCII2NC Runs ASCII2NC to create NetCDF files used by Point Stat (output in pointData directory) sfcobs_YYYYMMDD_HHHH.nc upperobs_YYYYMMDD_HHHH.nc Automatically accesses MADIS FTP server to obtain files for each case study date and hour Users need MADIS account Creating ASCII files from MADIS

Scripts: runPointStat.pl Green Point Stat circle under Statistics in workflow Runs Point Stat to match point observations (e.g. METARs and RAOBs) to the nearest grid point in the model output field to generate text files to be read into Stat Analysis (output in pointStatOutput directory) point_stat_EXP_LEV_FF0000L_YYYYYMMDD_VV0000V_*.stat EXP = 5 character experiment name LEV = surface (sfc) or pressure level (PPPmb) FF = forecast hour time YYYYMMDD = valid year, month, & day VV = valid hour

Scripts: runPointStatAnalysis.pl Green Stat Analysis circle under Analysis in workflow Uses the *.stat files output from Point Stat to generate statistics comparing the model output to the observations (output in pointStatAnalysisOutput) LOC_YYYYMMDDFF_VAR_POLY_LEV_EXP_MPR.txt LOC = surface land (lnd), surface water (wtr), or upper air (upa) YYYYMMDDFF = year, month, day, and forecast hour VAR = variable POLY = verification subdomain LEV = surface (sfc) or pressure level (PPPmb) EXP = experiment name

Scripts: runGridStat.pl Green GenPolyMask and PCP Combine circles under Reformat and green Grid Stat circle under Statistics in workflow Maps gridded verification data to model forecast grid and produces files of the differences (gridStatOutput directory) grid_stat_EXP_VAR_FF0000L_YYYYYMMDD_VV0000L_*.stat EXP = 5 character experiment name VAR = variable name FF = forecast initialization time YYYYMMDD = valid year, month, and day VV = valid hour Currently supports grid comparisons between NCEP Stage IV precipitation and NAM-218 analysis (must download manually)

Scripts: runGridStatAnalysis.pl Green Stat Analysis circle under Statistics in workflow Uses the *.stat files output from Grid Stat to generate statistics comparing the model output to the gridded verification dataset (output in gridStatAnalysisOutput) YYYYMMDDFF_EXP_VAR_sfc_NBHOOD_POLY.txt YYYYMMDDFF = year, month, day, and forecast hour EXP = experiment name VAR = variable NBHOOD = number of surrounding grid points to consider in matching POLY = verification subdomain

namelist.met: Overview namelist.met text file is designed for easy modification to perform evaluations on only metrics of interest Each script begins with a series of calls to the readNamelist.pm module which scours namelist.met for only variables needed by each script and defines these variables as designated by the user Some variables can be a list and should be separated by a comma (with no spaces) Blocks separate like variables

Tutorial Objective: set up data files and namelist.met to verify 0-24h control and SPoRT forecasts output every three hours from 27 April 2011 Should have already downloaded tarballs containing updated MET scripts and tutorial data Untar MET file Create a directory called forecasts Move Forecast file to newly created directory; untar Move Stage IV file to gridData directory; untar Open namelist.met using your favorite text editor

namelist.met: &DirectoryInfo Block Gets scripts familiar with users directory structure and points script to data to process RunDir: full pathname of directory where the SPoRT MET script was untarred and will run NetCDFDir: full pathname of NetCDF directory for running ncdump to process MADIS data EMSBinDir: full pathname of directory in WRF-EMS software package containing wgrib, wgrib2, etc. utility binaries METDir: full pathname of highest level MET directory ForecastDir: full pathname of location of all GRIB forecast files to be processed

namelist.met: &Domain and &Time &Domain helps to reduce processing time by only extracting observations that are within the model domain Input the lower left and upper right latitude and longitude (enter the LL and UR coordinates of your local domain) &Time controls the number of forecast days are going to be evaluated Enter the first initialization date and time for the Start variables (enter 2011, 04, 27, and 00 for Start variables) Enter the final forecast date and time for the End variables (enter 2011, 04, 28, and 00 for End variables) NumberVerifyDays: number of consecutive case study days (1)

namelist.met: &ForecastInfo &ForecastInfo provides the needed information to tell the script how many hours each forecast is, how frequently a GRIB file of model output was generated, and the names of the two (or more) experiments being verified TotalForecastHours: total number of hours per forecast (24) ForecastVerifyInterval: how often were the forecasts output? Or with what frequency does the user want to verify (3) ExperimentNames: a list of names that match the five- character experiment appended to the beginning of the GRIBNAME variable in WRF-EMS (SPORT,CNTRL)

namelist.met: &MADISInfo &MADISInfo allows the user to configure which MADIS observations he/she wants to verify against Currently, ACARS profiles, profiler, RAOB, METAR, Mesonet, Maritime, and SAO are available Set the ProcessMADIS variable to true to do point verifcation; set to false if only doing grid verification (true) Set the UseVAR variables to true to use a dataset; set to false to exclude from verification (set all UseVAR variables to true) TimeRangeVAR variables tell MET to match up observations that fall within ±n minutes of the forecast valid time (Raob to 30; all others to 10) Useful for stations that do not always report exactly at the top of the hour when the forecasts are valid

namelist.met: &METInfo &METInfo allows for configuration of which parts of MET will run and allows for a user-defined verification domain (if a sub-domain of the overall domain is desired) RunPROGRAM: set to true to run each component of the MET package; set to false to not run selected components (all true) PressureLevels: upper air pressure levels (in hPa) to be verified by either Point Stat or Grid Stat (850,500) VerificationRegions: NCEP verification region on which to verify, USER for a user-defined domain, GRID for entire model grid (USER) UserVerifyCOORDINATES: lower left and upper right corners of user-defined verification grid (enter the LL and UR coordinates of your local domain)

namelist.met: &PointStatInfo &PointStatInfo provides the needed information to run Point Stat UseVerifySurfacePoint/UpperPoint: set to true to verify against surface observations and/or upper air observations respectively (true) Surface/UpperPointVerificationVariables: GRIB table variable name for variables on which to perform verification (Surface: TMP,DPT,PRMSL; Upper: TMP,DPT) VerticalObsWindow: Vertical pressure range (hPa) over which upper air observations will be accepted for forecast matching (10) StatsFilter: Easiest to just set this to MPR for now (MPR)

namelist.met: &GridStatInfo &GridStatInfo provides the needed information to run Grid Stat NeighborhoodBox: provides the width of the neighboorhood grids over which verification is performed (5) Must be an odd number If set to 1 will only do grid point to grid point matching PercentCoverage: determines percentage of a neighborhood grid has to contain the forecasted value for a hit to register (0.25) UseVerifyVAR: set to true to verify against precipitation and/or gridded analysis (Precipitation: true; Grids: false) GriddedPrecipitationVerificationAnalysis: set to STIV as NCEP Stage IV is the only precipitation analysis currently supported (STIV)

namelist.met: &GridStatInfo (contd) &GridStatInfo provides the needed information to run Grid Stat AccumulatedPrecipitationHours: accumulated precipitation totals to be verified for (must be greater than forecastInterval (03,12) PrecipitationThresholds: precipitation thresholds to use for binning skill scores (in mm) (1,5,10,25,50) GriddedVerificationModel: determines which large-scale NCEP analysis will be used for gridded verification of non- precipitation variables (none for tutorial) Surface/UpperGridVerificationVariables: GRIB table variable name of variable on which to verify (none for tutorial)

namelist.met: &PlottingInfo &PlottingInfo allows the user to automatically produce plots of the hour-by-hour forecast validation using the Open Source GD::Graph Perl module MakePlots: set to true to generate plots or set to false to make your own from the ASCII output Surface/UpperPlotVariables: GRIB table variable name for surface/upper-air variables for which to generate plots PressurePlotLevels: pressure levels (hPa) on which to generate plots NCEPPlotRegions: NCEP verification regions to generate plots PlotStatistics: statistics to plot PlotColors: color of line for each forecast (in same order as experiments were defined in ExperimentsNames variable under &ForecastInfo