WDSS-II Training Module IV Algorithms and Tools

General Notes Output from WDSS-II applications may be shared across multiple machines Any application can use the output of another application as input The wg display is an example of this It provides input/launch to the “Filter” algorithms It uses products from other algorithms Real-time and “data playback” modes are essentially the same modes of operation

WDSS-II application types Data ingest applications (“ingestors”) Single-source algorithms Usually single-radar applications Multi-source algorithms Combine input data from multiple sources of one or more instrument types General use tools Data filters, objective analysis tools, data remapping, data converters, verification tools, etc.

WDSS-II primary data types LatLonGrid: geographic projection Equal spacing in degrees latitude and longitude RadialSet: cylindrical projection Accommodates any number of radials with variable radial widths PolarGrid: an indexed RadialSet DataTable: for point data Trends tracks CartesianGrid: equidistant projection equal spacing in N/S/E/W directions Other types to be described in a later presentation

Data ingest Data-ingesting programs read “raw” data files and convert them to one of the internal WDSS-II formats New input types are easy to add Maintains a consistent internal structure for data sharing among applications

WDSS-II Real-time data flow ldm2netcdf Reflectivity Velocity Sp. W. w2qcnn ReflectivityQC swatScit2D Scit2D (table) w2circ AzShear Divergence AzShear layers Single-radar products WSR-88D data (level 2) netssap CellTable MesoTable TvsTable RUC analysis data (grib)satellite data* gribToNetcdf nse 1 w2cloudcover *Satellite data are required to be in netcdf format. w2vil Reflectivity OR ReflectivityQC w2hail MESH POSH MESHTracking Echo Tops (H_*) VIL Comp. Ref. Other optional algorithms Dashed lines represent optional inputs, data sources, or applications Applications are in boxes Data sources are in ovals Legend 1 If nse is not used as an input, then PolarHail.xml and ssaparm.dat should be updated twice daily. It is highly recommended to use nse data if accurate hail guidance are desired.

The most-used single-source algorithms w2qcnn: quality control neural network May use radar-only data, or radar plus cloud cover information Output: ReflectivityQC & ReflectivityQComposite w2circ: radial velocity derivatives Produces rotational (AzShear) and divergent (Divergence) shear fields for every tilt Also produces layer maxima (e.g. 0-3 km MSL)

The most-used single-source algorithms nse: near-storm environment Parameters are derived from the RUC model analysis Provides input to other algorithms Output similar to SPC mesoanalysis web page

Other single-source algorithms w2hail: hail grids and echo tops w2vil: VIL and composite reflectivity netssap: the original SSAP MDA, TDA, SCIT, HDA, DDPDA Requires copy of *.dat configuration files in working directory dealias: independent executable of WSR-88D build 10 dealiasing Note that dealiasing is usually done automatically in data ingest process for WSR-88D data (ldm2netcdf)

WDSS-II Real-time data flow scit3D Reflectivity[QC] (x N radars) Scit2D (x N radars; or from w2merger) w2merger ClusterTable MergedReflectivity[QC]Composite Forecast (15,30,45,60 min) Windfield Multi-radar products w2segmotion MergedReflectivity[QC] MergedReflectivity[QC]Composite VIL products Reflectivity_X 1 C EchoTop_Y 2 HY 2 _Above_HX 1 (“Height Above Isosurface”) MESH /POSH / SHI (Hail) Scit2D (from 3D grids) 1 isosurface(C); 2 reflectivity value (dBZ) nse * qcinfo Dashed lines represent optional inputs, data sources, or applications Applications are in boxes Data sources are in ovals Legend MR_Celltable QCTimeInfo w2accumulator MESH Tracks (2 hr, 6 hr, etc) *If nse is not used as an input, then MRScitHail.xml should be updated twice daily. It is highly recommended to use nse data if accurate hail guidance are desired. AzShear[layer] w2merger MergedAzShe ar[layer] (RotationTracks)

w2merger Multi-radar data merging 2D or 3D Continuously updating The grid is updated each time data are received from any source Writes output at user-specified time intervals Any resolution (Vertical/horizontal) Also runs algorithms on the 3D data field ger.pdf ger.pdf

w2merger preparations: cache Pre-compute the radars that will sample the grid point (the “cache”) Makes all computations faster Beam blockage is considered Use program “createCache” (once for each radar) w2merger will create a cache on-the-fly if one is not available, but: It will not include terrain data Data will not be processed until the cache creation is complete (which might take a while)

w2merger preparations: cache By default, the cache is stored in ~/.w2mergercache It might be big! If you are finished processing a domain, you should delete it A cache may be extracted from a cache with larger spatial extents (“createCache –e”) Within NSSL: extract from /mnt/radararchive Another option: createSubdomains – create caches for all radars in the domain

w2merger preparations: cache You may reduce the number of radars that affect a point by running “postprocessCache” e.g. if you only want the 3 “best” radars to impact the calculation at a point

Merging strategies Different products may require different ways of combination Set through the ‘-C’ option Some examples: Reflectivity: ExponentialTimeAndDistance or Distance AzShear: MagnitudeMaximum Velocity: InverseVAD or MultiDoppler Choose the most appropriate method for the product you are merging. There are others: see w2merger usage for list If you need a different merging option, add it!

Running merging and algorithms separately Algorithms may be run each time w2merger writes out 3D grids of reflectivity data If the merger is CPU-intensive or I/O- intensive, then run the algorithms separately, perhaps on another machine w2merger option “-C 10”

w2merger algorithms (-a option) Composite or VerticalMaximum vertical maximum at each lat/lon VerticalMinimum vertical minimum product at each lat/lon AbsMax or AbsoluteMaximum abs-max product at each lat/lon. The result retains the sign of the maximum. VIL vertical integrated liquid product at each lat/lon (assumes that the 3D grid is a grid of Reflectivity) Includes different integration strategies (e.g. along storm tilt, VIL Density, etc)

w2merger algorithms (-a option) HDA produces SHI, POSH, and MESH at each lat/lon (assumes that the 3D grid is a grid of Reflectivity). SCIT creates 2D storm cell features from the multi- radar grid (assumes a grid of Reflectivity). LayerAverage or Isotherms produces Reflectivity at various isotherms (0,-10 and -20C), ReflectivityBelowZero, LowestReflectivity, etc.

w2merger supplemental output MergerInputRadarsTable Provides information about the current data streams Age Tile VCP Useful for determining which radars went into the output

w2segmotion: storm segmentation and motion estimation Multiple scales Can generate statistics based on storm areas Motion estimates feed back into w2merger for time/space correction /kmeans_motion.pdf /kmeans_motion.pdf

Mr. SCIT (Multi-radar storm cell identification and tracking “scit3D” executable Use “-g” option for Scit2D features generated by w2merger Use “-t” option to ingest grid fields of various parameters that should be added to the output table Environmental data from RUC analysis Precipitation rate field Etc. Produces “MR_CellTable” output

w2accumulator Take the: Maximum Minimum, or Sum of all tables or grids produced over a specified time interval. E.g.: 2-hour max MESH = a hail swath 6-hour precipitation rate integration 4-hour max of 0-3 km Azimuthal Shear (“Rotation Tracks”) DataTable, RadialSet, or LatLonGrid

Other useful algorithms w2cloudcover: estimate cloud cover over a region using IR satellite and surface temperature w2vortdiv: compute vorticity and divergence from a 2D wind field w2alarm: collect statistics within an earth-relative polygon for any grid

Data Converters w2awipsnc: convert WDSSII netcdf grid files to AWIPS format w2cropconv: convert and remap any WDSSII RadialSet or LatLonGrid to a LatLonGrid w2csv2table: convert a CSV file (spreadsheet) to a WDSS-II DataTable w2table2csv: vice versa

Data Converters w2geotiff: convert a WDSSII netcdf file to a geoTIFF file A TIFF image file with geographic information tags (for GIS) w2grib2conv: convert a WDSS-II file to GRIB2 netcdf2ldm: convert a set of WDSSII netcdf files to WSR-88D level II format Can replace AliasedVelocity with Velocity, Reflectivity with ReflectivityQC for example

Objective analysis / filters w2smooth: smooth the data using one of many strategies: Gauss Cressman Percent (e.g. median) Oriented Ellipse Various wavelets

Objective analysis / filters w2threshold: Thresholds one field based on another Example, remove VIL in areas where the IR temperature is > 250K Various options to smooth (using w2smooth internally) and/or segment field

Objective analysis / filters w2oban: convert point data to a LatLonGrid w2morph: morphological filters Dilate Erode w2contour: create contours of a data field

File manipulation w2get: copy a file via rssd w2mirror: mirror all the files listed in an lb to a different machine Limits the number of users “hitting” a real-time machine w2simulator: simulate real-time data playback w2stitcher: stitch together two different domains into one larger one

Suggested exercise on archive data Download KTLX and KINX data from May 20, 2001 from 21:00 to 22:00 UTC from NCDC Convert it into WDSS-II netcdf format Run w2vil to produce VIL estimates in rapid-update mode Merge the VIL estimates using w2merger What is a valid combination strategy here? createCache before merging! Compute VIL from merging reflectivity data Compare the two VIL estimates Find their difference field using w2scoregrid

Suggested exercise on real- time data Connect to two adjacent radars that are currently experiencing weather Look at the 2DConUS index Overlay the radarsites shapefile Find LB names from the tensor list Create cache for domain using createSubdomains. Extract from /mnt/radararchive Run w2vil, w2merger and w2scoregrid as described before. Set up a w2alg.conf to do this.

End of WDSS-II Training Module IV What to do next : Practice running some algorithms and tools. You will not be able to follow module 6 (writing a WDSS-II algorithm) unless you are familiar with how WDSS-II algorithms in general work. Run both a single-radar algorithm and a multi- sensor algorithm. Run both on archived cases and real-time cases. Next module: Configuring WDSS-II