1. Automated Extraction of Storm Characteristics
Valliappa Lakshmanan
National Severe Storms Laboratory & University of Oklahoma
11/20/2018

2. Automated Extraction of Storm Characteristics
Goal
WDSS-II
K-Means
Local Maximum
Summary
11/20/2018

3. Project 1: Skill Score By Storm Type
Try to answer this question (posed by Travis Smith)
Very critical, but hard to answer based on current knowledge
Is it the type of weather or is it the forecaster skill?
Initially, concentrate on tornadoes
Based on radar imagery, classify the type of storms at every time step
Take NWS warnings and ground truth information for a lot of cases
Compute skill scores by type of storm
Summer REU project
Eric Guillot, Lyndon State
Mentors: Travis Smith, Don Burgess, Greg Stumpf, V Lakshmanan
Does the skill score of a forecast office as evaluated by the NWS depend on the type of storms that the NWS office faced that year?
11/20/2018

4. Project 2: National Storm Events Database
Build a national storm events database
With high-resolution radar data combined from multiple radars
Derived products
Support spatiotemporal queries
Collaboration between NSSL, NCDC and OU (CAPS, CSA)
11/20/2018

5. Approach
Project 1: How to get classify lots and lots of radar imagery?
Need automated way to identify storm type
Technique:
Cluster radar fields
Extract storm characteristics for each cluster
Associate storm characteristics to human-identified storm type
Train learning technique (NN/decision tree) to do this automatically
Let it loose on entire dataset
Project 2: How to support spatiotemporal queries on radar data?
Can create polygons based on thresholding data
But need to tie together different data sources
Need automated way to extract storm characteristics for querying
11/20/2018

6. Automated Extraction of Storm Characteristics
Goal
WDSS-II
K-Means
Local Maximum
Summary
11/20/2018

7. Warning Decision Support System: Evolution
Single-radar
SCIT, MDA, TDA
Part of 88D Radar Product Gen.
Single-radar with multi-sensor input
NSE inputs
Part of Open RPG-8
2003
Multi-radar multi-sensor over regional domain
(1000km x 1000 km)
Gridded products
Shipped to select WFOs, SPC
2005
Multi-radar multi-sensor over CONUS
CONUS 1km grids
Available on the Internet
Used in SPC, licensed commercially
2007
Polarimetric, phased array radars; 0.25km x 0.5 degree resolution
11/20/2018

8. Methods Need way to extract storm characteristics in automated manner
WDSS-II has two techniques to do this
K-Means hierarchical segmentation (w2segmotionll)
Tracking of local maxima (w2localmax)
What about the input data?
WDSS-II provides a variety of CONUS grids and derived products
11/20/2018

9. WDSS-II CONUS Grids In real-time, combine data from 130+ WSR-88Ds
Reflectivity and azimuthal shear fields
Use these to derive products:
Reflectivity Composite
VIL
Echo top heights
Hail probability (POSH), Hail size estimates (MESH), etc.
Low-level, mid-level shear
Many others (90+)
Have the 3D reflectivity and shear products archived
Can use these to recreate derived products
11/20/2018

10. Hail Case (Apr. 19, 2003; Kansas)
Reflectivity Composite from KDDC, KICT, KVNX and KTWX
11/20/2018

11. Height of echo above 18 dBZ
Echo Top
Height of echo above 18 dBZ
11/20/2018

12. Maximum expected size of hail
MESH
Maximum expected size of hail
11/20/2018

13. Vertical Integrated Liquid
VIL
Vertical Integrated Liquid
11/20/2018

14. Automated Extraction of Storm Characteristics
Goal
WDSS-II
K-Means
Local Maximum
Summary
11/20/2018

15. Technique Identify storm cells based on reflectivity and its “texture”
Merge storm cells into larger scale entities
Estimate storm motion for each entity by comparing the entity with the previous image’s pixels
Interpolate spatially between the entities
Smooth motion estimates in time
Use motion vectors to make forecasts
Courtesy: Yang et. al (2006)
11/20/2018

16. Why it works
Hierarchical clustering sidesteps problems inherent in object-identification and optical-flow based methods
11/20/2018

17. Trends What about trends? Compute properties of current cluster
Min, max, mean, count, histogram, etc.
Project cluster backwards onto previous sets of images
Can use fields other than the field being tracked
Compute properties of projected cluster
Use to diagnose trends
11/20/2018

18. w2segmotionll Parameters
K-Means segmentation algorithm
Tracks Reflectivity Composite
In data range dBZ
Use VIL, MESH, EchoTop, SHI fields
Computes statistics specified in stormType.xml
Starts clustering at size=20
Minimum size increases 10x, so 2nd scale is 200 and 3rd scale is 2000
Zero indicates that smaller regions are pruned at coarser scales
Used default values for this slideshow, but 20:50:1 may suit better
w2segmotionll
-f "VIL MESH EchoTop_18 SHI“
-T MergedReflectivityQCComposite
-d "20 60"
-X ~/w2config/algs/stormTypeInput.xml
-p 20:10:0
11/20/2018

19. Identified Cluster IDs (Intermediate Scale)
Identified clusters on tracked image
11/20/2018

20. Identified Clusters (Intermediate Scale)
Identified clusters scale=1
11/20/2018

21. Identified Clusters (Detailed Scale)
Identified clusters scale=0
11/20/2018

22. Identified Clusters (Coarsest Scale)
Identified clusters scale=2
11/20/2018

23. Cluster Table One XML table per scale One row of table per cluster
Three XML tables per frame
One XML table per scale
One row of table per cluster
ID in the table keeps temporal continuity as much as possible
Each identified cluster has these properties:
ConvectiveArea in km^2
MaxEchoTop and LifetimeEchoTop
MESH and LifetimeMESH
MaxVIL, IncreaseInVIL and LifetimeMaxVIL
Centroid, LatRadius, LonRadius, Orientation of ellipse fitted to cluster
MotionEast, MotionSouth in m/s
Size in km^2
11/20/2018

24. Controlling the Cluster Table
Can choose any gridded field for output
From gridded field, can compute the following statistics within cluster
Minimum value, Maximum value
Average, Standard deviation
Area within interval (Useful to create histograms)
Increase in value temporally
Does not depend on cluster association being correct
Computed image-to-image
Lifetime maximum/minimum
Depends on cluster association being correct, so better on larger clusters
11/20/2018

25. Automated Extraction of Storm Characteristics
Goal
WDSS-II
K-Means
Local Maximum
Summary
11/20/2018

26. Technique Identify local maxima in the tracked field
This is a more generic version of traditional centroid tracking method
Identify local maxima in the tracked field
Find region of support for each local maximum
Associate regions between frames based on overlap and proximity to expected position
Can use region of support to calculate properties over time
11/20/2018

27. Local Maxima Tracking Data range to look for local maxima in
Parameters
Data range to look for local maxima in
Minimum size of a valid region
How many data levels are allowed in a peak
11/20/2018

28. w2localmax Parameters W2localmax -I MergedReflectivityQCComposite
Local maximum tracking algorithm
Tracks Reflectivity Composite
In data range dBZ
A region can comprise depth of up to 15 dBZ
As small as necessary
Minimum size of 100 km^2
Better parameters may be needed
W2localmax
-I MergedReflectivityQCComposite
-d " “
-D 3
-S 100 -s
11/20/2018

29. Identified Local Maxima
Identified maxima on tracked image
11/20/2018

30. Identified Local Maxima
Identified maxima with regions of support
11/20/2018

31. Tracking Output Average Maximum Minimum
Currently no capability to track other fields and compute their properties
Only properties of tracked field are reported
Average
Maximum
Minimum
Increase in Average, Max and Min
Ellipse fit parameters (centroid, radii, orientation)
11/20/2018

32. Automated Extraction of Storm Characteristics
Goal
WDSS-II
K-Means
Local Maximum
Summary
11/20/2018

33. KMeans vs. LocalMax Advantages of LocalMax
Cluster ID is more robust across time
Easy to understand rules on what a storm id is
Disadvantages of LocalMax
Not hierarchical although it can be made multi-scale
No tracking of other fields (can be added)
Advantages of K-Means
Hierarchical
Hierarchical is more than just multi-scale
Cluster of detailed scale is inside cluster of coarser scale (“contained”)
Motion estimates are very robust
Time delta properties not dependent on time association of clusters
Disadvantages of K-Means
Cost-function minimization to identify clusters makes it harder to understand
Cluster identification not robust across time
Small changes in image can cause large changes in cluster result
11/20/2018

34. Way Forward Determine ideal parameters for each algorithm
Determine storm characteristics that need to be collected
Choose algorithm to use for study
Is true hierarchical clustering needed or is multi-scale enough?
Enhance algorithm if needed to support desired features
Choose data cases
Create storm type truth for row of table for each data case
Train learning system (NN/decision tree) on truthed data
Create data cases for entire year
Run trained system on rest of data
Place output of training set into GIS system
Compute skill score on data set by storm type
11/20/2018