An overview of real-time radar data handling and compression Valliappa Lakshmanan National Severe Storms Laboratory & University of Oklahoma http://cimms.ou.edu/~lakshman/ 11/24/2018 lakshman@ou.edu

Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary 11/24/2018 lakshman@ou.edu

Operational radar network Weather Surveillance Radar 1988 Doppler (WSR-88D) 142 radars in CONUS 11 in Hawaii Alaska S-band Reflectivity to 460km Velocity to 230km 11/24/2018 lakshman@ou.edu

Operational radar data Each radar operates according in one of several volume coverage patterns NWS forecast offices choose VCP for “their” radars All VCPs involve several elevation angles Resolution (current) 0.95 degrees 1km (ref) 0.25 km (velocity) Tilt in 20-60s Volume in 4-10 min Super-resolution 0.25km x 0.5 deg Faster VCPs 11/24/2018 lakshman@ou.edu

Radar data: levels Level-I (or I&Q data) Signal processed at radar 64 values at each range gate Not transmitted/stored routinely Level-II (moment data) Reflectivity, Velocity, Spectrum Width 3 scalar values at each range gate (pixel) Distributed via LDM (more on this later) Level-III (products) Produced by algorithms in the Radar Product Generator Distributed via NOAA Port Somewhat obsolete, now that Level-II is available in real-time Better products can be created in real-time off-site 11/24/2018 lakshman@ou.edu

Size of radar data Focus on Level II Size of radar data depends on the VCP Clear-air VCPs have fewer (4-5) tilts collected more slowly 7.1 MB in 990 seconds for VCP 31 0.6 GB per day Bandwidth: 7.3 kB/s Storm-type VCPs have more tilts, therefore more data 15 MB in 460 seconds for VCP 12 2.8 GB per day Bandwidth: 33.4 kB/s A CONUS radar processing system needs to process 142 radars On a typical storm day, 1/3 of CONUS radars are in storm mode: 200 GB Mean: 60.6 GB/day when compressed Bandwidth: 736 kB/s (peak: 1771 kB/s) 11/24/2018 lakshman@ou.edu

Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary 11/24/2018 lakshman@ou.edu

Real-time Distribution 11/24/2018 lakshman@ou.edu

Distribution: components Radar Interface and Data Distribution System (RIDDS) Taps into radar stream at RPG Transmits over NWS network to regional server Regional servers on Abilene network Load dependent on geography Southern region average bandwidth: 484 kB/s Western region average bandwidth: 74 kB/s Real-time data may be requested via Local Directory Manager (LDM) Developed by Unidata Peer-to-peer realtime data distribution Top-tier providers: Oklahoma, Purdue, Maryland Transmission is in chunks of radials (all 3 moments) 50 radials at a time Compressed using bzip2 11/24/2018 lakshman@ou.edu

Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary 11/24/2018 lakshman@ou.edu

bzip2! Not further compressible gzip bzip2 Linear prediction, Custom compression methods can work better than bzip2 Example: V. Lakshmanan, “Lossless coding and compression of radar reflectivity data,” in 30th International Conference on Radar Meteorology, pp. 50–52, American Meteorological Society, (Munich), July 2001 Not further compressible gzip bzip2 Linear prediction, Run length encoding, Huffman coding 11/24/2018 lakshman@ou.edu

So why bzip2? Order of improvement of custom compression over bzip2: 10% Maintenance of compress/uncompress software has significant costs Every user of radar data would have to implement uncompress code Costs are spread out over many organizations: really adds up 10% decrease in bandwidth/space not worth the increased cost Would be different if this would bring bandwidth below some cost point Example: T1 line or dial-up? Choose among publicly available, non-proprietary compression formats gzip, bzip2, etc. Burrows-Wheeler (bzip2) performs best, so chosen 11/24/2018 lakshman@ou.edu

Preprocessing Some preprocessing done at radar site to improve compressibility Level I: signal-to-noise threshold Level II: height threshold Both done at radar site Part of radar system No cost on part of users Compression algorithm is still bzip2 Theoretically “lossy” compression But loss is on part of image not useful to end-users (JPEG?) 11/24/2018 lakshman@ou.edu

Compressibility But … Compression has uses beyond saving storage and bandwidth The ideas behind compressibility have wider use 11/24/2018 lakshman@ou.edu

Instrument artifacts Mostly good data Instrument artifact 11/24/2018 lakshman@ou.edu

Quality control of radar data QC needs to be done before data are fed to automated algorithms Done by means of machine intelligence algorithms Helps distinguish ground clutter, biological contamination etc. But … Hard to distinguish between stratiform rain and instrument artifacts Both fill the radar volume With reflectivity values around 30-50 dBZ Velocity and spectrum width do not help Texture-based quality control doesn’t help Reflectivity fields in both cases are locally smooth 11/24/2018 lakshman@ou.edu

Shannon entropy Theoretical maximum compressibility based on information content No need to actually compress file Simply compute Shannon’s information measure (entropy) The information content in the two fields (globally) is different Stratiform rain has high information content Even if most of the radar volume is not filled Instrument artifacts have low information content Even though most of the radar volume is filled Can easily detect changes in entropy Reject instrument artifacts even if it comes during a time period of rain 11/24/2018 lakshman@ou.edu

Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary 11/24/2018 lakshman@ou.edu

Summary Radar data is transmitted routinely in real-time to interested users Compressed in chunks of 50 radials each Peer-to-peer sharing (software: LDM) Compressed using bzip2 Custom compression techniques are not cost-effective Perform better than bzip2 But incremental (10%) improvement not worth the increased cost Compression ideas have other uses For example in quality control of radar data 11/24/2018 lakshman@ou.edu