Evolution of the SZ-2 Algorithm Sebastian Torres CIMMS/NSSL Technical Interchange Meeting Fall 2006

Recent Evolution of SZ-2 Several “areas of interest” identified and ranked by the ROC –Solutions to critical AOIs were discussed at previous TIM Assessment of engineering fixes –After-TIM discussions Provided draft AEL Analyzed ROC’s implementation Last AEL delivered on June 9, 2006 –Real-time implementation validated with off-line MATLAB simulator –SZ-2 implemented following latest AEL Many cases presented to the DQ team –Problem with noisy velocities due to bug in the code TAC approved SZ-2 for Build 9 of ORDA

06/09/06 AEL Revisions Algorithm description fits real-time implementation –dB-for-dB censoring –Strong-point clutter suppression –Required outputs (T 0, R 0, R 1, R 2 ) Addition of overall logic flow Efficient processing of non-overlaid echoes Dynamic use of data windows Spectrum width computations –Unbiased autocorrelation for data windows –Adaptable spectrum width estimator Censoring –dB-for-dB –Rules and classification Strong-point clutter suppression

Dynamic Use of Data Windows SZ-2 uses three data windows depending on the situation –The PNF needs the von Hann (or more aggressive) window –GMAP needs the Blackman window to achieve required clutter suppression Dynamic data windowing rules –Use the rectangular window with non-overlaid, non-clutter- contaminated echoes Implemented algorithm uses the “default” window (currently the Hamming window) –Use the von Hann window with overlaid, non-clutter- contaminated echoes –Use the Blackman window with clutter-contaminated echoes

Spectrum Width Computations Rules for choosing a spectrum width estimator –Use the R 0 /R 1 estimator with non-overlaid echoes –Use the R 1 /R 2 estimator with overlaid echoes Unbiased spectrum width estimates require unbiased autocorrelation estimates –The data window must be accounted for in the autocorrelation estimator Correct factor for arbitrary data windows Divide by the number of pairs?

Censoring in SZ-2 What is “black” and what is “purple”? –Aimed at clear classification rules “Black” corresponds to non-significant returns “Purple” corresponds to gates that have significant returns but cannot be recovered –Need to maintain accepted system behavior For example, dB-for-dB censoring is tagged as “black” Gates are classified as having one of the following three types of returns –Signal return: above adjusted SNR threshold and recoverable (passes all tests) –Noise return: below SNR threshold or strong clutter in non-overlaid case –Overlaid return: Unrecoverable with two or more overlaid trips (at least fails one of the tests)

Censoring Rules RuleThresholdClassNotes SNR long PRTK SNR,V NOISE SNR short PRTK SNR,V NOISE CNR short PRTadjusted K SNR,V NOISE Non-overlaid echoes OVERLAID Overlaid Echoes CSR long PRTK CSR1 NOISE Non-overlaid echoes OVERLAID Overlaid Echoes SNR*KSKS OVERLAID Strong Trip Censoring Rules

Censoring Rules RuleThresholdClassNotes SNR long PRTK SNR,V NOISE SNR short PRTK SNR,V NOISE CNR short PRTadjusted K SNR OVERLAID CSR long PRTK CSR2 OVERLAID SNR*KWKW OVERLAID Recovery regionK r =f(w S,w w,C T,C S,C I )OVERLAID Clutter locationOVERLAID Width long PRTw max OVERLAID Censoring applies to spectrum width only Weak Trip Censoring Rules

Censoring Rules RuleThresholdClassNotes SNR long PRTK SNR,V NOISE Non-significant return OVERLAID Significant return Other Trip* Censoring Rules * This censoring applies to the two weakest trips

Data Windowing Issues Sebastian Torres CIMMS/NSSL Technical Interchange Meeting Fall 2006

The Effect of Data Windows Data windows reduce the equivalent number of independent samples available to estimate spectral moments –Non-rectangular windows are tapered so end samples “contribute less” to the estimation process –The more aggressive the data window, the larger the errors of estimates for all spectral moments Data windows need to be accounted for in the autocorrelation estimator –Normalization by “lag window” (D&Z, ch5)

Standard Error of Moment Estimates for Different Windows vs. SNR Parameters of VCP 211 M = 64, PRT = 780  s (PRI #8), f = 2800 MHz,  v = 4 m/s

Relative Standard Error of Moment Estimates for Different Windows vs. SNR Parameters of VCP 211 M = 64, PRT = 780  s (PRI #8), f = 2800 MHz,  v = 4 m/s

Data Windows in SZ-2 Default window with non-overlaid, non-clutter-contaminated echoes –AEL recommends using the rectangular window –ORDA uses the Hamming window as the default window von Hann with overlaid, non-clutter-contaminated echoes Blackman with clutter-contaminated echoes RectangularHammingvon HannBlackman SD(Z) (dB) SD(v) (m/s) SD(  v ) (m/s) Parameters of VCP 211 M = 64, PRT = 780  s (PRI #8), f = 2800 MHz True  v = 4 m/s, SNR for Z &  v = 10 dB, SNR for v = 8 dB

SZ-2 Velocity Hamming Window KCRI (ORDA) VCP /19/06

SZ-2 Velocity Rectangular Window KCRI (ORDA) VCP /19/06

SZ-2 and Super Resolution (NPI) Sebastian Torres CIMMS/NSSL Technical Interchange Meeting Fall 2006

What is Super Resolution? Legacy Resolution spatial sampling –Reflectivity: 1-km by 1-deg grid –Doppler: 250-m by 1-deg grid Super Resolution spatial sampling –All moments: 250-m by 0.5-deg grid Finer spatial sampling and smaller resolution volume lead to about 50% improvement in range of detection for mesocyclone and tornado signatures (Brown et al. 2002) Tornado outbreak in Oklahoma City 9 May 2003 (Curtis et al. 2003) Legacy ResolutionSuper Resolution Two elements of super-resolution data

Super Resolution for NEXRAD Super-resolution data scheduled for operational use on NEXRAD –Short-term goals - Phase I: ORDA Build 10 (FY 2008) Data used for visualization only Legacy- and super-resolution data available in the RPG –RPG algorithms ingest legacy-resolution data –Long-term goals - Phase II: ORDA Build 13? (FY 2012) Data used by the algorithms Super-resolution data produced on lower-elevation scans (split cuts) –Higher likelihood of finding tornado and meso signatures –SZ-2 may also run on these scans Elevation Azimuth RDARPG w v Z Products Super resolution base data RDA

Super Resolution on the ORDA (I) RDA must produce base data with finer spatial sampling and resolution –Finer spatial sampling grid Radials collected at 0.5 deg azimuth intervals –Legacy-resolution radials collected at: 0.5, 1.5, 2.5, … deg –Super-resolution radials collected at: 0.25, 0.75, 1.25, … deg »Recombined radials created at: 0.5, 1.5, 2.5, … deg No range averaging to maintain 250 m sampling in range –Finer resolution Selective data windowing 1 km 1 deg 0.5 deg 250 m Legacy-resolution reflectivity grid Super-resolution reflectivity grid

Super Resolution on the ORDA (II) Solution: Overlapping 1-deg radials with data windowing sampled every 0.5 deg and no range averaging –For each range gate, M time-series data samples are weighted … with von Hann window if clutter filtering is not needed … with Blackman window if clutter filtering is needed 0 deg 0.5 deg 1.0 deg 1.5 deg 2.0 deg bypass filterbypass GCF Map 1 deg Pulse # M = 64

Super Resolution on the ORDA (III) Doppler-derived reflectivities and noise power are needed in the ORPG to produce legacy-like data –Reflectivity is range unfolded together with the Doppler moments –Noise power is added to metadata Doppler moments produced up to 300 km –Data beyond 230 km is not discarded Throughput is doubled –Twice the number of radials in an elevation cut Computational complexity is doubled –Super-resolution radials have the same number of samples as legacy-resolution radials

Super Resolution on the ORPG (I) ORPG algorithms expect data with legacy resolution and quality –Super-resolution data does not have the required resolution or quality for the algorithms (it’s OK for visualization) Super resolution radial recombination –Two super-resolution radials are recombined to form one legacy-resolution radial –Recombination assumes a bimodal spectrum –Approach exhibits low risk and provides data with acceptable quality –Algorithm must deal with missing data SNR thresholds Overlaid echoes Acceptable data quality v Doppler velocity v1v1 v2v2 Radial 1 PSD Radial 2 PSD

Super Resolution on the ORPG (II) Velocity dealiasing algorithm must run on super-resolution and recombined legacy-resolution data –Both velocity fields will available for visualization –Legacy-resolution fields will be fed to the algorithms –Increased processing requirements

SZ-2 Changes to run in Super Resolution mode ORDA –Finer spatial sampling grid: no changes –Finer resolution: no changes Default window for super resolution will be set to von Hann –Range unfolding of Doppler reflectivities: no changes Unfolded Doppler powers already exist within SZ-2 –Addition of noise power to metadata: no changes –Doppler data up to 300 km: no changes Data already exist within SZ-2 –Doubled throughput: no changes –Doubled CPU load: needs testing Current maximum CPU usage with SZ-2 is ~35% Expect maximum CPU usage of ~70% with super-resolution SZ-2 ORPG –SZ-2 is already transparent to the ORPG: no changes

Reflectivity – FFT Legacy resolution

Velocity – FFT Legacy resolution

Reflectivity – SZ-2 Legacy resolution

Reflectivity – FFT Legacy resolution

Velocity – SZ-2 Legacy resolution

Velocity – FFT Legacy resolution

Reflectivity – SZ-2 Super resolution

Reflectivity – SZ-2 Legacy resolution (Recombined)

Reflectivity – SZ-2 Legacy resolution

Velocity – SZ-2 Super resolution

Velocity – SZ-2 Legacy resolution (Recombined)

Velocity – SZ-2 Legacy resolution

“All-Bins” Clutter Filtering and SZ-2 Sebastian Torres CIMMS/NSSL Technical Interchange Meeting Fall 2006

“All-Bins” Clutter Filtering and SZ-2 SZ-2 cannot recover overlaid signals if multiple trips have clutter contamination (overlaid clutter) Operator-selected “all-bins” clutter filtering forces overlaid clutter in every bin –A large percentage of the bins will not have clutter contamination –Is there a simple way to detect which bins have clutter contamination? Answer: GMAP

“All-Bins” Clutter Filtering and SZ-2 GMAP is used to “detect” clutter –The clutter power removed by GMAP during the long PRT is used as an indicator of the presence of clutter –Recommended SZ-2 algorithm uses the long-PRT CSR –Preliminary tests show that the long-PRT CNR may be a better indicator We rely on only one estimate from GMAP

Reflectivity – All bins KCRI (ORDA) VCP /19/06

Reflectivity – Bypass Map KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CSR > 15 dB KCRI (ORDA) VCP /19/06

Velocity – Bypass Map KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CSR > 10 dB KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CSR > 15 dB KCRI (ORDA) VCP /19/06

Censoring – All bins GCF Clutter if CSR > 10 dB KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CNR > 10 dB KCRI (ORDA) VCP /19/06

Censoring – All bins GCF Clutter if CNR > 10 dB KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CNR > 20 dB KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CNR > 30 dB KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CNR > 40 dB KCRI (ORDA) VCP /19/06

Velocity – Bypass Map KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CSR > 15 dB KCRI (ORDA) VCP /19/06

Velocity – All bins GCF Clutter if CNR > 40 dB KCRI (ORDA) VCP /19/06

Thank you!