Considerations in the Observation of Weather Sebastian Torres CIMMS/NSSL

2 Limitations in Radar Observations Errors of estimates Weather signal decorrelation Acquisition time Ambiguities Range ambiguities Velocity ambiguities Antenna rotation Antenna sidelobes Artifacts Hardware imperfections Quantization and saturation noise Amplitude and phase imbalances Phase jitter Strong point targets Ground clutter Biological scatterers Non-stationary signals Goal of Weather Signal Processing Maximize meteorologically relevant information that can be extracted from radar returns Spectral moments Polarimetric variables

3 Thresholding for Data Quality Need to flag data that may be corrupted or of bad quality Only use accurate parameters for further data processing Clarity of presentation of the meteorological variables is important to users Criteria for thresholding (a.k.a. data censoring) SNR Spectrum width CSR SQI (i.e., lag-1 correlation coefficient) Geometric criteria (e.g., storm tops, speckle) Statistical criteria (e.g., local continuity) There is a trade-off between data quality and obscuration Complex relations and usually not implemented in real-time Coherency

4 SNR Censoring in the NEXRAD network Range gates with non- significant powers are censored Non significant returns have a SNR below a user-defined threshold The system allows a different threshold for each moment 2 dB SNR threshold

5 Overlaid Echo Censoring in the NEXRAD network Range gates that have significant returns but an unrecoverable velocity are censored Velocity is unrecoverable due to overlaid echoes in the short PRT The strong trip can be recovered if the overlay is weak ( P s / P w > 10 dB) The weak can never be recovered

Part I Methods to Mitigate Range and Velocity Ambiguities

7 Maximum Unambiguous Range If targets are located beyond cT s /2, their echoes from the n- th transmitted pulse are received after the ( n +1)-st pulse is transmitted. Thus, they appear to be closer to the radar than they really are! This is known as range folding Maximum unambiguous range : r a = cT s /2 Echoes between r a and 2 r a are called 2 nd trip echoes, echoes between 2 r a and 3 r a are called 3 rd trip echoes, etc time True delay > T s ( n +1)th pulse n th pulse TsTs Apparent delay < T s

8 Maximum Unambiguous Velocity A pulsed Doppler radar measures radial Doppler velocity by keeping track of phase changes between samples that are T s (pulse repetition time) apart Recall that the phase shift is  =  4  r /. Then, the phase change from pulse to pulse is  =  4  r / =  4  v r T s / Note that only phase changes between –  and  can be unambiguously detected Maximum unambiguous velocity 4  v a T s / =   v a = /4 T s This is related to the Nyquist sampling theorem: Doppler velocities outside the ± v a interval will be aliased!

9 Range and Velocity Ambiguities on Pulsed Weather Radars Maximum unambiguous range r a = cT s /2 Maximum unambiguous Doppler velocity v a = /(4 T s ) The Doppler Dilemma: r a v a = c /8 Insufficient to observe severe convective storms at practical wavelengths NEXRAD specifications:  = 10 cm r a = 230 km This problem is worse for smaller wavelengths! v a ≈ 16 m s -1

10 What kind of velocities do we expect to measure? For the WSR-88D ( = 10 cm): …and we would have to deal with overlaid echoes! To measure these velocities unambiguously we need a Nyquist interval of at least ±50 m/s

11 Range Ambiguities T s = 3.1 ms and r a = 466 km T s = 780  s and r a = 117 km Overlaid echoes Ambiguous echoes

12 Velocity Ambiguities T s = 3.1 ms and v a = 8.9 m/s T s = ms and v a = m/s Velocity aliasing

13 Another PRT Trade-Off Correlation of pairs: This is sometimes called Signal Quality Index It’s a measure of signal coherency Accurate measurement of power requires long PRTs More independent samples (low coherency) Accurate measurement of velocity requires short PRTs High correlation between pairs (high coherency)

14 Signal Coherency How large a T s can we pick? Recall: Correlated pairs: Spectrum width much smaller than unambiguous velocity interval Increasing T s decreases correlation exponentially var( v ) and var(  v ) increase exponentially also! Pick a threshold: Violation of this condition results in very large errors of estimates! ^ ^

15 Signal Coherency and Ambiguities Range and velocity dilemma: r a v a = c Signal coherency:  v < v a /  r a constraint: This is a more basic constraint on radar parameters than the first equation above Then,  v and not v a imposes a basic limitation on Doppler weather radars Example: Severe storms have a median  v ~ 4 m/s and 10% of the time  v > 8 m/s. If we want accurate velocity estimates 90% of the time with an S-band radar ( = 10 cm); then, r a ≤ 150 km. This will likely result in range ambiguities 150 km 8 m/s

16 Range and Velocity Ambiguities Short PRTs are needed to maintain signal coherency and provide acceptable velocity aliasing Dilemma: r a v a = c /8 Given, we can pick v a to satisfy our needs. Then, r a is fixed, and it is usually so small that there can be 2 nd and even 3 rd trip overlaid echoes. Goal: Reduce obscuration from overlaid echoes (a.k.a. “purple haze”)

17 Legacy R/V Ambiguity Mitigation in the NEXRAD network Long PRTs are used to estimate powers (reflectivity) and short PRTs to estimate velocity Long-PRT powers are used to unfold short-PRT velocities Range unambiguous powers from the long PRT tell us where the echoes come from in the short PRT Overlaid echoes with comparable strengths cannot be resolved! 300 km long PRT 100 km short PRT range 70 km170 km270 km70 km 150 km50 km 2 nd trip 1 st trip overlaid echoes 170 km270 km 150 km Overlaid indication

18 Legacy R/V Ambiguity Mitigation in the NEXRAD network Split cut at low elevation angles Collect two scans at the same elevation angle (one using a long PRT and one using a short PRT) The long-PRT scan is used to retrieve unambiguous powers and the short-PRT scan to retrieve (range-folded) velocities Good ground clutter suppression but antenna scans twice at the same elevation Batch mode at intermediate elevation angles Collect one scan with interlaced batches of short and long PRTs The long-PRT scan is used to retrieve unambiguous powers and the short-PRT scan to retrieve (range-folded) velocities Reduced ground clutter suppression but antenna scans once at each elevation

19 Performance of Legacy R/V Ambiguity Mitigation in the NEXRAD network Velocity field is obscured by range-overlay censoring (“purple haze” syndrome) In case of overlaid echoes, only strong-trip velocities are recovered Strong-trip power must exceed weaker-trips powers by ~10 dB Velocities from weaker echoes cannot be recovered! Can we do better?

20 Other Techniques to Mitigate R/V Ambiguities Performance of R/V ambiguity mitigation techniques in the NEXRAD network is inadequate There are several signal processing techniques for extending the unambiguous range and velocity: Staggered PRT Dual PRF Phase coding Techniques based on physical modeling (e.g., continuity) are not truly signal processing techniques Wish list: Good ground clutter cancellation Same or improved moment estimate accuracy Same or improved acquisition time Comparable computational complexity

21 Staggered PRT Technique Transmitter alternates two PRTs Assume T 1 < T 2 PRT ratio:  = T 1 / T 2 = m / n ( m, n integers) r a 1 = cT 1 /2, r a 2 = cT 2 /2 v a 1 = /4 T 1, v a 2 = /4 T 2 Velocities can be estimated for each PRT Velocities v 1 and v 2 alias in different ways The true velocity can be obtained by using how v 1 and v 2 alias T1T1 T2T2 time T1T1 T2T2 … R1R1 R2R2 R1R1 R2R2 Can get unambiguous powers up to r a2 Can get unambiguous velocities up to mv a1 = nv a2 !

22 Velocity De-Aliasing Algorithm (I) Algorithm relies on how the short- and long-PRT velocity estimates alias Example ( = 10 cm) T 1 = 1 ms → v a 1 = 25 m/s T 2 = 1.5 ms → v a 2 = m/s True v (m/s) v 1 (m/s) v 2 (m/s)De-aliasing formulaDe-aliased v -500±16.67 v = v v a v = v v a v = v v = v v = v v a ±16.67 v = v v a simple rules

23 Velocity De-Aliasing Algorithm (II) v 1 and v 2 are computed from R 1 and R 2 A velocity difference transfer function determines the intervals for the different de-aliasing rules Beware: Estimates have errors True v v2v2 va2va2 v1v1 va1va1 v 1 – v 2 ^^ v 1 - v 2 closest level True velocity add 2v a1 to v 1 ^ ^ ^ ^ ^ Aliased v

24 Staggered PRT Performance v a = 25.4 m s -1 v a = 45.2 m s km 184 km KTLX VCP 11 – Batch Mode KOUN Staggered PRT (184 km/276 km) EL = 2.5 deg 04/06/03 4:42 GMT

25 Dual PRF Technique Transmitter alternates two “batches” of PRTs Batches of pulses are more conducive to better ground clutter filtering We can use same velocity dealiasing technique as with Staggered PRT Technique fails with strong shear (i.e., large velocity changes between adjacent batches) Tornado! T1T1 T2T2 time … T1T1 R1R1 R2R2 R1R1

26 Systematic Phase Coding Technique Overlaid echoes with no phase coding:, where K is the number of overlaid trips Transmitted pulses are phase-modulated with SZ(8/64) switching code Overlaid echoes with phase coding: TsTs  (0)  (1)  (2)  (3)  (4)

27 Systematic Phase Coding Technique (II) Cohering for the 1 st trip Two-trip case: Autocorrelation Modulation code Switching code What kind of R  would work? Received uncohered signal 1 st trip cohered signal Undesired term Desired 1 st trip autocorrelation Cohered 1 st trip signal Modulated 2 nd trip signal

28 The SZ(8/64) Code A contaminant signal with zero lag-one autocorrelation will not bias the pulse pair velocity estimator a We need a code with zero lag-one autocorrelation (Chu 1972) SZ(8/64) modulation code: SZ(8/64) switching code: The power spectrum of the SZ(8/64) code is a train of 8 deltas equally spaced in the Nyquist interval Modulated overlaid trips are evenly spread in the Nyquist interval f S(f)S(f) This product has to be zero S2(f)S2(f) S(f)S(f)

29 The Systematic Phase Coding Technique Transmitted pulses are phase-encoded with SZ(8/64) switching code Recovery of the weak-trip velocity is not always possible Phase-coded scan can be preceded by long-PRT scan Powers from the long-PRT scan are used to determine overlaid echoes in the phase-coded scan v Weak trip filtered Weak trip modulated Strong trip cohered vsvs Weak trip cohered Sidebands vwvw

30 Phase Coding Performance (I) v a = 28.1 m s -1, r a = 148 kmv a = 8.9 m s -1, r a = 466 km Reflectivity “Split cut” EL = 0.5 deg 03/03/04 20:28 GMT Legacy Velocity “Split cut”

31 Phase Coding Performance (II) v a = 28.1 m s -1, r a = 148 kmv a = 35.5 m s -1, r a = 117 km SZ-2 Velocity Short PRT EL = 0.5 deg 03/03/04 20:28 GMT Legacy Velocity “Split cut”

32 Mitigation Strategy 0.5° 1.5° 19.5° Regular long-PRT scan to retrieve powers up to 460 km and phase coded short-PRT scan to retrieve Doppler velocities up to 230 km (operational in 2007) Staggered PRT scan to retrieve powers up to 300 km and Doppler velocities up to 230 km Phase coding (SZ-2) 2 scans at each elevation angle Staggered PRT 1 scan at each elevation angle Uniform PRT (Legacy) 1 scan at each elevation angle 7.0°

33 Staggered PRT vs. Phase Coding Staggered/Dual PRT Non-uniform sampling Complex transmitter timing Spectral processing is difficult Use longer PRTs Good r a = cT 2 /2 Need velocity de-aliasing algorithm Good v a = /4( T 2 - T 1 ) Echoes are separated in time Clean recovery Difficult to transfer to operations Phase Coding Uniform sampling Existing transmitter sampling Spectral processing is simple Shorter PRTs Good v a = /4 T s Need separation of overlaid echoes Good r a (can recover two overlaid trips) Echoes are separated in frequency Messy recovery Easier to transfer to operations

Part II Methods to Suppress Artifacts

35 What is Clutter? Clutter refers to echoes that might interfere with desired signals Depending on the application, weather signals can be regarded as clutter! Types of clutter (for weather radars) Point targets Ground clutter Vegetation (seasonal!) Ground terrain Man-made structures Sea clutter Biological scatterers

36 What does clutter do? Clutter targets are usually very efficient reflectors of EM energy Clutter contaminates the signal of interest Biases reflectivity, Doppler velocity, and spectrum width This affects all downstream radar products! Clutter can saturate the radar receiver Ground clutter can be useful! It’s always there Radar calibration Azimuth Range Sensitivity Refractivity measurements (Courtesy of Boon Leng Cheong, OU)

37 Fighting Clutter (I) The Antenna Scatterers seen through antenna sidelobes bias reflectivity, velocity, and spectrum width estimates ground clutter birds airplane Antenna pattern of the WSR-88D without radome 29 dB

38 Effects of Antenna Sidelobes (Courtesy of Khoi Le, OU) Point target

39 Fighting Clutter (II) The Radar Site

40 The Radar Wavelength Power returned from Rayleigh scatterers goes inversely as the 4 th power of the radar wavelength Power returned from clutter targets (specular reflectors) have a lesser wavelength dependence Shorter wavelengths offer better clutter-to-signal ratios The Receiver Need large dynamic range Governed by A/D bits (and AGC) and system phase noise Powers returned from clutter targets can saturate the receiver Loss of sensitivity (from Billingsley 2002) Fighting Clutter (III)

41 Fighting Clutter (IV) The Antenna Rotation Rate Antenna motion combined with pulse-to- pulse processing creates an effective broadened beamwidth A faster antenna rotation rate results in a larger  c A phased-array radar has better clutter suppression The Signal Processor Stay tuned! Effective beamwidth for a scanning antenna as a function of the rotation rate This is not a problem for a phased array radar!

42 Clutter Suppression Clutter suppression can be done … On the time-series data (Level I data) On the moment data (Level II data) Suppression can be achieved by … Ignoring gates with clutter (censoring) Filtering Range time Sample time  Time domain  Frequency domain

43 What is a Filter? A filter is “a device” that alters the frequency spectrum of signals passing through it Continuous-time filters have discrete-time equivalents that can be implemented as a computer program or with programmable hardware processors Time-domain representation: y(n) = x(n)  h(n) Frequency-domain representation: Y(f) = X(f) H(f) Filter InputOutput x(n)h(n)y(n) X(f)H(f)Y(f) f X(f) f Y(f)

44 Strong Point Target Suppression Point targets bias all estimates! Point targets are easily identified in the Doppler spectrum How would you remove this artifact? Interpolation in the frequency domain Go to the frequency domain Remove point target Interpolate through the gap Go back to the time domain Plane flying toward the radar at 10 m/s

45 Strong Point Clutter Suppression in the legacy NEXRAD network Spectral processing is not available Search for discontinuities along range Range gate k has contamination from point clutter if P ( k )>8 P ( k  2) and P ( k )>8 P ( k+ 2) Remove and Interpolate (based on continuity!)  ( k  1) =  ( k  2)  ( k  1) =  ( k  2)  ( k ) = [  ( k  1) +  ( k+ 1)]/2 where  can be P, v, or  v range P 8x k k -2 k -1 k k +1 range gate Actual weather parameters are not recovered!

46 Ground Clutter Suppression Characteristics of most ground clutter returns Zero mean Doppler velocity Narrow spectrum width Non-zero  c due to: wind, antenna motion, window In the NEXRAD network  c ~ 0.3 m/s Ground clutter contamination can be suppressed with a high-pass filter Time domain FIR, IIR, regression, matrix, non-linear Frequency domain notch, notch & interpolation, spectral fitting Notch width determination Depends on the CSR and  c This does not apply to wind turbines or sea clutter!

47 Ground Clutter Filters v (m s -1 ) Clutter Return Weather Return Notch Width Unfiltered Doppler spectrum Filtered Doppler spectrum Notch filter frequency response (high-pass filter) Ground clutter residue Ground clutter will bias all estimates! Select the filter’s notch width to… Minimize ground clutter residue Maximize weather signal retention Trade-off!

48 To Filter or not to Filter… v (m s -1 ) Weather Return Notch Width Unfiltered Doppler spectrum Filtered Doppler spectrum Notch filter frequency response Should we apply the GCF everywhere? No! Missing weather spectrum will bias all estimates!

49 Filtering everywhere is not the solution! Ground clutter filters may introduce a bias Signals with narrow spectrum width and near-zero Doppler velocity are more vulnerable (why?) Stratiform precipitation case from a Great Plains region WSR-88D (Courtesy of Rich Ice, ROC) ReflectivityDoppler velocity Z bias Zero isodop

50 Can you guess? Filter was: not applied, applied in selected gates, applied everywhere? Reflectivity Doppler velocity

51 Ground Clutter Filter Maps Indicate where to apply the GCF and which GCF to use Note: GCF maps should be updated periodically This map is obsolete now

52 Examples of Ground Clutter Filters Time domain FIR: Pulse canceller IIR: Elliptic filter Regression Frequency domain SIGMET’s Gaussian Model Adaptive Processing (GMAP)

53 Time Domain GCF (FIR) Pulse Cancellers Two-pulse canceller Assumes ground clutter is DC y(n) = x(n) – x(n-1) |H(  )| = 2|sin(  /2)| Three-pulse canceller Cascade of 2 two-pulse cancellers |H(  )| = 4sin 2 (  /2) Characteristics Very simple to implement: No multiplications required! Short transient response Poor approximation to ideal high-pass filter Delay + - x(n)y(n)

54 Time Domain GCF (IIR) Elliptic Filter Elliptic filters generally meet filter requirements with the lowest order, compared to other classical designs. Design parameters: passband ripple ( R p ) stopband attenuation ( R s ) stopband edge velocity ( V p ) Characteristics More complex to implement Better approximation of ideal high-pass filter Filter’s transfer function

55 Time Domain GCF Regression Filter From Torres and Zrnic (JTECH, 1999) Regression filters approximate input signals with polynomial functions in the time domain We assume a slowly varying clutter signal (why?) that can be approximated with a polynomial of small degree Approximations are done using projections of the input signals onto a basis of orthonormal polynomials

56 Weather and Ground Signals

57 Weather Signal after Clutter Filter

58 Frequency Domain GCF Gaussian Model Adaptive Processing (GMAP) Filter GMAP is a proprietary filter developed by Sigmet GMAP assumes that clutter has a Gaussian Doppler spectrum and that  c is approximately known GMAP assumes that the weather signal has a Gaussian Doppler spectrum Steps: Window and DFT Noise power estimation Clutter power removal Iterative interpolation

59 Time Domain vs. Frequency Domain Time Domain Computationally simple No transformation is required Some FIR filters can be implemented without multiplications Extensive experience (+30 years) Artifacts are not apparent Ideal high-pass filter is not possible Compensation for removed signal is difficult Frequency Domain Computationally complex FFTs are required Window effects Relatively new! Easy to identify artifacts Ideal high-pass filter is possible Compensation for removed signal is easy

60 Ground Clutter Suppression Performance on Weather Data (No GCF) Squall line over KOUN June 11, 2003

61 Ground Clutter Suppression Performance on Weather Data (GMAP) Squall line over KOUN June 11, 2003

62 Ground Clutter Suppression Performance on Weather Data (No GCF) Squall line over KOUN June 11, 2003

63 Ground Clutter Suppression Performance on Weather Data (GMAP) Squall line over KOUN June 11, 2003

64 Anomalous Propagation Clutter In the presence of a strong inversion near the ground, the radar beam may be refracted such that it strikes the ground some distance from the radar. The resultant echo is from the ground (ground clutter) and is called anomalous propagation (AP) What time of the day would you more likely expect to see AP? Suppression of AP contamination is a challenge There’s no map!

65 AP at KOUN (Norman, OK) Sept 09, UTC Without GCF

66 AP at KOUN (Norman, OK) Sept 09, UTC With GCF everywhere

67 With canceller Without canceller (Courtesy of Cathy Kessinger, NCAR) AP Clutter Canceling in the NEXRAD Network

68 The Future of AP Clutter Canceling in the NEXRAD Network (Courtesy of Rich Ice, ROC)

69 Biological Scatterer Contamination Birds Wind (Courtesy of Svetlana Bachman, CIMMS)

70 Wind Turbine Clutter Contamination (Courtesy of Bradley Isom, OU)

71 Wind Turbine Clutter Cancelling Filtering Results Median Filter with Simulated Weather (Courtesy of Bradley Isom, OU)

72 Clutter Suppression Conclusions Clutter signals contaminate weather signals Clutter suppression is essential to obtain unbiased spectral moments Clutter contamination can be mitigated through… Hardware design and radar location Signal processing techniques Point clutter suppression Ground clutter suppression  Time domain and frequency domain filters  Non-linear filters, adaptive signal processing Clutter maps control how and where the GCF is applied AP clutter suppression is a challenge! Notch width trade-off