1. Sebastian Torres NEXRAD Range-Velocity Ambiguity Mitigation Spring 2004 – Technical Interchange Meeting

2. Part One Update on Previous Work

3. 148 km184 km KTLX VCP 11 – Batch Mode KOUN Staggered PRT (184 km/276 km) EL = 2.5 deg 04/06/03 4:42 GMT v a = 25.4 m s -1 v a = 45.2 m s -1 The Staggered PRT Algorithm Images with artifacts

4. v a = 25.4 m s -1 v a = 45.2 m s -1 KTLX VCP 11 – Batch Mode KOUN Staggered PRT (184 km/276 km) EL = 2.5 deg 04/06/03 4:42 GMT The Staggered PRT Algorithm Correct Images

5. Time-Domain Filter 5-Pole Elliptic Filter Frequency-Domain Filter Fixed Notch Filter EL = 1.5 deg 02/14/03 1:06 GMT The SZ-2 Algorithm Ground Clutter Filter Effects

6. Part Two The GMAP Ground Clutter Filter

7. How does GMAP work? Inputs –Power spectrum from time-series data –Apply window –Compute DFT –Compute magnitude squared –Noise level (optional) –Ground clutter spectrum width Outputs –Filtered power spectrum –Removed power

8. How does GMAP work? Noise power computation using rank order technique –Sort power spectrum –Compare with theoretical curve for white noise –Find component at which power spectrum departs from white noise –Identify noise components –Compute noise power

9. How does GMAP work? Clutter filtering –Generate clutter model based on  c, T s, M, and window –Determine notch width (gap) from clutter model and noise level –Notch clutter components

10. How does GMAP work? Spectrum reconstruction –Compute v and  v from signal components –Fill gap with signal model Re-compute v and  v –Repeat until v and  v converge Compute removed power –Subtract reconstructed power from original power in the gap

11. GMAP Analysis Tool MATLAB port from RVP8 RDA 8.04 (4 Nov 2003) –No significant changes to GMAP in subsequent releases Current release: RVP8 RDA 8.05.2 (29 Mar 2004) –Ported function fSpecFilterGMAP() and all supporting functions Tool is useful for qualitative analysis

12. GMAP Analysis Tool

13. GMAP Performance GMAP noise estimation –Inadequate for small number of samples GMAP interpolation –Restores weather signal power in the gap GMAP in the absence of clutter –Some components are filtered nevertheless –Spectrum can become distorted

14. Estimating Clutter Power from GMAP Simulation: –Input: Signal + Clutter Signal velocity is random in (-v a, v a ) Signal spectrum width is 1, 2, 4, and 8 m/s CSR is varied from -30 to 50 dB CNR > 20 dB for all cases –Process: GMAP –Output: Removed power P REM Goal: Establish suitability of P REM as an estimate of clutter power

15. Estimating Clutter Power from GMAP Removed power from GMAP is unreliable as an estimate of clutter power, especially for low CSR CSR(dB) P REM = 0 if clutter filtering “adds” power

16. Suitability of GMAP Algorithm is not available in the literature –Details are proprietary –Uses several “empirical constants” GMAP works better for large M –Modifications made by SIGMET to handle small M Ice et al. (2004) reported compliance with NEXRAD requirements using “black-box” analysis –Recommended using Blackman window and providing noise level to GMAP Good candidate for SZ: no phase distortion –Minor changes are required (stay tuned!)

17. Part Three GMAP and Phase Coding

18. GMAP designed for uniform sampling, non-phase coded signals Issues: –Window effect –Noise estimation –Spectral reconstruction –Filtered time series

19. Window Effect Window choice –Sachidanda et al. (NSSL Report 2, 1998) recommend Von Hann window to minimize errors –Ice et al. (ORDA Report, 2004) recommend Blackman window to achieve larger clutter suppression Blackman is more aggressive than Von Hann –Should expect larger errors of estimates

20. Window Effect Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip 1 st Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width varies from 0.5 to 8 m/s 2 nd Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width is 1, 2, 4, and 8 m/s S 1 /S 2 varies from 0 to 70 dB –Process: SZ-2 with GMAP Case 1: Blackman window Case 2: Von Hann window –Output: Statistics of v 1 and v 2 estimates

21. Window Effect Blackman WindowVon Hann Window

22. Noise Estimation Issue: Use GMAP noise estimation or provide noise to it? Out-of-trip echo looks like white noise –GMAP noise estimation fails –GMAP notch width based on over-estimated noise level is narrower than required

23. Noise Estimation Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip + Clutter in the 1 st trip 1 st Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width varies from 0.5 to 8 m/s 2 nd Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width is 1, 2, 4, and 8 m/s S 1 /S 2 varies from 0 to 70 dB C/S 1 varies from -30 to 50 dB –Process: SZ-2 with GMAP Case 1: GMAP with noise estimation Case 2: GMAP with provided noise –Output: Statistics of v 2 estimates

24. Noise Estimation GMAP with noise estimationGMAP with provided noise

25. Spectrum Reconstruction Issue: Use GMAP interpolation or apply notch filter? Interpolation helps reducing biases (in strong signal moments) due to clutter filtering Interpolation assumes coherent weather signal is present after clutter filtering Two cases to consider: –Clutter with the strong signal –Clutter with the weak signal

26. Spectrum Reconstruction Clutter with the strong signal v

27. Spectrum Reconstruction Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip + Clutter in the 1 st trip 1 st Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width varies from 0.5 to 8 m/s 2 nd Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width is 1, 2, 4, and 8 m/s S 1 /S 2 varies from 0 to 70 dB C/S 1 varies from -30 to 50 dB –Process: SZ-2 with GMAP Case 1: GMAP with interpolation Case 2: GMAP without interpolation –Output: Statistics of v 1 estimates

28. Spectrum Reconstruction GMAP with interpolationGMAP without interpolation

29. Spectrum Reconstruction Clutter with the weak signal v

30. Spectrum Reconstruction Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip + Clutter in the 2 nd trip 1 st Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width varies from 0.5 to 8 m/s 2 nd Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width is 1, 2, 4, and 8 m/s S 1 /S 2 varies from 0 to 70 dB C/S 1 varies from -30 to 50 dB –Process: SZ-2 with GMAP Case 1: GMAP with interpolation Case 2: GMAP without interpolation –Output: Statistics of v 1 estimates

31. Spectrum Reconstruction GMAP with interpolationGMAP without interpolation

32. Filtered Time Series GMAP returns power spectrum Phases must be saved to reconstruct full spectrum and return to time domain Issue: What are the phases of the reconstructed components? –Original phases –Zero phases –Something else?

33. Filtered Time Series Simulation: –Input: Clutter in the 1 st trip + Signal in the 2 nd trip 2 nd Trip Signal: –Velocity is random in (-v a, v a ) –Spectrum width varies from 0.5 to 8 m/s C/S 2 varies from 0 to 70 dB –Process: SZ-2 with GMAP Case 1: Reconstruction with original phases Case 2: Reconstruction with random phases Case 3: Reconstruction with zero phases –Output: Statistics of v 2 estimates

34. Filtered Time Series Original PhasesRandom PhasesZero Phases Weather is in the 2 nd trip and Clutter is in the 1 st trip

35. GMAP and Phase Coding Summary Use Blackman window for required suppression at the expense of loss of accuracy Provide noise level to GMAP Reconstruct filtered spectrum using zero phases in the “gap” Use GMAP interpolation if clutter is with strong signal Don’t use GMAP interpolation if clutter is with weak signal

36. GMAP vs. Elliptic GCF Velocity SZ-2 with medium PRT EL = 0.5 deg 10/08/02 15:11 GMT Velocity Legacy medium PRT v a = 23.7 m s -1, r a = 175 km

37. GMAP vs. Elliptic GCF Velocity SZ-2 with GMAP GCF EL = 0.5 deg 10/08/02 15:11 GMT Velocity Legacy with Elliptic GCF v a = 23.7 m s -1, r a = 175 km

38. GMAP vs. Elliptic GCF Velocity SZ-2 with medium PRT EL = 0.5 deg 06/04/03 15:07 GMT Velocity Legacy medium PRT v a = 23.7 m s -1, r a = 175 km

39. GMAP vs. Elliptic GCF Velocity SZ-2 with GMAP GCF EL = 0.5 deg 06/04/03 15:07 GMT Velocity Legacy with Elliptic GCF v a = 23.7 m s -1, r a = 175 km

40. Part Four Clutter Filtering in the SZ-2 Algorithm

41. Clutter Filtering in SZ-2 Clutter filtering is controlled by map –Bypass map: automatically generated –Clutter censor zones: operator defined Issues: –Sequence of operations –Conditions for filtering –Recovery of weak-trip signal –Ground clutter in any trip Overlaid ground clutter –Anomalous propagation in any trip Overlaid ground clutter and AP

42. Basic Sequence of Operations 1.Cohere for trip with clutter 2.Apply clutter filter 3.Cohere for trip with strong signal 4.Recover strong-trip velocity 5.Apply PNF 6.Cohere for trip with weak signal 7.Recover weak-trip velocity

43. Conditions for Filtering Ground clutter –Determined by clutter map AP –Determined by operator (censor zones) and GMAP during long-PRT scan Filter could be bypassed for low CSR –CSR from GMAP is unreliable –Issue: Will clutter maps in ORDA contain clutter power?

44. To filter or not to filter? Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip + Clutter in the 1 st trip –Process: SZ-2 with GMAP Case 1: No filtering Case 2: GMAP with noise estimation Case 3: GMAP with provided noise Case 4: GMAP without interpolation –Output: Statistics of v 1 and v 2 estimates

45. To filter or not to filter? No GCF GMAP GMAP with noise estimation GMAP without interpolation

46. To filter or not to filter? No GCF GMAP GMAP with noise estimation GMAP without interpolation

47. To filter or not to filter? Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip + Clutter in the 2 nd trip –Process: SZ-2 with GMAP Case 1: No filtering Case 2: GMAP with noise estimation Case 3: GMAP with provided noise Case 4: GMAP without interpolation –Output: Statistics of v 1 and v 2 estimates

48. To filter or not to filter? No GCF GMAP GMAP with noise estimation GMAP without interpolation

49. To filter or not to filter? No GCF GMAP GMAP with noise estimation GMAP without interpolation

50. Clutter Filtering Issues If clutter is not with the strong signal, the weak signal cannot be recovered –Weak trip must be censored Should use GMAP without interpolation (notch) when clutter is not with the strong signal –Minor changes to function fSpecFilterGMAP() are required

51. Clutter in Any Trip Clutter can be ground clutter or AP Go after clutter first –Cohere for trip with clutter –Apply clutter filter Censor gates with overlaid clutter –Clutter location can be obtained from bypass map and AP map (generated during long PRT from clutter censor zones and GMAP removed power)

52. Part Five Proposed SZ-2 Algorithm

53. Basic algorithm as reported by NCAR- NSSL in joint report of Aug 15, 2003 Minor changes to handle: –GMAP ground clutter filter –Ground clutter in any trip –Processing notch filter –Spectrum width computation –Censoring Prototype of proposed SZ-2 algorithm coded and tested in MATLAB

54. Changes to use GMAP Window data using Blackman window Compute DFT –Save phases Compute power spectrum Apply GMAP –Save number of coefficients with clutter –Minor changes to function fSpecFilterGMAP() are required

55. Changes to Handle Clutter in Any Trip Analyze bypass map –Determine whether ground clutter is present in No trips One trip Multiple trips If ground clutter is not present –Do not filter If ground clutter is present in just one trip –Cohere to trip with clutter and remove it –Proceed as usual If ground clutter is present in multiple trips –Censor

56. Changes to PNF Center PNF tries to remove most of the strong-trip signal and preserve 2 “clean” replicas of the out-of-trip signal If clutter is not present –PNF is centered at v S (no change) If clutter is present –PNF is centered at “adjusted” v S

57. Processing Notch Filter Location determined by v s and presence of clutter Notch Width determined by strong and weak trip numbers –8 replicas → NW = 3M/4 –4 replicas → NW = M/2 1 st trip cohered 2 nd trip modulated vvsvs v PNF

58. PNF Center Simulation: –Input: Signal in the 1 st trip + Signal in the 2 nd trip + Clutter in the 1 st trip –Process: SZ-2 with GMAP Case 1: PNF centered at v S Case 2: PNF centered at v S /2 Case 3: PNF centered at “adjusted v S ” –Output: Statistics of v 2 estimates

59. PNF Center PNF centered at v S /2 PNF centered at “adjusted v S ” PNF centered at v S

60. Changes to PNF Center PNF must be centered such that –v PNF is the closest to v S –PNF stop band includes GCF notch PNF center is computed from –vS–vS –NW –k GMAP

61. Changes to Spectrum Width Computation Spectrum widths are obtained –From the short-PRT scan for the strong trip –From the long-PRT scan for the weak trip Strong-trip spectrum width computation –Legacy algorithm uses S/R 1 –S must be computed after determination of weak-trip power

62. Spectrum Width Computation Reflectivity Long PRT EL = 0.5 deg 04/06/03 4:26 GMT Spectrum Width Long PRT v a = 8.9 m s -1, r a = 466 km,  v,max = 5.15 m s -1

63. Spectrum Width Computation Spectrum Width Legacy EL = 0.5 deg 04/06/03 4:26 GMT Spectrum Width SZ-2 v a = 23.7 m s -1, r a = 175 km,  v,max = 13.7 m s -1

64. Changes to Censoring Basic censoring remains the same –Use same thresholds as in legacy processing SZ-2 censoring –May need refined constants –Plots of SD(v 1 ) and SD(v 2 ) on the S 1 /S 2 vs.  1 plane are useful to determine censoring constants Issue: Have we only looked at SD(v 2 )? Additional censoring to handle clutter in any trip –Gates with overlaid clutter are censored

65. Summary of Changes to the SZ-2 Algorithm Determine Overlaid Trips and Censoring Determine Location of Ground Clutter Cohere for Ground-Clutter Trip Compute Filtered Power Determine Overlaid Trips and Censoring Cohere for First Trip Compute Filtered and Unfiltered Powers SZ-2 as of Aug 15 2003 Proposed SZ-2 Filter Ground Clutter

66. Summary of Changes to the SZ-2 Algorithm Compute lag-one Correlations for Trips A and B Determine Strong and Weak Trips Compute Strong-Trip Velocity Compute lag-one Correlations for Trips A and B Compute CSR Compute Strong-Trip Velocity Determine Strong and Weak Trips Cohere for Trips A and B

67. Summary of Changes to the SZ-2 Algorithm Apply Window Apply PNF Compute IDFT Cohere for Weak Trip Compute DFT Apply PNF Compute IDFT Cohere for Weak Trip Compute DFT Compute PNF Center Velocity

68. Summary of Changes to the SZ-2 Algorithm Adjust Powers Compute Weak-Trip lag-one Correlation Compute Weak-Trip Velocity Compute Strong-Trip Spectrum Width Adjust Powers Compute Weak-Trip lag-one Correlation Compute Weak-Trip Velocity Compute Weak-Trip Power

69. Summary of Changes to the SZ-2 Algorithm Censor and Threshold Clip and Scale Compute Reflectivity Censor and Threshold Clip and Scale Compute Reflectivity Assign Correct Range

70. Part Six Further Refinements of the SZ-2 Algorithm

71. Further Refinements Proposed SZ-2 algorithm works fine However, there’s room for improvement –Improvements require more work Some are still under research All involve larger changes to proposed SZ-2 algorithm –Improvements are proposed for later releases of SZ-2

72. AP in Any Trip Operator defines zones with AP using system’s clutter censor zones GMAP is used during the long-PRT scan to determine gates with significant clutter in these zones AP map is generated during long-PRT scan AP map and bypass map are combined –Composite map is used in the algorithm

73. Strong Overlaid Echoes Situations where |S 1 /S 2 | < 5 dB may require “double processing” –Cohere clutter-filtered time series to strong trip –Apply PNF –Cohere to weak trip –Compute v w –Cohere clutter-filtered time series to weak trip –Apply PNF –Cohere to strong trip –Compute v s

74. Spectrum Width Computation Weak-trip spectrum width computed from long-PRT scan is limited –Legacy maximum spectrum width = v a /√3 Could use deconvolution –Same drawbacks as in SZ-1 –Needs further testing

75. The End