1. March 29, 2010 RFI Mitigation Workshop, Groningen The Netherlands 1 Statistics of the Spectral Kurtosis Estimator Gelu M. Nita and Dale E. Gary New Jersey Institute of Technology

2. March 29, 2010 2 RFI Mitigation Workshop, Groningen The Netherlands Population Spectral Kurtosis

3. March 29, 2010 3 RFI Mitigation Workshop, Groningen The Netherlands Spectral Kurtosis Estimator

4. March 29, 2010 4 RFI Mitigation Workshop, Groningen The Netherlands The SK Spectrometer Key features:   Conceptual simplicity   Frequency channel independence   Straightforward FPGA implementation

5. March 29, 2010 5 RFI Mitigation Workshop, Groningen The Netherlands Statistical thresholds for the rejection of RFI outliers (M>>1) (Nita et. al, 2007 PASP, 119, 805:827) Hardware implementation of the SK excision algorithm and initial tests revealed that, although it performs generally well, the lower threshold level is set too low, failing to reject some RFI contaminated channels the upper threshold level is set too low, rejecting more RFI free channels than statistically expected CONCLUSION: More accurate RFI rejection levels are needed for improved and reliable performance.

6. March 29, 2010 6 RFI Mitigation Workshop, Groningen The Netherlands Characteristics of the SK Estimator (Monte Carlo Simulations) The probability distribution of the SK estimator remains asymmetric even for a fairly large accumulation number M, approaching normality at a slower pace than needed for practical applications. Main goals of this study:   Redefine the SK estimator to remove bias   Find an analytical expression for probability distribution of the SK estimator to allow accurate calculation of its tail probabilities

7. March 29, 2010 7 RFI Mitigation Workshop, Groningen The Netherlands Starting Point

8. March 29, 2010 8 RFI Mitigation Workshop, Groningen The Netherlands Joint Distribution (Monte Carlo): Mean of Squares-Square of Mean  S 2  and  S 1  2 are strongly correlated random variables Their (unknown) joint distribution would be needed to derive the distribution of their ratio Is there any work-around approach?

9. March 29, 2010 9 RFI Mitigation Workshop, Groningen The Netherlands Joint Distribution (Monte Carlo): Mean of Squares to Square of Mean Ratio - Square of Mean Monte Carlo simulations suggest:  S 2  /  S 1  2 and  S 1  2 are uncorrelated random variables Their individual distributions are independent This is the fundamental property that makes the whole SK concept work by allowing SK to have a unity value independently of the power level This property was analytically proven for the exponential distribution based on first principles (Nita & Gary, PASP, in press)

10. March 29, 2010 10 RFI Mitigation Workshop, Groningen The Netherlands Raw statistical moments of the Mean of Squares to Square of Mean Ratio

11. March 29, 2010 11 RFI Mitigation Workshop, Groningen The Netherlands Analytical Results (Nita & Gary, PASP, in press)

12. March 29, 2010 12 RFI Mitigation Workshop, Groningen The Netherlands Redefinition of the SK Estimator

13. March 29, 2010 13 RFI Mitigation Workshop, Groningen The Netherlands Standard Moments of the SK Estimator

14. March 29, 2010 14 RFI Mitigation Workshop, Groningen The Netherlands Moment based approximation of the SK estimator distribution using Pearson Probability Curves

15. March 29, 2010 15 RFI Mitigation Workshop, Groningen The Netherlands Pearson Type IV PDF

16. March 29, 2010 16 RFI Mitigation Workshop, Groningen The Netherlands Pearson IV CF and CCF To compute the tail probabilities of the SK estimator, one needs to evaluate the cumulative function (CF) and complementary cumulative function (CCF) of the Pearson IV probability curve Knowing the analytical expression for the Pearson IV PDF, the CF and CCF can be computed analytically, by using the closed form expressions involving Hypergeometric series provided Heinrich(2004) or Willink(2008). Alternatively, CF and CCF can be computed by a simple numerical integration. The asymmetrical RFI thresholds are then chosen so as to provide symmetric tails probabilities of rejecting true Gaussian signals of 0.13499%, which are equivalent to the ±3  thresholds of a normal distribution.

17. March 29, 2010 17 RFI Mitigation Workshop, Groningen The Netherlands RFI Threshold Computation Example

18. March 29, 2010 18 RFI Mitigation Workshop, Groningen The Netherlands Pearson IV PDF vs. Monte Carlo Simulations

19. March 29, 2010 19 RFI Mitigation Workshop, Groningen The Netherlands Time Domain Kurtosis Estimator

20. March 29, 2010 20 RFI Mitigation Workshop, Groningen The Netherlands Standard Moments of the time domain Kurtosis Estimator

21. March 29, 2010 21 RFI Mitigation Workshop, Groningen The Netherlands Kurtosis estimator Pearson IV PDF and RFI thresholds (M>45)

22. March 29, 2010 22 RFI Mitigation Workshop, Groningen The Netherlands Summary The Spectral Kurtosis RFI excision algorithm recommends itself by   Conceptual simplicity   Frequency channel independence   Straightforward FPGA implementation (See Dale Gary’s following presentation)   Theoretically determined RFI thresholds for arbitrary integration time

23. March 29, 2010 23 RFI Mitigation Workshop, Groningen The Netherlands Main References Gelu M. Nita and dale E. Gary, “Statistics of The Spectral Kurtosis Estimator”, 2010, PASP, in press Gelu M. Nita, Dale E. Gary, Zhiwei Liu, Gordon J. Hurford, & Stephen M. White, 2007, "Radio Frequency Interference Excision Using Spectral Domain Statistics," Publications Of The Astronomical Society Of The Pacific, 119, 805. Yuichi Nagahara, 1999, "The PDF and CF of Pearson type IV distributions and the ML estimation of the parameters," Statistics & Probability Letters, 43, (1999), page 251. Joel Heinrich, 2004, "A Guide to the Pearson Type IV Distribution," Collider Detector at Fermilab internal note 6820, 2004, http://www- cdf.fnal.gov/publications/cdf6820 pearson4.pdf Willink, R., 2008, "A Closed-Form Expression For The Pearson Type IV Distribution Function,“ Aust. N. Z. J. Stat. 50(2), 199, 205