1. RFI Mitigation Techniques at the ATA Garrett “Karto” Keating RFI2010 – Groningen, Netherlands March 31 st, 2010

2. A Quick (Re)Introduction Data volume is destined to become HUGE 20 ft/6.1 m Primary Currently 42 Dishes Currently 4 independent IFs and 3 independent FX 64-input correlators Planned to move to 350 dishes, producing 61,425 unique baselines Wideband (0.5 to 11 GHz)

3. Welcome to RAPID Rapid Automated Processing and Imaging of Data. Flag, Calibrate, Image, Repeat.

4. RAPID and ARTIS “Offline” data reduction package for ATA data Development began in January 2008 “Online” data reduction package for ATA data Offshoot of the RAPID project First light in March 2008 In regular use since Feb 2009

5. The truth will set you free… “Effective” troubleshooting I am not a software/ computer engineer A “real” engineer would program in a “real” language I “program” using shell scripts (awk/grep/sed) wrapped around MIRIAD

6. MacGyver would be proud “Like building a radio telescope from a Swiss army knife and duct tape” –Anonymous ATA Engineer Limitations arise from programming choices: Limited math support Messy code Slow processing The impact of these limitations can be minimized!

7. Overall Processing Philosophy Assume Nothing Nothing is every truly “static” – LNAs, antennas, RFI, correlators and even objects in the sky are all dynamic Pragmatic Processing Processing choices must maximize time utility “Fallback” Processing Assume tasks will fail to do everything correctly Allow for recovery at each processing stage (RFI/Calibration/Imaging)

8. System Overview Correlator Antennas Catcher Master Obs Master Obs ARTIS Flag Calibrate Image Archive

9. The RFI “cube” Time Intensity Bandwidth Long Medium Short Narrow Multichannel Broadband Weak Moderate Strong “Axes of RFI” Most RFI cases can be adequately described by its length, bandwidth and intensity Each case requires a unique strategy and multiple layers of mitigation Not to be confused with Rubik’s cube

10. Spectral Occupancy Power Frequency “Counts” Frequency

11. Spectral Occupancy

12. x100 Scale 0.1  0.2  0.1 

13. Spectral Occupancy Data is broken up into temporal windows - the more spectra processed, the better/deeper the RFI processing can go With enough spectra, extremely weak RFI can be identified and excised. Bad channels and surrounding channels are flagged.

14. Benefits No gains solutions required Most spectra can be processing without having solved for the gains solution first Results can be easily combined Spectral occupancy results can be combined across antennas and correlator dumps, or “windowed” to limit processing to only select antennas, baselines, etc. Processing is fast/computationally inexpensive Achieves an 8:1 observing to processing ratio at the ATA

15. Drawbacks SNR Limitations Results dependant on noise-to-bandpass feature ratio Normally becomes an issue with bright objects Can be corrected by normalizing datasets Can also be corrected with a good gains solution and sky model

16. Drawbacks Is it really RFI? Results can confuse wanted and unwanted RF emissions Can be solved by creating “good frequency range” masks when flagging Can also be corrected for during the imaging stage

17. Drawbacks Results require a large number of samples The smaller the array, the higher the required dump rate or the lower the sigma threshold Multiple iterations sometimes required More powerful RFI may need to be culled first before reaching weaker RFI Trouble with “burst” RFI RFI with an extremely short duration not likely to be picked up as consistently

18. Threshold RFI Removal Real Imaginary

19. Benefits Fast with calibrated data Hours of data can be processed with this method in a matter of minutes Effective against “burst” RFI Good at catching RFI left behind by spectral occupancy flagging No iterative processing generally required With calibrated data, generally one pass is adequate

20. Drawbacks Slower with uncalibrated data When used in conjunction with gains-solving processes, normally a 1:2 observing to processing ratio is achieved Iterative processing with uncalibrated data Process must be repeated after a new gains solution is built Vulnerable against weaker RFI RFI below the noise threshold difficult to catch

21. WRATH RFI Removal Continuum Image Channel by channel

22. WRATH RFI Removal Pre-WRATHPost-WRATH WRATH flagging stands as the “last guard” against RFI Dynamic range and fidelity in images can easily double following use of the WRATH mechanism

23. Benefits Excellent “last guard” Can generally catch the weakest of RFI Fast processing, not dependent on data size Processing scales with number of spectral channels, relatively invariant to the size of the dataset Robust against calibration errors Gains-solution errors usually affect all channels, thus don’t bias results

24. Who needs HI?

25. Spectral line problems Spectral line features can (again) be misrecognized as RFI Not a lost cause! Spectral line channels can be masked a priori Spectral line features should still image with “better” deconvolution, RFI should not

26. Drawbacks Relatively slow for small datasets Again, processing time is primarily proportional to number of channels Vulnerable against “burst” RFI RFI with an extremely short duration not likely to be picked up “Baby and bathwater” dilemma Highest risk for throwing away good data, since imaging artifacts may only be due to a single datapoint

27. Summary Current system is robust against most RFI Still some trouble with transient/”burst” RFI, but future upgrades to fix that Processing able to keep up with observing As processing speed increases, more processing tasks and choices can be made to render better images Processing is highly scalable Good astronomer acceptance Most astronomers using ATA data use the RAPID package, particularly the RFI flagging routines

28. Source code available at svn.hcro.org/mmm/karto/RAPIDBeta karto@hcro.org Office number: 1-530-335-2364 Allen Telescope Array/Hat Creek Radio Observatory