RFI Mitigation Techniques at the ATA Garrett “Karto” Keating RFI2010 – Groningen, Netherlands March 31 st, 2010

A Quick (Re)Introduction Data volume is destined to become HUGE 20 ft/6.1 m Primary Currently 42 Dishes Currently 4 independent IFs and 3 independent FX 64-input correlators Planned to move to 350 dishes, producing 61,425 unique baselines Wideband (0.5 to 11 GHz)

Welcome to RAPID Rapid Automated Processing and Imaging of Data. Flag, Calibrate, Image, Repeat.

RAPID and ARTIS “Offline” data reduction package for ATA data Development began in January 2008 “Online” data reduction package for ATA data Offshoot of the RAPID project First light in March 2008 In regular use since Feb 2009

The truth will set you free… “Effective” troubleshooting I am not a software/ computer engineer A “real” engineer would program in a “real” language I “program” using shell scripts (awk/grep/sed) wrapped around MIRIAD

MacGyver would be proud “Like building a radio telescope from a Swiss army knife and duct tape” –Anonymous ATA Engineer Limitations arise from programming choices: Limited math support Messy code Slow processing The impact of these limitations can be minimized!

Overall Processing Philosophy Assume Nothing Nothing is every truly “static” – LNAs, antennas, RFI, correlators and even objects in the sky are all dynamic Pragmatic Processing Processing choices must maximize time utility “Fallback” Processing Assume tasks will fail to do everything correctly Allow for recovery at each processing stage (RFI/Calibration/Imaging)

System Overview Correlator Antennas Catcher Master Obs Master Obs ARTIS Flag Calibrate Image Archive

The RFI “cube” Time Intensity Bandwidth Long Medium Short Narrow Multichannel Broadband Weak Moderate Strong “Axes of RFI” Most RFI cases can be adequately described by its length, bandwidth and intensity Each case requires a unique strategy and multiple layers of mitigation Not to be confused with Rubik’s cube

Spectral Occupancy Power Frequency “Counts” Frequency

Spectral Occupancy

x100 Scale 0.1  0.2  0.1 

Spectral Occupancy Data is broken up into temporal windows - the more spectra processed, the better/deeper the RFI processing can go With enough spectra, extremely weak RFI can be identified and excised. Bad channels and surrounding channels are flagged.

Benefits No gains solutions required Most spectra can be processing without having solved for the gains solution first Results can be easily combined Spectral occupancy results can be combined across antennas and correlator dumps, or “windowed” to limit processing to only select antennas, baselines, etc. Processing is fast/computationally inexpensive Achieves an 8:1 observing to processing ratio at the ATA

Drawbacks SNR Limitations Results dependant on noise-to-bandpass feature ratio Normally becomes an issue with bright objects Can be corrected by normalizing datasets Can also be corrected with a good gains solution and sky model

Drawbacks Is it really RFI? Results can confuse wanted and unwanted RF emissions Can be solved by creating “good frequency range” masks when flagging Can also be corrected for during the imaging stage

Drawbacks Results require a large number of samples The smaller the array, the higher the required dump rate or the lower the sigma threshold Multiple iterations sometimes required More powerful RFI may need to be culled first before reaching weaker RFI Trouble with “burst” RFI RFI with an extremely short duration not likely to be picked up as consistently

Threshold RFI Removal Real Imaginary

Benefits Fast with calibrated data Hours of data can be processed with this method in a matter of minutes Effective against “burst” RFI Good at catching RFI left behind by spectral occupancy flagging No iterative processing generally required With calibrated data, generally one pass is adequate

Drawbacks Slower with uncalibrated data When used in conjunction with gains-solving processes, normally a 1:2 observing to processing ratio is achieved Iterative processing with uncalibrated data Process must be repeated after a new gains solution is built Vulnerable against weaker RFI RFI below the noise threshold difficult to catch

WRATH RFI Removal Continuum Image Channel by channel

WRATH RFI Removal Pre-WRATHPost-WRATH WRATH flagging stands as the “last guard” against RFI Dynamic range and fidelity in images can easily double following use of the WRATH mechanism

Benefits Excellent “last guard” Can generally catch the weakest of RFI Fast processing, not dependent on data size Processing scales with number of spectral channels, relatively invariant to the size of the dataset Robust against calibration errors Gains-solution errors usually affect all channels, thus don’t bias results

Who needs HI?

Spectral line problems Spectral line features can (again) be misrecognized as RFI Not a lost cause! Spectral line channels can be masked a priori Spectral line features should still image with “better” deconvolution, RFI should not

Drawbacks Relatively slow for small datasets Again, processing time is primarily proportional to number of channels Vulnerable against “burst” RFI RFI with an extremely short duration not likely to be picked up “Baby and bathwater” dilemma Highest risk for throwing away good data, since imaging artifacts may only be due to a single datapoint

Summary Current system is robust against most RFI Still some trouble with transient/”burst” RFI, but future upgrades to fix that Processing able to keep up with observing As processing speed increases, more processing tasks and choices can be made to render better images Processing is highly scalable Good astronomer acceptance Most astronomers using ATA data use the RAPID package, particularly the RFI flagging routines

Source code available at svn.hcro.org/mmm/karto/RAPIDBeta Office number: Allen Telescope Array/Hat Creek Radio Observatory