Imaging and Calibration Challenges S. Bhatnagar NRAO, Socorro RMS ~15mJy/beam RMS ~1mJy/beam

Overview Theoretical background Pieces of the puzzle Full beam, full Stokes imaging Dominant sources of error Single pointing (EVLA) Mosaicking (ALMA) Overview of algorithms DD calibration (PB, PB-poln., Wideband effects, etc.), SS deconvolution Computing and I/O loads

The Measurement Equation Generic Measurement Equation: [HBS papers] Data Corruptions Sky :direction independent corruptions :direction dependent corruptions Jij is multiplicative in the Fourier domain Jsij is multiplicative in the Image domain only if Jsi = Jsj 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 ,𝑡 𝑊 𝑖𝑗 𝐽 𝑖𝑗 𝑆 𝐬,,𝑡𝐼𝐬 𝑒  𝑠.𝑏 𝑖𝑗 𝑑𝐬 𝐉 𝑖𝑗 = 𝐉 𝐢 ⊗ 𝐉 𝐣 ∗ 𝐉 𝑖𝑗 𝐬 = 𝐉 𝐢 𝐬 ⊗ 𝐉 𝐣 𝐬∗ 𝐕 𝑖𝑗 𝑂𝑏𝑠 = 𝐉 𝑖𝑗 𝐖 𝑖𝑗 𝐄 𝑖𝑗 𝐕 𝐨 𝐰𝐡𝐞𝐫𝐞 𝐄 𝑖𝑗 =𝐅 𝐉 𝑖𝑗 𝐬 𝐅 𝐓

Pieces of the puzzle Unknowns: Jij ,Jsij, and IM. Efficient algorithms to correct for image plane effects Decomposition of the sky in a more appropriate basis Frequency sensitive Solvers for the “unknown” image plane effects As expensive as imaging! Larger computers! (CPU power + fast I/O) 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀

Dominant sources of error: Single Pointing Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.” EVLA full beam PB effects: Frequency scaling EVLA Memo 58, Brisken

Dominant sources of error: Single Pointing Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.” EVLA full beam, full band PB rotation, pointing errors

Dominant sources of error: Single Pointing Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.” EVLA full beam, full band Error analysis  𝐈 𝐑 =  𝑃𝑆𝐹 ∗ 𝑃𝐵 𝐈 𝐨 Azimuthal cuts at 50%, 10% and 1% of the Stokes-I error pattern AvgPB - PB(to) AvgPB - PB(to)

Dominant sources of error: Single Pointing Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.” EVLA full beam, full Stokes EVLA Polarization squint, In-beam polarization Stokes-V pattern Cross hand power pattern

Dominant sources of error: Single Pointing Requirements: “...full beam, full Stokes, wide-band imaging at full sensitivity”. EVLA full beam Estimated Stokes-I imaging Dynamic Range limit: ~10^4 Stokes-I Stokes-V RMS ~15mJy/beam

Dominant sources of error: Mosaicking ALMA Antenna pointing errors PB rotation Deconvolution errors for extended objects 𝐈 𝐝 = 𝑖 𝑃𝑆𝐹  𝐬 𝐢 ∗ 𝑃𝐵 𝐬 𝐢  𝐈 𝑆𝑘𝑦  𝐬 𝐢  Max. error due to antenna mis-pointing at HPP By definition significant flux at the HPP and in the sidelobes Stokes-I mosaic, VLA @45GHz

Hierarchy of algorithms Unknowns of the problem: Jij ,Jsij, and IM. Jsi = Jsj and independent of time Imaging and calibration as orthogonal operations Self-calibration Deconvolution 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 ,𝑡 𝐽 𝑖𝑗 𝑆 𝐬,,𝑡𝐼𝐬 𝑒  𝑠.𝑏 𝑖𝑗 𝑑𝐬 Data Corruptions Sky 𝑚𝑖𝑛: ∣ 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝐽 𝑖𝑗 . 𝑉 𝑖𝑗 𝑀 ∣ 2 w.r.t. 𝐽 𝑖 𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

Hierarchy of algorithms Jsi (t) = Jsj (t) (Poln. squint, PB correction, etc.) Jsij is multiplicative in the image plane for appropriate Assumptions: Homogeneous arrays :-) [Think ALMA!] Identical antennas ;-) [Think RMS noise of 1microJy/beam!] ∇𝑇  𝐼 𝐷 ∝ℜ 𝑛 𝐽 𝑠 𝑇 𝑛∇𝑇 𝑖𝑗  𝑉 𝑖𝑗 ∇𝑇 𝑒  𝑆.𝐵 𝑖𝑗 (Cornwell, EVLA Memo 62) 𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

Hierarchy of algorithms 𝐉 𝐢 𝐬 𝐭≠ 𝐉 𝐣 𝐬 𝐭 (Pointing offsets, PB variations, etc.) Corrections in the visibility plane Image plane effects not known a-priori Pointing selfcal Correct for Jsij during image deconvolution W-Projection, Aperture-Projection Direct evaluation of the integral Simultaneous solver for Jij ,Jsij, and IM!! (EVLA Memo 84, '04) (EVLA Memo 67, '03) (EVLA Memo 100, '06) (Bhatnagar et al., A&A (submitted)) (Cotton&Uson, EVLA Memo 113,'07) 𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 ,𝑡 𝐽 𝑖𝑗 𝑆 𝐬,,𝑡𝐼𝐬 𝑒  𝑠.𝑏 𝑖𝑗 𝑑𝐬

Parametrization, DoFs, etc.... (Bhatnagar et al.) Corrections in the visibility plane (uses FFT) No assumption about the sky Direct evaluation of the integral (needs DFT) Needs to decompose the image into components Peeling? [Think 300GB database, 1000's of components] 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐸 𝑖𝑗 𝑡∗𝐹𝐹𝑇 𝐼 𝑀 where 𝐸 𝑖𝑗 𝑡= 𝐸 𝑖 𝑡∗ 𝐸 𝑗 ∗ 𝑡 𝐄 𝑖𝑗 𝐓 𝐄 𝑖𝑗 ≈𝟏.𝟎(approximately Unitary operator) 𝐼 𝑑 = 𝐹𝐹𝑇 −1 𝐸 𝑖𝑗 𝑇 ∗ 𝑉 𝑖𝑗 𝑀 Approx. update direction (Cotton&Uson) 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝑘 𝐽 𝑖𝑗 ,𝑡 𝐽 𝑖𝑗 𝑆  𝐬 𝐤 ,,𝑡𝐼 𝐬 𝐤  𝑒  𝐬 𝐤 . 𝐛 𝑖𝑗 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝑘 𝐷𝐹𝑇 𝑃𝐵 𝑖  𝑠 𝑘  𝑃𝐵 𝑗  𝑠 𝑘  𝐺 𝑖𝑗  𝑠 𝑘  𝐼 𝑀  𝑠 𝑘  𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

Direction Dependent Corrections 𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

Pointing SelfCal: “Unit” test Test for the solver using simulated data Red: Simulated pointing offsets t=60sec Blue: Solutions t=600sec 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀

Scale sensitive imaging: Asp-Clean Pixel-to-pixel noise in the image is correlated Keep the DoF in control! Sub-space discovery Scale & strength of emission separates signal (Io) from the noise (IN). Asp-Clean (Bhatnagar & Cornwell, A&A,2004) Search for local scale, amplitude and position 𝐈 𝐃 = 𝑃𝐼 𝐨  𝑃𝐼 𝐍 where𝐏=Beam Matrix Model Image 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀 Asp-Clean MS-Clean Residuals Asp-Clean Residuals

SKA Issues: LNSD vs. SNLD 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀 Simulation of LWA station beam @50MHz (Masaya Kuniyoshi, UNM/AOC) Simulation of ionospheric phase screen (Abhirup Datta, NMT)

Computing & I/O costs Data sizes: (Make a mistake, lose a day! MMLD-database) ALMA: 6 TB/day (Peak), 500 GB/day (avg) EVLA: Typical: 200-500 GB. Could be larger. Residual computation cost dominates the total runtime Test data: ~300GB VLA C-array @ 4.8 GHz 512 Channels, 2 Pol. 4K x 4K imaging A 1000 components deconvolution: ~20hrs. Time split: 35% Computing: 65% I/O Total I/O: ~ 2 TB (10-20 reads of the data for typical processing) DD calibration: As expensive as imaging Increase in computing with more sophisticated parameterization 𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

Computing and I/O costs Cost of computing residual visibilities is dominated by I/O costs for large datasets (~200GB for EVLA) Deconvolution: Approx. 20 access of the entire dataset Calibration: Each trial step in the search accesses the entire dataset Significant increase in run-time due to more sophisticated parameterization Deconvolution: Fast transform (both ways) E.g. limits the use of MCMC approach Calibration: Fast prediction 𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀