1. Imaging and Calibration Challenges
S. Bhatnagar
NRAO, Socorro
RMS ~15mJy/beam
RMS ~1mJy/beam

2. Overview Theoretical background Pieces of the puzzle
Full beam, full Stokes imaging
Dominant sources of error
Single pointing (EVLA)
Mosaicking (ALMA)
Overview of algorithms
DD calibration (PB, PB-poln., Wideband effects, etc.), SS deconvolution
Computing and I/O loads

3. The Measurement Equation
Generic Measurement Equation: [HBS papers]
Data Corruptions Sky
:direction independent corruptions
:direction dependent corruptions
Jij is multiplicative in the Fourier domain
Jsij is multiplicative in the Image domain only if Jsi = Jsj
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 ,𝑡 𝑊 𝑖𝑗 𝐽 𝑖𝑗 𝑆 𝐬,,𝑡𝐼𝐬 𝑒  𝑠.𝑏 𝑖𝑗 𝑑𝐬
𝐉 𝑖𝑗 = 𝐉 𝐢 ⊗ 𝐉 𝐣 ∗
𝐉 𝑖𝑗 𝐬 = 𝐉 𝐢 𝐬 ⊗ 𝐉 𝐣 𝐬∗
𝐕 𝑖𝑗 𝑂𝑏𝑠 = 𝐉 𝑖𝑗 𝐖 𝑖𝑗 𝐄 𝑖𝑗 𝐕 𝐨 𝐰𝐡𝐞𝐫𝐞 𝐄 𝑖𝑗 =𝐅 𝐉 𝑖𝑗 𝐬 𝐅 𝐓

4. Pieces of the puzzle Unknowns: Jij ,Jsij, and IM.
Efficient algorithms to correct for image plane effects
Decomposition of the sky in a more appropriate basis
Frequency sensitive
Solvers for the “unknown” image plane effects
As expensive as imaging!
Larger computers! (CPU power + fast I/O)
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀

5. Dominant sources of error: Single Pointing
Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.”
EVLA full beam
PB effects: Frequency scaling
EVLA Memo 58,
Brisken

6. Dominant sources of error: Single Pointing
Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.”
EVLA full beam, full band
PB rotation, pointing errors

7. Dominant sources of error: Single Pointing
Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.”
EVLA full beam, full band
Error analysis
 𝐈 𝐑 =  𝑃𝑆𝐹 ∗ 𝑃𝐵 𝐈 𝐨
Azimuthal cuts at 50%, 10% and 1% of the Stokes-I
error pattern AvgPB - PB(to)
AvgPB - PB(to)

8. Dominant sources of error: Single Pointing
Requirements: “...full beam, full band, full Stokes imaging at full sensitivity.”
EVLA full beam, full Stokes
EVLA Polarization squint, In-beam polarization
Stokes-V pattern
Cross hand power pattern

9. Dominant sources of error: Single Pointing
Requirements: “...full beam, full Stokes, wide-band imaging at full sensitivity”.
EVLA full beam
Estimated Stokes-I imaging Dynamic Range limit: ~10^4
Stokes-I
Stokes-V
RMS ~15mJy/beam

10. Dominant sources of error: Mosaicking
ALMA
Antenna pointing errors
PB rotation
Deconvolution errors for extended objects
𝐈 𝐝 = 𝑖 𝑃𝑆𝐹  𝐬 𝐢 ∗ 𝑃𝐵 𝐬 𝐢  𝐈 𝑆𝑘𝑦  𝐬 𝐢 
Max. error due to antenna
mis-pointing at HPP
By definition significant
flux at the HPP and
in the sidelobes
Stokes-I mosaic,

11. Hierarchy of algorithms
Unknowns of the problem: Jij ,Jsij, and IM.
Jsi = Jsj and independent of time
Imaging and calibration as orthogonal operations
Self-calibration
Deconvolution
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 ,𝑡 𝐽 𝑖𝑗 𝑆 𝐬,,𝑡𝐼𝐬 𝑒  𝑠.𝑏 𝑖𝑗 𝑑𝐬
Data Corruptions Sky
𝑚𝑖𝑛: ∣ 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝐽 𝑖𝑗 . 𝑉 𝑖𝑗 𝑀 ∣ 2 w.r.t. 𝐽 𝑖
𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

12. Hierarchy of algorithms
Jsi (t) = Jsj (t) (Poln. squint, PB correction, etc.)
Jsij is multiplicative in the image plane for appropriate
Assumptions:
Homogeneous arrays
:-) [Think ALMA!]
Identical antennas
;-) [Think RMS noise of 1microJy/beam!] ∇𝑇
 𝐼 𝐷 ∝ℜ 𝑛 𝐽 𝑠 𝑇 𝑛∇𝑇 𝑖𝑗  𝑉 𝑖𝑗 ∇𝑇 𝑒  𝑆.𝐵 𝑖𝑗
(Cornwell,
EVLA Memo 62)
𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

13. Hierarchy of algorithms
𝐉 𝐢 𝐬 𝐭≠ 𝐉 𝐣 𝐬 𝐭
(Pointing offsets, PB variations, etc.)
Corrections in the visibility plane
Image plane effects not known a-priori
Pointing selfcal
Correct for Jsij during image deconvolution
W-Projection, Aperture-Projection
Direct evaluation of the integral
Simultaneous solver for Jij ,Jsij, and IM!!
(EVLA Memo 84, '04)
(EVLA Memo 67, '03)
(EVLA Memo 100, '06)
(Bhatnagar et al., A&A (submitted))
(Cotton&Uson, EVLA Memo 113,'07)
𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 ,𝑡 𝐽 𝑖𝑗 𝑆 𝐬,,𝑡𝐼𝐬 𝑒  𝑠.𝑏 𝑖𝑗 𝑑𝐬

14. Parametrization, DoFs, etc....
(Bhatnagar et al.)
Corrections in the visibility plane (uses FFT)
No assumption about the sky
Direct evaluation of the integral (needs DFT)
Needs to decompose the image into components
Peeling? [Think 300GB database, 1000's of components]
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐸 𝑖𝑗 𝑡∗𝐹𝐹𝑇 𝐼 𝑀 where 𝐸 𝑖𝑗 𝑡= 𝐸 𝑖 𝑡∗ 𝐸 𝑗 ∗ 𝑡
𝐄 𝑖𝑗 𝐓 𝐄 𝑖𝑗 ≈𝟏.𝟎(approximately Unitary operator)
𝐼 𝑑 = 𝐹𝐹𝑇 −1 𝐸 𝑖𝑗 𝑇 ∗ 𝑉 𝑖𝑗 𝑀 Approx. update direction
(Cotton&Uson)
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝑘 𝐽 𝑖𝑗 ,𝑡 𝐽 𝑖𝑗 𝑆  𝐬 𝐤 ,,𝑡𝐼 𝐬 𝐤  𝑒  𝐬 𝐤 . 𝐛 𝑖𝑗
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝑘 𝐷𝐹𝑇 𝑃𝐵 𝑖  𝑠 𝑘  𝑃𝐵 𝑗  𝑠 𝑘  𝐺 𝑖𝑗  𝑠 𝑘  𝐼 𝑀  𝑠 𝑘 
𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

15. Direction Dependent Corrections
𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

16. Pointing SelfCal: “Unit” test
Test for the solver using simulated data
Red: Simulated pointing offsets t=60sec
Blue: Solutions t=600sec
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀

17. Scale sensitive imaging: Asp-Clean
Pixel-to-pixel noise in the image is correlated
Keep the DoF in control!
Sub-space discovery
Scale & strength of emission separates
signal (Io) from the noise (IN).
Asp-Clean (Bhatnagar & Cornwell, A&A,2004)
Search for local scale, amplitude and position
𝐈 𝐃 = 𝑃𝐼 𝐨  𝑃𝐼 𝐍 where𝐏=Beam Matrix
Model Image
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀
Asp-Clean
MS-Clean Residuals
Asp-Clean Residuals

18. SKA Issues: LNSD vs. SNLD
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀
Simulation of LWA station
(Masaya Kuniyoshi, UNM/AOC)
Simulation of ionospheric phase screen
(Abhirup Datta, NMT)

19. Computing & I/O costs
Data sizes: (Make a mistake, lose a day! MMLD-database)
ALMA: 6 TB/day (Peak), 500 GB/day (avg)
EVLA: Typical: GB. Could be larger.
Residual computation cost dominates the total runtime
Test data: ~300GB
VLA 4.8 GHz
512 Channels, 2 Pol.
4K x 4K imaging
A 1000 components deconvolution: ~20hrs.
Time split: 35% Computing: 65% I/O
Total I/O: ~ 2 TB (10-20 reads of the data for typical processing)
DD calibration: As expensive as imaging
Increase in computing with more sophisticated parameterization
𝑚𝑖𝑛: ∣ 𝐽 𝑖𝑗 −1 𝑉 𝑖𝑗 𝑂𝑏𝑠 − 𝑉 𝑀 ∣ 2 w.r.t. 𝐼 𝑀

20. Computing and I/O costs
Cost of computing residual visibilities is dominated by I/O costs for large datasets (~200GB for EVLA)
Deconvolution: Approx. 20 access of the entire dataset
Calibration: Each trial step in the search accesses the entire dataset
Significant increase in run-time due to more sophisticated parameterization
Deconvolution: Fast transform (both ways)
E.g. limits the use of MCMC approach
Calibration: Fast prediction
𝑉 𝑖𝑗 𝑂𝑏𝑠 = 𝐽 𝑖𝑗 𝑉 𝑖𝑗 𝑀