A new calibration algorithm using non-linear Kalman filters Cyril Tasse The paper has been submitted, ask me if you want a copy of th draft?

Direction Dependent Effects

Direction dependent effects for MeerKAT … When Direction Dependent Effects (DDE) become a problem : Beam MeerKAT/KAT7 Phase center Direction dependent effects for MeerKAT - Each antenna is affect by differnt pointing error - Beam is variable in frequency and time (paralactic angle rotation, dish deformation) Not taking them into account has a direct effect on dynamic range limitations

LOFAR stations are phased arrays … When Direction Dependent Effects (DDE) become a problem : Beam y x 100-250 MHz 30-80 MHz 30-80 MHz y x LOFAR stations are phased arrays - Beam is variable in frequency and time - Projection of the dipoles in the sky is non trivial - Beam can be station-dependent - Individual clock effects --> Strong effects on polarisation

… When Direction Dependent Effects (DDE) become a problem : Ionosphere/troposphere Incoming wavefront Outcoming wavefront PSF Ionosphere Big field of view : station, direction, time and frequency dependent Other direction dependent effects : - Faraday rotation + Effect on the polarisation

? Interferometry TRUTH domain Measurement domain - Ionosphere - Troposphere - Beam - Sky - Faraday rotation - Electronics - etc baseline Non-linear operator h baseline Direction time Hamaker et al. 1994 freq time freq

Unknown physical process Unknown physical process Challenge for the SKA Those processes include: - Antenna beams - Ionosphere - Troposphere - Electronics But also: - Solar wind scintillation - ISM scintillation - and many more... Measurement (10^13++ points) Unknown physical process Unknown physical process Unknown Sky Very (very) large inverse (non-linear) problem

Solvers vs Filters

ill-conditioning Solvers vs Filters ? (iterative versus recursive) Calibration: only Non-Linear solvers have been used so far: - Levenberg-Marquardt - SAGE - StefCal Assuming a sky, find the maximum likelihood - each time/frequency bin independent - each variable independent ill-conditioning For example: “Find a and b given a+b=42” We need to be able to add additional information

While... there is regularity in the process 2 3 2 3 1 1 Incoming wavefront Outcoming wavefront Process parameters are correlated here

While... there is regularity in the process and here too... Beam Beam

While... there is regularity in the process Jones Matrix Time Solver output True underlaying instrumental state

While... there is regularity in the process Jones Matrix Frequency Solver output True underlaying instrumental state

Solvers vs Filters ? (iterative versus recursive) The idea is to: - Properly estimate covariances, to increase robustness in low SNR regime - Impose “smoothness”: Virtually reduce the number of degrees of freedom, and get better conditioning Filter Data string ordered in TIME - All baseline - All polarisations - All frequencies PROBLEM: Our system is non-linear

Non-linear Kalman Filters.... Data domain: Dim=104-106 Process domain: Dim=102-104 xt h Pt Data “predicted” Data h(xt, Pt) - Ionosphere - Clock - Beam - Sky - etc.

Non-linear Kalman Filters.... Data domain: Dim=104-106 Process domain: Dim=102-104 xt h Pt Data “predicted” Data h(xt, Pt) - Ionosphere - Clock - Beam - Sky - etc. xt+1 K Pt+1

Trick 1

First problem Huge diagonal matrix Huge non-diagonal matrix

First problem Using Woodbury matrix identity Huge diagonal matrix Huge non-diagonal matrix Using Woodbury matrix identity

Trick 2

Data projected to a different representation Non-linear Kalman Filters.... Data domain: Dim=104-106 Process domain: Dim=102-104 xt h Pt Data “predicted” Data h(xt, Pt) - Ionosphere - Clock - Beam - Sky - etc. Data projected to a different representation R xt+1 K Data Pt+1 “predicted” Data

Representation issues.... Visibility set R A grid of 3x3 sources Equivalent to 1-Dimentional raw-data images

Representation issues.... Visibility set R Equivalent to 1-Dimentional raw-data images With 1D-Poststamps

Pros and cons - We use much more data (The full bandwidth) ! - We solve for the TRUE underlaying physical parameter - (not for the effective Jones matrix) - We have much less parameters - If some parameters are expected to be correlated, we can specify it - We can specify their dynamical properties - Recursive scheme (against iterative for all other algorthms) - the information from the past is transfered along the recursion - Much better conditioning - Streaming approach - We can solve for everything (including sky term)

Let's try it....

Simulated dataset (ionosphere TEC-screen + Clock drift) - LOFAR HBA (1500 baselines, 4 pols) - 30 subbands from 100 to 150 MHz - S/N=5 - variable ionosphere TEC-screen - Variable clock drift - 18.000 data points / [ time bin (30s)] Measurement Equation Clocks Ionosphere

Simulated dataset (ionosphere TEC-screen + Clock drift)

Clocks solutions

Simulated dataset (Clock drift)

Westerbork dataset (simulated data) RIME

Westerbork dataset (real data) - With WSRT - 8 antenna are intentionally mis-pointed Least-square solutions Smirnov et al. in prep

Westerbork dataset (real data) - With WSRT - 8 antenna are intentionally mis-pointed

Westerbork dataset (real data) Least squares LM solver UKF

Solving for the sky term ?! - Robustness seems high enough now.... - Let's simulate a grid of sources, and solve for their position

Solving for the sky term ?! - Robustness seems high enough now.... - Let's simulate a grid of sources, and solve for their position

Conclusion We want to adress ill conditioning problem for huge amount of data, in low-SNR regime

UKF+Regularisation Pros: Conclusion We want to adress ill conditioning problem for huge amount of data, in low-SNR regime UKF+Regularisation Pros: - We calibrate the raw data in the image plane - We decrease number of parameters by orders of magnitude (seem to properly adress ill-conditioning) - The solutions are physical & valid everywhere (not only at discrete locations) - This robustness allows to add additional degrees of freedom such as sky itself ! - Fairly fast given what it does

UKF+Regularisation Pros: UKF+Regularisation Cons: Conclusion We want to adress ill conditioning problem for huge amount of data, in low-SNR regime UKF+Regularisation Pros: - We calibrate the raw data in the image plane - We decrease number of parameters by orders of magnitude (seem to properly adress ill-conditioning) - The solutions are physical & valid everywhere (not only at discrete locations) - This robustness allows to add additional degrees of freedom such as sky itself ! - Fairly fast given what it does UKF+Regularisation Cons: - Works only if instrument and sky are analytically decribable (need for a “analytically stable” instrument) - Need to access all the frequency data simultaneously (that's why it takes time to get to test the algorihm on real data)