1. A new calibration algorithm using non-linear Kalman filters
Cyril Tasse
The paper has been submitted, ask me if you want a copy of th draft?

2. Direction Dependent Effects

3. Direction dependent effects for MeerKAT
… When Direction Dependent Effects (DDE) become a problem : Beam
MeerKAT/KAT7
Phase center
Direction dependent effects for MeerKAT
- Each antenna is affect by differnt pointing error
- Beam is variable in frequency and time (paralactic angle rotation, dish deformation)
Not taking them into account has a direct effect on dynamic range limitations

4. LOFAR stations are phased arrays
… When Direction Dependent Effects (DDE) become a problem : Beam y x
MHz
30-80 MHz
30-80 MHz y x
LOFAR stations are phased arrays
- Beam is variable in frequency and time
- Projection of the dipoles in the sky is non trivial
- Beam can be station-dependent
- Individual clock effects
--> Strong effects on polarisation

5. … When Direction Dependent Effects (DDE) become a problem : Ionosphere/troposphere
Incoming wavefront
Outcoming wavefront
PSF
Ionosphere
Big field of view : station, direction, time and frequency dependent
Other direction dependent effects :
- Faraday rotation
+ Effect on the polarisation

6. ? Interferometry TRUTH domain Measurement domain - Ionosphere
- Troposphere
- Beam
- Sky
- Faraday rotation
- Electronics
- etc
baseline
Non-linear operator h
baseline
Direction
time
Hamaker et al. 1994
freq
time
freq

7. Unknown physical process Unknown physical process
Challenge for the SKA
Those processes include:
- Antenna beams
- Ionosphere
- Troposphere
- Electronics
But also:
- Solar wind scintillation
- ISM scintillation
- and many more...
Measurement
(10^13++ points)
Unknown physical process
Unknown physical process
Unknown Sky
Very (very) large inverse (non-linear) problem

8. Solvers vs Filters

9. ill-conditioning Solvers vs Filters ? (iterative versus recursive)
Calibration: only Non-Linear solvers have been used so far:
- Levenberg-Marquardt
- SAGE
- StefCal
Assuming a sky, find the maximum likelihood
- each time/frequency bin independent
- each variable independent
ill-conditioning
For example:
“Find a and b given a+b=42”
We need to be able to add additional information

10. While... there is regularity in the process2 3 2 3 1 1
Incoming wavefront
Outcoming wavefront
Process parameters are correlated here

11. While... there is regularity in the process
and here too...
Beam
Beam

12. While... there is regularity in the process
Jones Matrix
Time
Solver output
True underlaying instrumental state

13. While... there is regularity in the process
Jones Matrix
Frequency
Solver output
True underlaying instrumental state

14. Solvers vs Filters ? (iterative versus recursive)
The idea is to:
- Properly estimate covariances, to increase robustness in low SNR regime
- Impose “smoothness”: Virtually reduce the number of degrees of freedom, and get better conditioning
Filter
Data string ordered in TIME
- All baseline
- All polarisations
- All frequencies
PROBLEM: Our system is non-linear

15. Non-linear Kalman Filters....
Data domain: Dim=
Process domain: Dim= xt h Pt
Data
“predicted” Data
h(xt, Pt)
- Ionosphere
- Clock
- Beam
- Sky
- etc.

16. Non-linear Kalman Filters....
Data domain: Dim=
Process domain: Dim= xt h Pt
Data
“predicted” Data
h(xt, Pt)
- Ionosphere
- Clock
- Beam
- Sky
- etc.
xt+1 K
Pt+1

17. Trick 1

18. First problem
Huge diagonal matrix
Huge non-diagonal matrix

19. First problem Using Woodbury matrix identity Huge diagonal matrix
Huge non-diagonal matrix
Using Woodbury matrix identity

20. Trick 2

21. Data projected to a different representation
Non-linear Kalman Filters....
Data domain: Dim=
Process domain: Dim= xt h Pt
Data
“predicted” Data
h(xt, Pt)
- Ionosphere
- Clock
- Beam
- Sky
- etc.
Data projected to a different representation R
xt+1 K
Data
Pt+1
“predicted” Data

22. Representation issues....
Visibility set R
A grid of 3x3 sources
Equivalent to 1-Dimentional raw-data images

23. Representation issues....
Visibility set R
Equivalent to 1-Dimentional raw-data images
With 1D-Poststamps

24. Pros and cons - We use much more data (The full bandwidth) !
- We solve for the TRUE underlaying physical parameter
- (not for the effective Jones matrix)
- We have much less parameters
- If some parameters are expected to be correlated, we can specify it
- We can specify their dynamical properties
- Recursive scheme (against iterative for all other algorthms)
- the information from the past is transfered along the recursion
- Much better conditioning
- Streaming approach
- We can solve for everything (including sky term)

25. Let's try it....

26. Simulated dataset (ionosphere TEC-screen + Clock drift)
- LOFAR HBA (1500 baselines, 4 pols)
- 30 subbands from 100 to 150 MHz
- S/N=5
- variable ionosphere TEC-screen
- Variable clock drift
data points / [ time bin (30s)]
Measurement Equation
Clocks
Ionosphere

27. Simulated dataset (ionosphere TEC-screen + Clock drift)

28. Clocks solutions

29. Simulated dataset (Clock drift)

30. Westerbork dataset (simulated data)
RIME

31. Westerbork dataset (real data)
- With WSRT
- 8 antenna are intentionally mis-pointed
Least-square solutions
Smirnov et al. in prep

32. Westerbork dataset (real data)
- With WSRT
- 8 antenna are intentionally mis-pointed

33. Westerbork dataset (real data)
Least squares LM solver
UKF

34. Solving for the sky term ?!
- Robustness seems high enough now....
- Let's simulate a grid of sources, and solve for their position

35. Solving for the sky term ?!
- Robustness seems high enough now....
- Let's simulate a grid of sources, and solve for their position

36. Conclusion
We want to adress ill conditioning problem for huge amount of data, in low-SNR regime

37. UKF+Regularisation Pros:
Conclusion
We want to adress ill conditioning problem for huge amount of data, in low-SNR regime
UKF+Regularisation Pros:
- We calibrate the raw data in the image plane
- We decrease number of parameters by orders of magnitude (seem to properly adress ill-conditioning)
- The solutions are physical & valid everywhere (not only at discrete locations)
- This robustness allows to add additional degrees of freedom such as sky itself !
- Fairly fast given what it does

38. UKF+Regularisation Pros: UKF+Regularisation Cons:
Conclusion
We want to adress ill conditioning problem for huge amount of data, in low-SNR regime
UKF+Regularisation Pros:
- We calibrate the raw data in the image plane
- We decrease number of parameters by orders of magnitude (seem to properly adress ill-conditioning)
- The solutions are physical & valid everywhere (not only at discrete locations)
- This robustness allows to add additional degrees of freedom such as sky itself !
- Fairly fast given what it does
UKF+Regularisation Cons:
- Works only if instrument and sky are analytically decribable (need for a “analytically stable” instrument)
- Need to access all the frequency data simultaneously (that's why it takes time to get to test the algorihm on real data)