1. An FX software correlator for VLBI Adam Deller Swinburne University Australia Telescope National Facility (ATNF)

2. Outline Function of a correlator Reasons for software correlation Code and platforms Progress and current performance Immediate and near future plans Facilitation of real-time gigabit eVLBI

3. Correlator functionality Averaging the product of signals at two telescopes gives visibility

4. Correlator functionality Averaging the product of signals at two telescopes gives visibility To get frequency information, average additional multiplications by lagged signal, then FFT (or reverse for FX) What does it all boil down to?  Gigabit interferometry, 1024 channels with 6 telescopes, FFT cost ~ 200 Gflops (FX)

5. Why software correlate? Flexibility  Handle disparate file formats and bands  High spectral resolution + high b/w  Real time fringe checking Rapid development For Australian LBA  8 x bandwidth, using disk-based recorders instead of S2 tape system

6. Correlator code structure Developed in C++ Disk read/write threaded to hide latency Currently using IPP for vector mathematics fxcorr TelescopeBaseline ModeInputStream

7. Platform options Initial development to run on Swinburne cluster: 200+ 3GHz Pentium 4 machines Cray XD-1 chassis: 12 AMD Opteron processors, 6 Xilinx FPGAs

8. Hybrid Architecture PC flexibility with FPGA power FPGA used for unpacking/fringe rotating and FFT Possibly incorporate cross-multiply FPGA code compiled into library, available as C subroutine call Cray contributing via development of FPGA subroutine library

9. Milestones to date ‘Basic’ and ‘complex’ FX algorithms implemented and verified Fortran driver for CALC ported to C for delay modeling Used in search for first trans-Tasman fringes to 6m antenna in NZ Range of support/analysis packages created

10. Correlator output

11. Current performance LBA: Complex correlation, 6 telescopes, 2x 16MHz bands (128 Mb/s), 4 products/band, one node of cluster:  512 channels: 21x real time  2048 channels: 25x real time  32786 channels: 32x real time 15 telescopes, 128 Mb/s, 4 products  512 channels: 88x real time

12. Immediate applications Improved LBA sensitivity: parallax of double pulsar J0737-3039 High time and frequency resolution + high bandwidth: Wide field imaging Masers, SNRs, AGNs… NGC253 image credit: Emil Lenc (Swinburne)

13. Real-time gigabit eVLBI? All ATNF telescopes of LBA capable of 1 Gb/s, others 512 Mb/s Technical/cabling issues to overcome: disk-based recorders Software correlator ready now Simulated Gb/s data, 512 channels <160x real time  Current system has disk transfer limitation

14. In the near future… Fibre links overcome disk limitations? Investigate alternate architecture: ‘Cell’ processor array at VPAC Code improvements:  Sophisticated pulsar gating  Polyphase filterbank alternative to FFT  Automated configuration from schedule, GUI frontend, output FITS compatible…

15. Integration into the LBA Available for use from now on a limited basis Final release planned before March 2006 observing session Will incorporate automatic configuration, GUI frontend etc Feedback on desired features encouraged

16. Applications outside the LBA Potentially useful for niche applications anywhere Applicable to connected- element as well as VLBI “Piggy-backing” on existing interferometers - only need facility to get data to disk, and some processors Existing correlator limited + disk space available + processors = SOFTWARE CORRELATE

17. Conclusions Fast, flexible new FX software correlator for generic clusters developed Hybrid architecture will be explored Will provide significant increase in LBA sensitivity until next upgrade Real-time gigabit eVLBI using the software correlator is feasible on Swinburne cluster Final release planned in early 2006

18. Questions??

19. XD-1 Specifications 12x 2.2GHz Opteron processors - 106 Gflops total 1-8 GB RAM per processor, 12.8GB/sec 4-8 GB/s interconnect (between nodes) 6 Xilinx Virtex 4 FPGAs 4 PCI-X slots for ethernet/fibre channel Max 1.5 TB local storage