DiFX: Software Correlation at Swinburne for the LBA Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

9/6/2007VSOP-2 meeting Outline History Correlator architecture Capabilities Computing environment and resources Usage inside and outside LBA Future improvements

9/6/2007VSOP-2 meeting Correlator History Commenced with my PhD thesis (2005) Preliminary work (XF) undertaken in 2004 by Craig West First science late 2005 Conversion to distributed system 2006 Verification with LBA and VLBA correlators, code released late

9/6/2007VSOP-2 meeting Correlator Architecture Written in C++, MPI for message passing, Intel IPP for vector arithmetic

9/6/2007VSOP-2 meeting Capabilities Supports LBA, Mk5 * (K5->LBA converter) Pulsar gating, binning or matched filter Arbitrary time/frequency resolution Presently, visibilities in RPFITS format eVLBI: data can be streamed from network socket directly into correlator Also used for real-time fringe checks *export to linux filesystem

9/6/2007VSOP-2 meeting The Swinburne supercomputer Swinburne supercomputer presently consists of ~300 P4 processors Typically “reserve” ~30 processors for VLBI after observing run Next month: complete replacement of cluster with ~140 dual CPU, quad core nodes, >1100 cores total

9/6/2007VSOP-2 meeting Computing requirements DiFX requires little RAM (except extreme cases): processor speed, cache and SIMD support essential Performance scales linearly with aggregate bandwidth, near-linear with number of stations Realtime example: Max sensitivity LBA (6 x 1 Gbps) requires ~25 new nodes

9/6/2007VSOP-2 meeting Typical LBA usage Required to correlate ~5 days of 256 Mbps before next session (2 months) For each experiment, automatically select and correlate fringe-finders, solve clocks, followed by full correlation. Using ~30 (old) machines, high data rate experiments take 2x observe time

9/6/2007VSOP-2 meeting Non-LBA applications New geodetic array in Australia/NZ to use software correlator: modifications to control, output format VLBA is trialling software correlator to run in parallel with hardware correlator: experiments with specific requirements, testing high data rate

9/6/2007VSOP-2 meeting Future improvements Current version is production: future improvements will be incremental Direct read from MkV Alternative output formats: AIPS++ MS, FITS-IDI, UVFITS Support in AIPS: tasks like CVEL currently confuse with LBA S2 (XF)

9/6/2007VSOP-2 meeting

9/6/2007VSOP-2 meeting Interferometry & Correlators RA Dec Interferometry: delay signals from two dishes to common reference and multiply Common signal, independent noise: average improves S/N Earth rotates, interferometer samples visibility: Fourier transform of sky brightness

9/6/2007VSOP-2 meeting Interferometry & Correlators u ( ) v ( ) Finite bandwidth, not monochromatic, therefore visibility varies across band To get v( ), XF correlator accumulates lags, then FFTs FX correlator FFTs segments of baseband data, and cross- multiply/accumulates

9/6/2007VSOP-2 meeting Hardware vs software Software correlator: program running on a computer cluster/supercomputer Hardware correlator: ASIC boards, specialised data transport Software is unclocked, could be faster or slower than real-time No channel/integration time restrictions Floating pt vs int calculations

9/6/2007VSOP-2 meeting Why software? Flexibility - you can do things that are impossible with a hardware correlator Rapid (and cheap) development Add-ons MUCH easier in software Compatibility Expandability For me: Allow disk-based correlation, and improved pulsar binning (sensitivity)

9/6/2007VSOP-2 meeting DiFX (Distributed FX) Master Node Core 1DataStream 1 DataStream 2 DataStream N Core 2 Core M … … Timerange, destination Baseband data Visibilities Source data MPI is used for inter-process communications

9/6/2007VSOP-2 meeting DiFX (Distributed FX) Configured by text files (like jobscripts) Delay modelling - CALC 9 (separate) Output: RPFITS (built on-the-fly) Arbitrary time/frequency resolution Arbitrary pulsar binning (incoherent dedispersion) - allows weighted bin sum Real time 1 Gbps: CPUs

9/6/2007VSOP-2 meeting DiFX (Distributed FX) Verification: recent successful comparisons with LBA and VLBA

9/6/2007VSOP-2 meeting LBA Science All require one or more of flexibility, high time/frequency resolution, or sensitivity Wide field VLBI (Lenc & Tingay) Masers (Horiuchi) CDF radio counterparts (Norris et al.) RRATs (Kramer et al.) eVLBI (Phillips et al.) Pulsar parallax (me)

9/6/2007VSOP-2 meeting Worldwide science So far, motivated by very high frequency resolution, or flexibility and minimum effort for new system Pulsar scintillation (Brisken) requires extreme frequency resolution (244 Hz channels over 32 MHz bandwidth) Geodesy (MPIfR, Bonn, Germany) Geodesy, new Australian array (NCRIS) t

9/6/2007VSOP-2 meeting Status: Correlator code “Correlation” code completed and verified - now finishing GUI and packaging for public release VLBA, MPIfR and others keen to continue trialling code once released Should be online within a month PASP paper submitted simultaneously

9/6/2007VSOP-2 meeting Status: Observing and reduction Four sessions of eight allocated, two observed - applications for 2007/2008 year (four more sessions) soon Both observed sessions correlated and verified Starting to work on atmospheric/ ionospheric compensation and pipeline, crucial for efficient, accurate astrometry

9/6/2007VSOP-2 meeting Roadmap to submission 1 monthDiFX available online 3 monthsFirst four sessions complete 6 monthsReduction techniques tested, pipeline started 12 monthsPipeline complete, draft begun 18 monthsData collection, reduction and thesis draft complete 20 monthsThesis submission

9/6/2007VSOP-2 meeting Conclusions DiFX is a general purpose software correlator, publicly available soon Already used with success in Australia by the wider VLBI community, and generating interest internationally Pulsar parallax program is underway, with the bulk of observations to come Transitioning from code to science

9/6/2007VSOP-2 meeting

9/6/2007VSOP-2 meeting Interferometry & Correlators Major operations performed by FX software correlator: Delay Unpack quantized data to float Fringe rotate FFT Correct fractional sample error Cross multiply and accumulate A B Datastream Per station  N Per station  N 2