1. Distributed FX software correlation Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF) Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

2. 28/09/2006Torun, Poland Outline Software vs hardware correlators Why software (what it does best, and the price you pay) Current applications The DiFX architecture Performance and current status Conclusions Software vs hardware correlators Why software (what it does best, and the price you pay) Current applications The DiFX architecture Performance and current status Conclusions

3. 28/09/2006Torun, Poland Software vs Hardware “Software correlator”: program running on a supercomputer/cluster Software is unclocked, could be faster or slower than real-time No channel/integration time restrictions Floating pt vs int calculations “Software correlator”: program running on a supercomputer/cluster Software is unclocked, could be faster or slower than real-time No channel/integration time restrictions Floating pt vs int calculations

4. 28/09/2006Torun, Poland Why software Flexibility - can do science that is impossible with hardware correlators Rapid (and cheap) development Add-ons MUCH easier in software Compatibility and expandability In general, less approximations due to the use of floating point Flexibility - can do science that is impossible with hardware correlators Rapid (and cheap) development Add-ons MUCH easier in software Compatibility and expandability In general, less approximations due to the use of floating point

5. 28/09/2006Torun, Poland The price you pay Because software & hardware not specifically tuned, less computation per $$ hardware Therefore not useful for EVLA/SKA scale correlators Running costs (electricity, aircon) may be higher Because software & hardware not specifically tuned, less computation per $$ hardware Therefore not useful for EVLA/SKA scale correlators Running costs (electricity, aircon) may be higher

6. 28/09/2006Torun, Poland Current applications/results New science the software correlator has enabled: –Disk-based LBA (greater bandwidth) –Adding global disk-based antennas to LBA experiments –High time and frequency resolution allowing wide-field imaging –VERY high frequency resolution for pulsar scintillation studies New science the software correlator has enabled: –Disk-based LBA (greater bandwidth) –Adding global disk-based antennas to LBA experiments –High time and frequency resolution allowing wide-field imaging –VERY high frequency resolution for pulsar scintillation studies

7. 28/09/2006Torun, Poland The DiFX architecture Master Node Core 1 DataStream 1 DataStream 2 DataStream N Core 2 Core M … … … … Timerange, destination Baseband data Visibilities Source data MPI is used for inter-process communications

8. 28/09/2006Torun, Poland Usage in the LBA PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas) Data is recorded directly to Linux- formatted storage (Apple Xraid) Disks shipped to Swinburne Correlated at the Swinburne supercomputer (~300 P4 machines ) Ftp fringe tests and 256 Mbps eVLBI PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas) Data is recorded directly to Linux- formatted storage (Apple Xraid) Disks shipped to Swinburne Correlated at the Swinburne supercomputer (~300 P4 machines ) Ftp fringe tests and 256 Mbps eVLBI

9. 28/09/2006Torun, Poland Performance Real-time LBA @ 1 Gbps (6 stations x 4x64 MHz bands): 100-200 CPUs (circa 2004 P4s)

10. 28/09/2006Torun, Poland Current status Verification via correlator comparison with VLBA completed Used in production capacity with LBA - complete switch as soon as sufficient disks available Further development (graphical frontend, FITS-IDI etc) ongoing Exploring potential for geodesy Verification via correlator comparison with VLBA completed Used in production capacity with LBA - complete switch as soon as sufficient disks available Further development (graphical frontend, FITS-IDI etc) ongoing Exploring potential for geodesy

11. 28/09/2006Torun, Poland Conclusions Software correlators can be rapidly developed and enable science that cannot be done elsewhere Perfect for niche/part-time experiments and feasibility tests Used successfully with the LBA Early science results encouraging! Software correlators can be rapidly developed and enable science that cannot be done elsewhere Perfect for niche/part-time experiments and feasibility tests Used successfully with the LBA Early science results encouraging!

12. 28/09/2006Torun, Poland