Distributed FX software correlation Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof.

Slides:

Advertisements

Similar presentations

VLBI progress Down-under Tasso Tzioumis Australia Telescope National Facility (ATNF) 25 September 2008.

Advertisements

ELBA – Current Developments Chris Phillips eVLBI Project Scientist 16 June 2008.

SCARIe: Realtime software correlation Nico Kruithof, Damien Marchal.

E-VLBI down-under Tasso Tzioumis ATNF, CSIRO July 2005 Towards e-VLBI.

E-VLBI Progress with the LBA Chris Phillips, eVLBI Project Scientist November 2011.

E-VLBI progress in the South Tasso Tzioumis ATNF, CSIRO September 2006 “Towards e-VLBI”  “e-VLBI”

A supercomputer-based software correlator at the Swinburne University of Technology Tingay, S.J. and Deller, A.

Low-Frequency Pulsar Surveys and Supercomputing Matthew Bailes.

Baseband Processing and SKA Simulations using Supercomputers: Enhancing Australia‘s radio astronomy facilities and its bid for the SKA Steven Tingay Swinburne.

Summary Role of Software (1 slide) ARCS Software Architecture (4 slides) SNS -- Caltech Interactions (3 slides)

Virtual Machines for HPC Paul Lu, Cam Macdonell Dept of Computing Science.

Baseband processing, e-VLBI, and SKA simulations Steven Tingay Swinburne University of Technology MNRF Symposium, 8 June, 2004.

Ongoing e-VLBI Developments with K5 VLBI System Hiroshi Takeuchi, Tetsuro Kondo, Yasuhiro Koyama, and Moritaka Kimura Kashima Space Research Center/NICT.

VLBI Report LBA Operations & eVLBI Tasso Tzioumis ATNF May 2008.

Electronic Transmission of Very- Long Baseline Interferometry Data National Internet2 day, March 18, 2004 David LapsleyAlan Whitney MIT Haystack Observatory,

2012 DiFX user’s meeting, Sydney, Australia USNO Software Correlator: Status Report Outline USNO/WACO Background USNO Prototype Software Correlator (UPSC)

Department of Computer and Information Science, School of Science, IUPUI Dale Roberts, Lecturer Computer Science, IUPUI CSCI.

CVN software correlator development and its applications Zheng Weimin*, Zhang Juan, Tong Li, Tong Fengxian, Liu Lei, Chen Zhong, Shu Fengchun, Wang Guangli.

Current LBA Developments Chris Phillips CSIRO ATNF 13/7/2005.

DiFX: Software Correlation at Swinburne for the LBA Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven.

Tuning DiFX2 for performance Adam Deller ASTRON 6th DiFX workshop, CSIRO ATNF, Sydney AUS.

Recent Developments in Australia Phil Edwards Head of Science Operations CSIRO ATNF.

E-VLBI at ≥ 1 Gbps -- “unlimited” networks? Tasso Tzioumis Australia Telescope National Facility (ATNF) 4 November 2008.

LBA Calibrator Survey Chris Phillips eVLBI Project Scientist 23 July 2009.

Update on LBA Technical Developments CASS Chris Phillips 11 October 2013.

Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array The VLBA Sensitivity.

VLBA Sensitivity Upgrade Project Status Jon Romney NRAO, Socorro 7 th US VLBI Technical Coordination Meeting Haystack Observatory 2009 November 3–4.

Recent results from the Swinburne supercomputer VLBI software correlator(s) Steven Tingay, Craig West, Adam Deller, Shinji Horiuchi Centre for Astrophysics.

Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array New VLBA capabilities.

Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array The VLBA Sensitivity.

High Data Rate Transfer for HEP and VLBI Ralph Spencer, Richard Hughes-Jones and Simon Casey The University of Manchester Netwrokshop33 March 2005.

High Bandwidth Data Acquisition and Network Streaming in VLBI Jan Wagner, Guifré Molera et al. TKK / Metsähovi Radio Observatory.

Design of a Software Correlator for the Phase I SKA Jongsoo Kim Cavendish Lab., Univ. of Cambridge & Korea Astronomy and Space Science Institute Collaborators:

Accelerating a Software Radio Astronomy Correlator By Andrew Woods Supervisor: Prof. Inggs & Dr Langman.

VLBI/eVLBI with the 305-m Arecibo Radio Telescope Chris Salter Tapasi Ghosh Emmanuel Momjian Arun Venkataraman Jon Hagen.

Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array Software correlators.

EVLBI and the Australian SKA Pathfinder (ASKAP) Prof. Steven Tingay Curtin University of Technology Perth, Australia 7th eVLBI workshop SHAO, 2008 June.

Eleventh Synthesis Imaging Workshop Socorro, June 10-17, 2008 THE VLBA SENSITIVITY UPGRADE Craig Walker, NRAO.

Correlator Growth Path EVLA Advisory Committee Meeting, March 19-20, 2009 Michael P. Rupen Project Scientist for WIDAR.

Next Generation Digital Back-ends at the GMRT Yashwant Gupta Yashwant Gupta National Centre for Radio Astrophysics Pune India CASPER meeting Cambridge.

GPU DAS CSIRO ASTRONOMY AND SPACE SCIENCE Chris Phillips 23 th October 2012.

Status and capabilities of the Australian LBA Phil Diamond Chief, Astronomy & Space Science.

EVLA Transition to Science Operations: An Overview EVLA Advisory Committee Meeting, March 19-20, 2009 Bob Dickman AD, NM Operations.

DiFX Performance Testing Chris Phillips eVLBI Project Scientist 25 June 2009.

An FX software correlator for VLBI Adam Deller Swinburne University Australia Telescope National Facility (ATNF)

Update on the Software Correlator Nico Kruithof, Huseyin Özdemir, Yurii Pydoprihora, Ruud Oerlemans, Mark Kettenis, JIVE.

by Arjun Radhakrishnan supervised by Prof. Michael Inggs

Tenth Synthesis Imaging Summer School UNM, June 13-20, 2006 Cross Correlators Walter Brisken.

DiFX New Features. Disk based file specification Improved specification of files to correlate Wild cards with time of scan name filtering? External reference.

Pulsar tools in DiFX Adam Deller ASTRON 6th DiFX workshop, CSIRO ATNF, Sydney AUS.

DiFX Overview Adam Deller NRAO 3rd DiFX workshop, Curtin University, Perth.

Review of developments in Australasia and mainland Asia Steven Tingay Swinburne University of Technology Next Generation correlator meeting, JIVE 27 -

A real-time software backend for the GMRT : towards hybrid backends CASPER meeting Capetown 30th September 2009 Collaborators : Jayanta Roy (NCRA) Yashwant.

Prof. Steven Tingay (ICRAR, Curtin University) Workshop on East-Asian Collaboration on the SKA Daejeon, Korea, November 30 – December 2, 2011 A long baseline.

The DiFX software correlator DiFX is an FX style correlator written in C++ Designed to run on commodity clusters Optimised Intel vector libraries are used.

Mihai Lucian Cristea, on behalf of SCARIe team University of Amsterdam TERENA CONFERENCE ‘10, Vilnius, 1 June 2010.

E. Momjian, T. Ghosh, C. Salter, & A. Venkataraman (NAIC-Arecibo Observatory) eVLBI with the 305 m Arecibo Radio Telescope ABSTRACT Using the newly acquired.

Development of DiFX Adam Deller NRAO 3rd DiFX workshop, Curtin University, Perth.

Cross correlators are a highly computationally intensive part of any radio interferometer. Although the number of operations per data sample is small,

1 Masaki Hirabaru and Yasuhiro Koyama PFLDnet 2006 Febrary 2, 2006 International e-VLBI Experience.

Data-Acquisition and Transport – Looking Forward to 2010 and Beyond Definition of ‘Data-Acquisition and Transport’ Continuum of Transport Options Limitations.

NEXPReS Period 3 Overview WP8 FlexBuff High-Bandwidth, High-Capacity Networked Storage Ari Mujunen Aalto University Metsähovi Radio Observatory Finland.

SA1: second year overview Arpad Szomoru JIVE January 30EXPReS Board Meeting, Utrecht, the Netherlands: SA1Slide #2 Outline Accomplishments in 2007.

NEXPReS Period 3 Overview WP5 Cloud Correlation Arpad Szomoru, JIVE.

EXPReS FABRIC meeting at Poznan, Poland1 EXPReS FABRIC WP 2.2 Correlator Engine Meeting Poznan Poland JIVE, Ruud Oerlemans.

TOG, Torun, April 2009 Arpad Szomoru, JIVE EVN/JIVE Technical developments.

VLBI Developments in Australia ASTRONOMY AND SPACE SCIENCE Chris Phillips| LBA Lead Scientist 11 November 2014.

WIDAR Correlator Options and Potential Craig Walker NRAO Socorro U.S. VLBI Technical Coordination Meeting May 14-15, 2007.

2007 / 5 / 141 VLBA Requirements, Plans & Status Jon Romney NRAO, Socorro Second US VLBI Technical Coordination Meeting Socorro 2007 / 5 / 14.

Korea Astronomy and Space Science Institute

Presentation transcript:

Distributed FX software correlation Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF) Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

28/09/2006Torun, Poland Outline Software vs hardware correlators Why software (what it does best, and the price you pay) Current applications The DiFX architecture Performance and current status Conclusions Software vs hardware correlators Why software (what it does best, and the price you pay) Current applications The DiFX architecture Performance and current status Conclusions

28/09/2006Torun, Poland Software vs Hardware “Software correlator”: program running on a supercomputer/cluster Software is unclocked, could be faster or slower than real-time No channel/integration time restrictions Floating pt vs int calculations “Software correlator”: program running on a supercomputer/cluster Software is unclocked, could be faster or slower than real-time No channel/integration time restrictions Floating pt vs int calculations

28/09/2006Torun, Poland Why software Flexibility - can do science that is impossible with hardware correlators Rapid (and cheap) development Add-ons MUCH easier in software Compatibility and expandability In general, less approximations due to the use of floating point Flexibility - can do science that is impossible with hardware correlators Rapid (and cheap) development Add-ons MUCH easier in software Compatibility and expandability In general, less approximations due to the use of floating point

28/09/2006Torun, Poland The price you pay Because software & hardware not specifically tuned, less computation per $$ hardware Therefore not useful for EVLA/SKA scale correlators Running costs (electricity, aircon) may be higher Because software & hardware not specifically tuned, less computation per $$ hardware Therefore not useful for EVLA/SKA scale correlators Running costs (electricity, aircon) may be higher

28/09/2006Torun, Poland Current applications/results New science the software correlator has enabled: –Disk-based LBA (greater bandwidth) –Adding global disk-based antennas to LBA experiments –High time and frequency resolution allowing wide-field imaging –VERY high frequency resolution for pulsar scintillation studies New science the software correlator has enabled: –Disk-based LBA (greater bandwidth) –Adding global disk-based antennas to LBA experiments –High time and frequency resolution allowing wide-field imaging –VERY high frequency resolution for pulsar scintillation studies

28/09/2006Torun, Poland The DiFX architecture Master Node Core 1 DataStream 1 DataStream 2 DataStream N Core 2 Core M … … … … Timerange, destination Baseband data Visibilities Source data MPI is used for inter-process communications

28/09/2006Torun, Poland Usage in the LBA PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas) Data is recorded directly to Linux- formatted storage (Apple Xraid) Disks shipped to Swinburne Correlated at the Swinburne supercomputer (~300 P4 machines ) Ftp fringe tests and 256 Mbps eVLBI PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas) Data is recorded directly to Linux- formatted storage (Apple Xraid) Disks shipped to Swinburne Correlated at the Swinburne supercomputer (~300 P4 machines ) Ftp fringe tests and 256 Mbps eVLBI

28/09/2006Torun, Poland Performance Real-time 1 Gbps (6 stations x 4x64 MHz bands): CPUs (circa 2004 P4s)

28/09/2006Torun, Poland Current status Verification via correlator comparison with VLBA completed Used in production capacity with LBA - complete switch as soon as sufficient disks available Further development (graphical frontend, FITS-IDI etc) ongoing Exploring potential for geodesy Verification via correlator comparison with VLBA completed Used in production capacity with LBA - complete switch as soon as sufficient disks available Further development (graphical frontend, FITS-IDI etc) ongoing Exploring potential for geodesy

28/09/2006Torun, Poland Conclusions Software correlators can be rapidly developed and enable science that cannot be done elsewhere Perfect for niche/part-time experiments and feasibility tests Used successfully with the LBA Early science results encouraging! Software correlators can be rapidly developed and enable science that cannot be done elsewhere Perfect for niche/part-time experiments and feasibility tests Used successfully with the LBA Early science results encouraging!

28/09/2006Torun, Poland