High Rate Internet Data Transfer for eVLBI Ralph Spencer, Richard Hughes-Jones and Simon Casey The University of Manchester All Hands Sept 2005

Radio Astronomy The study of celestial objects at 1m wavelength. Sensitivity for continuum sources B=bandwidth,  integration time. High resolution achieved by interferometers. Some radio emitting X-ray binary stars in our own galaxy: GRS MERLIN SS433 MERLIN and European VLBI Cygnus X-1 VLBA

GRS : 15 solar mass BH in an X-ray binary: MERLIN observations receding 600 mas = 6000 A.U. at 10 kpc

Very Long Baseline Interferometry

How interferometry works Interferometer: signals from each telescope brought together coherently for correlation (multiplication) Resolution /D << that possible from a single telescope, e.g the pimples on a golf ball at 3000 miles at the highest resolution Output of the correlator is one component of the Fourier transform of the sky brightness  Dcos  X D

Earth-Rotation Synthesis. The baseline between each pair of telescopes forms an ellipse as the Earth rotates

The Aperture Plane (u,v) Need ~ 12 hours for full synthesis, not necessarily collecting data for all that time. Telescope data correlated in pairs: N(N-1)/2 baselines NBTrade-off between B and t for sensitivity ~sqrt(Bt) Data are calibrated, Fourier inverted and then deconvolved to give the images. EVN MERLIN

The European VLBI Network EVN Detailed radio imaging uses antenna networks over 100s-1000s km At faintest levels, sky teems with galaxies being formed Currently use disk recording at 512Mb/s (MkV) real-time connection allows greater –response –reliability –sensitivity –Need Internet eVLBI

Westerbork Netherlands Dedicate d Gbit link EVN-NREN Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo DWDM link Cambridge UK MERLIN Medicina Italy Chalmers University of Technolog y, Gothenbur g Torun Poland Gbit link

eVLBI Milestones January 2004: Disk buffered eVLBI session: –Three telescopes at 128Mb/s for first eVLBI image –On – Wb fringes at 256Mb/s April 2004: Three-telescope, real-time eVLBI session. –Fringes at 64Mb/s –First real-time EVN image - 32Mb/s. September 2004: Four telescope real-time eVLBI –Fringes to Torun and Arecibo –First EVN, eVLBI Science session January 2005: First “dedicated light-path” eVLBI –Australia to JIVE via Canarie: data from Huygens descent transferred at ~450Mb/s

20 December connection of JBO to Manchester by 2 x 1 GE eVLBI tests between Poland Sweden UK and Netherlands at 256 Mb/s February 2005 TCP and UDP memory – memory tests at rates up to 450 Mb/s (TCP) and 650 Mb/s (UDP) Tests showed inconsistencies between Red Hat kernels, rates of 128 Mb/s only obtained on 10 Feb Haystack (US) – Onsala (Sweden) runs at 256 Mb/s Regular tests with VLBI data every ~6 weeks 128 Mpbs OK, 256 Mpbs often, But not 512 Mbps – WHY NOT? Correlator can cope with large error rates up to 2 % but need high throughput for sensitivity implications for protocols

Throughput vs packet spacing Manchester: 2.0G Hz Xeon Dwingeloo: 1.2 GHz PIII Near wire rate, 950 Mbps UDPmon Packet loss CPU Kernel Load sender CPU Kernel Load receiver 4 th Year project Adam Mathews Steve O’Toole UDP Throughput Oct-Nov 2003 Manchester-Dwingeloo Production

January 2005 repeated UDP over 2 days

ESLEA Packet loss will cause low throughput in TCP/IP Congestion will result in routers drooping packets: use Switched Light Paths! Tests with MB-NG network Jan- JBO connected to JIVE via UKLight in June (thanks to John Graham, UKERNA) Comparison tests between UKLight connections JBO-JIVE and production (SJ4-Geant)

UKLight Switched light path

Tests on the UKLight switched light- path Manchester : Dwingeloo Throughput as a function of inter-packet spacing Packet loss Maximum size packets can reach full line rates with no loss, and there was no re- ordering (plot not shown).

Tests on the production network Manchester : Dwingeloo. Throughput Small (0.2%) packet loss was seen Re-ordering of packets was significant

UKLight using MkV terminals

The GÉANT2 Launch June 2005

Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Torun Poland e-VLBI at the GÉANT2 Launch Jun 2005

UDP Performance: 3 Flows on GÉANT Throughput: 5 Hour run 1500 byte MTU Jodrell: JIVE 2.0 GHz dual Xeon – 2.4 GHz dual Xeon Mbit/s Medicina (Bologna): JIVE 800 MHz PIII – Mk5 (623) 1.2 GHz PIII 330 Mbit/s limited by sending PC Torun: JIVE 2.4 GHz dual Xeon – Mk5 (575) 1.2 GHz PIII Mbit/s limited by security policing (>600Mbit/s  20 Mbit/s) ? Throughput: 50 min period  Period is ~17 min

18 Hour Flows on UKLight Jodrell – JIVE, 26 June 2005 Throughput: Jodrell: JIVE 2.4 GHz dual Xeon – 2.4 GHz dual Xeon Mbit/s Traffic through SURFnet Packet Loss –Only 3 groups with lost packets each –No packets lost the rest of the time Packet re-ordering –None

Conclusion and Future directions eVLBI needs to get to rates of 512 Mpbs – not there yet. Rates of 256 Mpbs on production networks possible now using TCP/IP in good conditions, but not higher rates. G2 launch showed continuous long term flows as in VLBI have different characteristics to those in short term tests (Iperf, UDPmon) : more investigation needed – collaborating with DANTE and SURFnet Packet loss is a killer for TCP – new protocols needed (ESLEA) - VSI-E standard The End Host Problem –The performance of Motherboards, NICs, RAID controllers and Disks matter: Mk V units (Conduant) – MkV-B 2 Gbps data recording in 2 yrs? Next generation correlator 10 Gbps in 5-10 yrs

Thanks to Paul Burgess JBO Steve Parsley, Arpad Szomoru, Cormac Reynolds JIVE Nicola Pezzi, John Graham, Colin Greenwood, Peter Clarke ESLEA EVN observatories staff