1. e-VLBI Development at Haystack Observatory Alan Whitney Chet Ruszczyk MIT Haystack Observatory 10 Jan 2006 IVS General Meeting Concepion, Chile

2. Scientific Advantages of e-VLBI Bandwidth growth potential for higher sensitivity –VLBI sensitivity (SNR) proportional to square root of Bandwidth resulting in a large increase in number of observable objects (only alternative is bigger antennas – hugely expensive) –e-VLBI bandwidth potential growth far exceeds recording capability (practical recordable data rate limited to ~1 Gbps) –e-VLBI is a key technology to realize goals of VLBI2010 Rapid processing turnaround –Astronomy Ability to study transient phenomena with feedback to steer observations –Geodesy Higher-precision measurements for geophysical investigations Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation

3. Highlights of recent e-VLBI developments August 2004: –Network link to Haystack upgraded to 2.5 Gbps –Real-time fringes at 128 Mbps, Westford and GGAO antennas, Haystack Correlator February 2005 –Real-time fringes Westford-Onsala at 256 Mbps –Used optically-switched light paths over part of route Starting April 2005 –Start routine e-VLBI transfers from Tsukuba and Kashima Starting ~June 2005 –Automated regular e-VLBI UT1 Intensive data transfers from Wettzell September 2005 –CONT05 data from Tsukuba transferred to Haystack via e-VLBI Fall 2005 –Effort initiated to connect NyAlesund to Haystack November 2005 –Global real-time e-VLBI demos at 512 Mbps

4. Challenges Network bottlenecks well below advertised rates Performance of transport protocols –untuned TCP stacks, fundamental limits of regular TCP Throughput limitations of COTS hardware –Disk-I/O - Network Complexity of e-VLBI experiments –e-VLBI experiments currently require significant network expertise to conduct Time-varying nature of network Standard formats for transfer of data and control information between different VLBI systems ‘Last-mile’ connectivity to telescopes –Most telescopes are deliberately placed in remote areas –Extensive initiatives in Europe and Japan to connect; U.S. is lagging

5. Current Projects at Haystack Observatory Network interfacing equipment for e-VLBI –Mark 5 VLBI data system Standardization (VSI-E) Intelligent Applications –Automation of e-VLBI transfers an ongoing process –Development of optimization-based algorithms for intelligent applications ongoing (EGAE) –Intelligent optically-switched networks (DRAGON) e-VLBI test experiments Production e-VLBI –Put e-VLBI into routine use – progressing well in limited venues Support e-VLBI development for VLBI2010 initiative

6. Mark 5 and e-VLBI “Triangle of connectivity” allows flexibility in e-VLBI operations Achieving full 1024Mbps e-VLBI is a challenge with current generation motherboards and NIC cards In lab, Mark 5A back-to-back sustained transfers have been achieved at ~1200Mbps –High-end motherboard; careful choice of NIC cards –Dual aggregated GigE links –Careful tuning Robust e-VLBI at 512 Mbps is realizable over real networks New generation of motherboards and 10GigE NIC cards should allow routine e-VLBI at 1024Mbps Triangle of connectivity

7. VSI-E Goals: –Efficient transport mechanism –Standard protocols –Internet-friendly transport –Scalable Implementation –Ability to transport individual data-channel streams as individual packet streams –Ability to make use of multicasting to transport data and/or control information in an efficient manner could be used in the future for support of distributed correlation

8. VSI Model

9. RTP Architecture

10. VSI-E Status Beta version of VSI-E is being tested in transfers from Kashima to Haystack (has been a bit delayed by network problems); expect result soon Plan to submit VSI-E protocol for approval by Internet Engineering Task Force (IETF) as international standard after agreement within e-VLBI community and successful demonstrations

11. New Application-Layer Protocols for e-VLBI Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity New protocols are being developed which are tailored to e-VLBI characteristics; for example: –Can tolerate some loss of data (perhaps 1% or so) in many cases –Can tolerate delay in transmission of data in many cases EGAE allows user to specify a profile of experiment requirements, which are then translated into network requirements and strategies ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy developed at Haystack Observatory under NSF grant –Now used in some routine e-VLBI transfers

13. EGAE Progress ‘Production’ e-VLBI facility has been established at Haystack to support routine e-VLBI transfers EGAE is now supporting routine non-real-time e-VLBI data transfers from Tsukuba EGAE will soon be used for routine e-VLBI transfers from Wettzell and NyAlesund

14. (Early demo – supported by DARPA) Bossnet 1 Gbps e-VLBI demonstration experiment Oct 2002 (sustained rate of ~768 Mbps achieved non-real-time) Initial experiment Future

15. Real-time e-VLBI SC2004 Demo Bossnet DRAGON Haystack Westford Goddard GGAO Pittsburgh Convention Center 128 Mbps

16. Real-time e-VLBI SC05 Demo Nov 2005 Real-time transmission and processing of data from antennas in Westford, MA, Greenbelt, MD, and Onsala, Sweden at 512 Mbps/antenna All except Kashima equipped with Mark 5 data systems; Kashima uses Japanese K5, included via VSI-E Correlation results displayed in real-time at SC05 meeting

17. Progress towards routine e-VLBI April 2005 –Start routine e-VLBI transfer from Kashima and Tsukuba (>200Mbps) Starting ~June 2005 –Automated regular e-VLBI UT1 Intensive data transfers from Wettzell to ISI-E (disks hand-carried to USNO for correlation) –A few start-up problems, but now operating fairly smoothly Spring 2005 –Commitment to connect Hobart via optical fiber (schedule unknown) September 2005 –All CONT05 data from Tsukuba transferred to Haystack via e-VLBI (~15 TB!) –Also – all Syowa data transferred via e-VLBI from Japan to Haystack November 2005 –Project initiated to connect NyAlesund to Haystack through NASA/GSFC at up to 100Mbps –Expect routine e-VLBI from NyAlesund within a few months December 2006 –Funds secured to connect Forteleza at 2.5 Gbps

18. Scorecard of Antenna/Correlator Connectivity (geodetic sites shown in red) JIVE Correlator (6 x 1 Gbps) Haystack (2.5 Gbps) Westford, MA (10 Gbps to Haystack; 1 Gbps to outside world) Kashima, Japan (2.5 Gbps) Usuda, Japan (2.5 Gbps) Nobeyama, Japan (2.5 Gbps) Koganei, Japan (2.5 Gbps) Tsukuba, Japan (2.5 Gbps) GGAO, MD (1 Gbps) Onsala, Sweden (1 Gbps) Torun, Poland (1 Gbps) Westerbork, The Netherlands (1 Gbps) Medicina, Italy (1 Gbps) Jodrell Bank (1 Gbps) Arecibo, PR (155 Mbps) Wettzell, Germany (~30 Mbps) Kokee Park, HA (nominally ~30 Mbps, but currently disconnected) TIGO (~2 Mbps) In progress Hobart – agreement reached to install high-speed fiber NyAlesund – work in progress to provide 100Mbps link for e-VLBI data to Haystack Observatory Forteleza – funds secured for fiber connection at 2.5Gbps; early 2006 Metsahovi – 1Gbps in 2006

19. International e-VLBI workshops Held every year since 2002, rotating between U.S., Europe, Japan and Australia Successful e-VLBI workshop held in Sydney, Australia in July 2005 –Hosted by CSIRO/ATNF –60 attendees from 9 countries –Many up-to-date developments in e-VLBI and network research –e-VLBI Technical Working Group re-formed –Many thanks to all for a fine workshop Next e-VLBI workshop will be held at Haystack, probably in mid-Oct 2006 –YOU ARE INVITED!

20. History of e-VLBI 1977 – Canada/U.S.: Algonquin-NRAO, 20 Mb/sec real-time satellite link 1979 – U.S.: OVRO-Haystack, 1 Mb/station using 2400-baud modems 1995 – Japan: Keystone project; 4-telescope real-time at 256 Mbps 1999 – Europe: 1 Mb/station over Internet using ftp 2001 – Japan: 2-station 1 Gbps real-time over dedicated fiber links 2002 – U.S.: Westford-GGAO direct data transfer at 768 Mbps (Bossnet) 2004 – Europe: 4-telescope real-time at 64 Mbps, including Arecibo, PR 2004 – U.S.: 2-telescope real-time at 512 Mbps; 700km baseline 2005 – Europe: 2-telescope real-time at 256 Mbps within Europe 2005 – Europe: 5-station real-time at 64 Mbps, including Arecibo, PR 2005 – U.S.: 3-telescope real-time at 512 Mbps; transoceanic