e-VLBI: Connecting the World’s Radio Telescopes with High-Speed Networks Alan R. Whitney MIT Haystack Observatory Westford, Massachusetts, USA

Traditional VLBI The Very-Long Baseline Interferometry (VLBI) Technique (with traditional data recording on magnetic tape or disk) The Global VLBI Array (up to ~20 stations can be used simultaneously)

Science Missions –Development of the very early universe –Formation and evolution of galaxies –Black holes in galactic cores –Stellar nurseries –Evolution and formation of elements VLBI for Astronomy

Combines data from global telescope array to synthesize a very large, distributed antenna Highest resolution technique available to astronomers – tens of microarcseconds (corresponds to resolving dimples on a golf ball at distance 5000 km!) Allows detailed studies of the most distant objects in the Universe Data is derived from national and international array of large radio telescopes Disadvantage: Very large amount of data and huge amount of computing required to make a single image VLBI for Astronomy

VLBI astronomy example

Galaxy NGC6251 Distance: ~500Mlight-yrs 1 parsec = light-yrs = ~0.1arcseconds Allows observations to focus in on energetic core

Resolution: ~10 marcsec

VLBI astronomy example Fundamental VLBI measurement is time-of-arrival difference of signals between telescopes in array; typical single-measurement precision is <10 picosec Highest precision (few mm) technique available for global tectonic measurements Highest precision Earth Orientation and Earth Rotation measurements Earth-rotation measurements are important for military/civilian navigation Stable reference frame formed by distant quasars Fundamental calibration for GPS constellation within Celestial Reference Frame Highest spatial and time resolution of Earth’s motion in space for the study of Earth’s interior International observing program includes ~35 stations around the world, coordinated by International VLBI Service In South American, stations in Concepcion, Chile and Fortaleza, Brazil are regular contributors VLBI for Geodesy

VLBI astronomy example Plate Tectonic Motion from VLBI measurements

VLBI astronomy example Correlation between Earth Rotation and Atmospheric Angular Momemtum Conclusion: Primary driver of variations in Earth rotation rate is weather!

Disadvantages of Traditional ‘Record & Ship’ VLBI Long interval between data collection and results (typically weeks to months) Uncertainty of proper equipment operation during experiment Expensive media (tape, disks) sometimes lost or damaged in transit Limited bandwidth limits sensitivity (few Gbps per station is practical maximum)

A Little History: Mark 4 VLBI data system

16-station VLBI correlator at JIVE in The Netherlands

Enter ‘e-VLBI’ – Electronic transmission of VLBI data over high-speed global networks!

e-VLBI is not new! 1979 – OVRO-Haystack 1 Mb/station; 2400-baud modems (Mark III) 1980’s – JPL DSN for EOP; 500 kb/s sample rate buffered to disc, 56 kbps transfer over land lines 1995 – Japanese Keystone project; 256 Mbps over dedicated fiber 1999 – Europe; 1 Mb/station over Internet using ftp Japan develops 1 Gbps e-VLBI system within Japan over dedicated links 1977 – NRAO-Algonquin – 20 Mb/sec real-time satellite link But the advent in global high-speed networks is completing changing the game!

Scientific Advantages of e-VLBI Bandwidth growth potential for higher sensitivity –VLBI sensitivity (SNR) proportional to square root of Bandwidth resulting in a large increase in number of observable objects (only alternative is bigger antennas – hugely expensive) –e-VLBI bandwidth potential growth far exceeds recording capability (practical recordable data rate limited to few Gbps) Rapid processing turnaround –Astronomy Ability to study transient phenomena with feedback to steer observations –Geodesy Higher-precision measurements for geophysical investigations Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation

Practical Advantages of e-VLBI Increased Reliability –remove recording equipment out of field –remote performance monitor & control capability in near real-time Lower Cost –Automated Operation Possible eliminates manual handling and shipping of storage media –Real-time or near-real-time Processing forestalls growth of storage-capacity requirements with bandwidth growth –Elimination of recording-media pool (millions of $’s!) Avoid unexpected media-shipping interruptions and losses

Elements of e-VLBI Development Develop e-VLBI-compatible data system –Mark 5 system developed at MIT Haystack Observatory with support from several national and international institutions Demonstrate e-VLBI in feasibility experiment –~700 km link between Haystack Observatory and NASA/GSFC Develop specialized e-VLBI data-transport formats and protocols –Develop international standard for e-VLBI data format –New IP-based protocol tailored to operate in shared-network ‘background’ to efficiently using available bandwidth Extend e-VLBI to national and global VLBI community

Elements of e-VLBI Development Develop e-VLBI-compatible data system Demonstrate e-VLBI in feasibility experiment Develop specialized e-VLBI data-transport formats and protocols Extend e-VLBI to national and global VLBI community

Mark 5 VLBI Disk-Based Data System 1 Gbps continuous recording/playback to/from set of 8 inexpensive (ATA) disks Developed at MIT Haystack Observatory with multi-institutional support Mostly COTS components Two removable ‘8-pack’ disk modules in single 5U chassis With currently available 250GB disks – capacity of single ‘8-pack’ 2.0TB; expected to increase to 3.0TB by early 2005 at cost of ~$0.5/GB GigE connection for real-time and quasi-real-time e-VLBI operations Inexpensive: <$20K ~75 Mark 5 systems now installed at stations and correlators

Elements of e-VLBI Development Develop e-VLBI-compatible data system Demonstrate e-VLBI in feasibility experiment Develop specialized e-VLBI data-transport formats and protocols Extend e-VLBI to national and global VLBI community

Bossnet 1 Gbps e-VLBI demonstration experiment (Fall 2002) 788 Mbps e-VLBI transfer achieved over shared IP infrastructure, but took much tuning Full report at Initial experiment Future ~700 km

Westford-GGAO e-VLBI results First near-real-time e-VLBI experiment demonstrated on 6 Oct 02 –Recorded data at 1152 Mbps on Westford-GGAO baseline –GGAO disk-to-disk transfer at average 788 Mbps transfer rate Direct data transfer experiment demonstrated on 24 Oct 02 –Direct transfer of GGAO data to disk at Haystack at 256 Mbps –Immediate correlation with Westford data –Nominal fringes observed Conclusion –e-VLBI at near Gbps speeds over ordinary shared networks is possible but still difficult

Elements of e-VLBI Development Develop e-VLBI-compatible data system Demonstrate e-VLBI in feasibility experiment Develop specialized e-VLBI data-transport formats and protocols Extend e-VLBI to national and global VLBI community

VSI-E VSI-E = VLBI Standard Interface for e-VLBI Follows on heels of VSI-H and VSI-S specifications over last 3 years Goal is to allow compatible interchangeable of data between heterogeneous VLBI data sytems VSI-E currently under development by international VSI committee RTP protocol has been chosen Draft specification currently under discussion. Goal: Complete VSI-E specification by mid-2004 Prototype software will be available June 04

RTP Capabilities RTP provides an Internet-standard format for: –Transmission of sampled analog data –Dissemination of session information –Monitoring of network and end system performance (by participants and third parties) –Adaptation to varying network capability / performance –Message Sequencing / reordering –Multi-cast distribution of statistics, control and data RTP allows the reuse of many standard monitoring / analysis tools RTP seen as internet-friendly by the network community: –attention to efficiency protocol designed to have minimum overhead for in-band data –attention to resource constraints won't use up all your bandwidth with control information –attention to scaling issues RTCP Capabilities Monitors network’s real-time data delivery performance Statistics collected from receivers Information delivered to –Senders (adapt to prevailing conditions) –Network management (identifies faults, provisioning problems) Adaptive, bandwidth-limited design

Possible VSI-E Topologies

New Application-Layer Protocols for e-VLBI Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity New protocols are being developed which are tailored to e-VLBI characteristics; for example: –Can tolerate some loss of data (perhaps 1% or so) in many cases –Can tolerate delay in transmission of data in many cases ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy being developed at Haystack Observatory under 3-year NSF grant: –Will ‘scavenge’ and use ‘secondary’ bandwidth –‘Less than best effort’ service will not interfere with high-priority users –Dr. David Lapsley has joined Haystack staff to lead this effort

Typical bit-rate statistics on Abilene network Conclusion: Average network usage is only a few % of capacity Usage >20Mbps less than 1% of the time Mbps

Typical distribution of heavy traffic on Abilene Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes secs <10% of ‘bulk’ transfers exceed ~100 secs

New Transport Protocols for e-VLBI TCP is very inefficient is there are virtually any losses on a network due either to network sharing or due to physical packet losses –e-VLBI is particularly sensitive due to typical long RTT’s UDP can used, but is sometimes ‘unfriendly’ to other users New transmission protocols are being developed which are much more aggressive, but fair – some examples: –FAST (Caltech) –Tsunami (Indiana University) –UDT (UDP-based Data Transfer; Univ. of Illinois) –High-speed TCP (Sally Floyd) –XCP (Explicit Control Protocol; Dina Katabi, MIT)

Acceptable Latencies for e-VBLI Latency delays must not exceed the ability of the correlator to buffer the data Near real-time operation –At 1 Gbps, disk buffers of large size can be used for; in this case latency is not a real issue. –At 10 Gbps, disk buffers are not economically feasible and will require real-time correlation Real-time operation –At 1 Gbps, a buffer size of ~0.5GB is currently available from each station; a latency delay of up to ~2 seconds is acceptable –At 10 Gbps, it is likely that larger buffers will need to be built to accommodate latency delays up to at least 1 second Optically switched networks with dedicated wavelengths may eventually put the latency issue to rest as high-performance applications are able to use dedicated-lambda facilities.

Elements of e-VLBI Development Develop e-VLBI-compatible data system Demonstrate e-VLBI in feasibility experiment Develop specialized e-VLBI data-transport formats and protocols Extend e-VLBI to national and global VLBI community

Westford-to-Kashima e-VLBI experiments First Westford/Kashima experiment conducted on 15 Oct 02 –Data recorded on K5 at Kashima and Mark 5 at Westford at 256 Mbps –Files exchanged over Abilene/GEMnet networks Nominal speed expected to be ~20 Mbps, but achieved <2 Mbps for unknown reasons - investigating –File formats software translated –Correlation on Mark 4 correlator at Haystack and PC Software correlator at Kashima –Nominal fringes obtained Kashima data is now regularly transmitted to Haystack for processing

Westford to Sweden ‘Real-time’ experiments ‘Real-time’ experiments conducted in March/April 2004 –Data transmitted and processed in real-time (i.e. no disk buffering); data transmitted directly from stations to correlator –First experiment at 32 Mbps due to temporary low-speed connection to Haystack –Plan to extend these experiments to at least 512 Mbps with GigE links

UT1 ‘Intensive’ e-VLBI Daily ~1 hour VLBI sessions between Kokee Park, Hawaii and Wettzell, Germany are used for UT1 measurements Data are time sensitive since they are used for predicting UT1 Currently requires ~4 day turnaround shipping media These measurements are an ideal candidate for routine e-VLBI –Short daily session collect <100 GB of data –Even 100 Mbps will allow transfer in a few hours Work now in progress to make necessary connections –Network being organized from Kokee Park to USNO; connection speed OC-3 –Data from Mark 5 system in Wettzell will be carried over dedicated fiber at ~30Mbps to Univ. of Regensberg, then to GEANT

First e-VLBI to South America! First e-VLBI data transmitted from TIGO in Concepion, Chile to Haystack Observatory in April 2004 Two scans of ~1.5GB each were transmitted at an average rate of Mbps Hope to continue this effort and upgrade the link!

A sample of international connections Possibilities for international connections –Surfnet – U.S. to Europe at 10 Gbps –TransPAC/APAN – U.S. to Japan at 2.5 Gbps x2 –GEMnet – U.S. to Japan at 2.5 Gbps (privately operated by NEC) –AMPATH – connections to telescopes in Chile and Brazil –AARNET – Recently announced 2x10 Gbps connections to Hawaii and U.S. –IEEAF –Europe/U.S./Japan link at 10 Gbps –A sample of others under construction GLORIAD – connecting China and Russis TEIN – Paris to Seoul EUMEDCONNECT – Europe to Mediterranean NeDAP – Europe to Russia ALIS – Europe to Latin and South America –And many others………..

622 Mbps +10 Gbps Transoceanic donations to IEEAF (in red) Credit: IEEAF

TransPAC Network (planned upgrade in 2004 to 2xGigE plus OC-48) Credit: J. Williams, IU

AMPATH: Research and Education Network and International Exchange Point for the Americas  Launched in March 2000 as a project led by Florida International University (FIU), with industry support from Global Crossing (GX), Cisco Systems, Lucent Technologies, Juniper Networks and Terremark Worldwide  Enables wide-bandwidth digital communications between the Abilene network and 10 National Research and Education Networks (NRNs) in South and Central America, the Caribbean and Mexico  Provides connectivity to US research programs in the region  AMPATH is a project of FIU and the National Science Foundation’s Advanced Networking Infrastructure & Research (ANIR) Division Note: VLBI telescopes currently in Chile and Brazil Credit: Julio Ibarra, FIU

0UKUK FRFR CHCH SESE ITITRORO HRHR EEEELVLVLTLT IEIE NLNL BEBE DEDE PLPL CZCZ HUHUATAT SISI SKSK LULU ESES PTPT GRGR BGBG CYCYILIL Gbps 10 Gbps 2.5 Gbps 2.5 Gbps SE - PoP for Nordunet GÉANT: The connectivity at 10 Gbps

Aarnet: SXTransport Project in 2004  Connect Major Australian Universities to 10 Gbps Backbone  Two 10 Gbps Research Links to the US  Aarnet/USLIC Collaboration on Net R&D Starting Now Credit: George McLaughlin, AARNET

GLORIAD: Global Optical Ring (US-Ru-Cn) “ Little Gloriad ” (OC3) Launched January 12; to OC192 in 2004

Abilene - Upgrade Completed! Credit: Internet2

National Light Rail Footprint Terminal, Regen or OADM site Fiber route NLR  Starting Up Now  Initially 4x10 Gb Wavelengths  Future: to 40x10Gb Waves  Transition beginning now to optical, multi-wavelength R&E networks.  Also Note: XWIN (Germany); IEEAF/GEO plan for dark fiber in Europe PIT POR FRE RAL WAL NAS PHO OLG ATL CHI CLE KAN OGD SAC BOS NYC WDC STR DAL DEN LAX SVL SEA SDG JAC

What is the future of e-VLBI? Global connectivity is increasing at a very rapid rate e-VLBI is being aggressively developed in the U.S., Europe and Japan and will likely become standard procedure within the next decade. In South America, the ALMA project in Chile will be a large driver for high-bandwidth communications to the rest of the world.

Bandwidth Growth of Int’l HENP Networks (US-CERN Example)  Rate of Progress >> Moore’s Law. (US-CERN Example)  9.6 kbps Analog (1985)  kbps Digital ( ) [X 7 – 27]  1.5 Mbps Shared (1990-3; IBM) [X 160]  2 -4 Mbps ( ) [X ]  Mbps ( ) [X 1.2k-2k]  Mbps (2001-2) [X 16k – 32k]  622 Mbps (2002-3) [X 65k]  2.5 Gbps (2003-4) [X 250k]  10 Gbps (2005) [X 1M]  A factor of ~1M over a period of (a factor of ~5k during )  HENP has become a leading applications driver, and also a co-developer of global networks; Credit: Harvey Newman, Caltech

What are the problems? Biggest problem: ‘Last-mile’ connection of telescopes –Most telescopes are deliberately built in remote locations –Biggest single obstacle is physical cost of laying fiber –Cost of terminal equipment is rapidly diminishing, even for 10 Gbps –A concerted international effort must be made to connect every major telescope in the world to high-speed network Some countries require buying services from service providers, which is very expensive; lighting dark fiber is far cheaper, if possible

e-VLBI is one among many applications HENP – high-energy physics community is currently heaviest user of international networks, mainly in dissemination of very large data files from major HENP facilities in Europe and U.S. Astronomy in general – optical telescopes are now sending high- resolution images in real-time to remote observers around the world; NVO will be among the world’s largest distributed database Education – tremendous opportunities here –Remote interactive learning from world experts in all fields –Demonstration of advanced surgical procedures –Easy international collaboration for all disciplines –Master classes in the performing arts, with remote students having access to the world’s great performers and teachers –Access to global databases and information of all kinds will help level the playing field between rich and poor nations.

Summary of Impact of e-VLBI Program Opens new doors for national and international astronomical and geophysical research. Represents an excellent match between modern Information Technology and a real science need. Motivates the development of a new shared-network protocols that will benefit other similar applications. Drives an innovative IT research application and fosters a strong international science collaboration.

The End Thank you!