AMPATH, FIU, Miami, 31 January 2003 eVLBI Research in the European VLBI Network Steve Parsley Joint Institute for VLBI in Europe Dwingeloo, Netherlands

AMPATH, FIU, Miami, 31 January 2003 Outline European VLBI Network (EVN) European Research Networks Fibre Connection to Dwingeloo iGrid Demonstration EVN-NREN: Proof of Concept Project

AMPATH, FIU, Miami, 31 January 2003

The European VLBI Network

EVN Telescopes

AMPATH, FIU, Miami, 31 January 2003 Correlator

AMPATH, FIU, Miami, 31 January 2003

European Research Networks National NREN = National Research and Education Network UKERNA, SURFnet, GARR, DFN, PSNC, NORDUnet ….. International GÉANT = Pan-European Research Network

0 SISI GRGR CZCZ IEIE LULU EEEE SKSK ILIL LTLTCYCY ESES HUHU ITIT FRFR UKUK BEBE NLNL DEDE SESE CHCH PTPT ATAT GEANT GEANT 2.5 G 1.2 G 622M 155 M BGBG LVLV RORO HRHR PLPL 45 M 34 M 310 M GÉANT: Access of NRENs to GÉANT EVN telescope

AMPATH, FIU, Miami, 31 January UKUK FRFR CHCH SESE ITITRORO HRHR EEEELVLVLTLT IEIE NLNL BEBE DEDE PLPL CZCZ HUHUATAT SISI SKSK LULU ESES PTPT GRGR BGBG CYCYILIL Gbps 10 Gbps 2.5 Gbps 2.5 Gbps SE - PoP for Nordunet GÉANT: The connectivity at 10 Gbps

3 from Dwingeloo to Amsterdam Internet Exchange 20 km of new fibre 180km to SARA via installed link Dwingeloo

AMPATH, FIU, Miami, 31 January 2003 Network SARA – ASTRON / JIVE GXSARAASTRON / JIVE Access to NetherLight Access to SURFnet5 (ar5.amsterdam1) GbE Project eVLBI GbE ISDN Cisco2620SNET-LAN 4912G Cisco1721 Cisco15216 EDFA (2x) Cisco15252 DWDM Cisco15252 DWDM RS 232 GbE ODF Amsterdam1Emmeloord1Dwingeloo1 GbE ASTRON router

AMPATH, FIU, Miami, 31 January 2003 iGrid Demonstration September 2002 Amsterdam Science and Technology Centre (WTCW) The Netherlands Grid 2 oo (COMING SOON) The International Virtual Laboratory Call for Applications with Insatiable Bandwidth Appetites! “We hereby challenge the international research community to demonstrate applications that benefit from huge amounts of bandwidth!” i

AMPATH, FIU, Miami, 31 January 2003 Westerbork Jodrell Bank SARA Dwingeloo Shared IP over GÉANT GEthernet over SURFnet Lambda Future fibre Link (now it’s a truck) Live telescope pictures Image Fringe Display JIVE Correlator About VLBI Other VLBI collaborators Science World of VLBI VLBI Technique & Operations vlb

AMPATH, FIU, Miami, 31 January 2003 vlbiGrid Objectives Trial use of UDP for VLBI data transfer across production research networks: Data rates Packet loss Packet reordering Exercise new link to JIVE Demonstrate eVLBI fringes Publicise VLBI to network researchers

AMPATH, FIU, Miami, 31 January 2003 iGrid Network Routing Geant Core SuperJANET4 Core Network NNW MAN Manchester Server DWDM link SURFnet Core iGrid2002 Amsterdam 1 Gbit Eth. 2.5 Gbit Pos 10 Gbit PCEVN

Data Transfer Process: Raid0 Disc UDP Data Raid0 Disc RingBuffer TCP Control Web Interface

Receiving Packets Time Incoming packet ids Move 6 Continue with 7 Packets have Header with packet id + Data Put header directly into control area Put data directly into ring buffer No extra copy Assume UDP packets arrive in order none lost Deposit data into next consecutive slot Inspect Header Move data onward to correct location Record: The received inter-packet spacing 1-way delay

AMPATH, FIU, Miami, 31 January 2003 iGrid Results UDP across GÉANT (server-server) : ~500Mbit/s <6E-5 lost packets <1E-5 reordered packets (all exchanges) FTP to Dwingeloo: ~40Mbit/s Normal fringes detected PR: Great success, new link used three days after first light Much interest in VLBI data rates and volumes

AMPATH, FIU, Miami, 31 January 2003 iGrid Lessons Learned 500Mbit/s VLBI data transfer on the production network using a simple UDP based protocol is feasible. Acceptable packet loss is achievable. Packets may be re-ordered. A simple data moving strategy corrects for this. Control/monitoring data should be on a separate, reliable channel. End hosts must have sufficient power in both compute cycles and input/out capability. An initial specification would include: 64bit 66MHz PCI bus 2 GHz processor Gigabit Ethernet interface: Intel Pro 1000 SysKonnect Fast disks with high speed buses Maxtor D740 EIDE 133

AMPATH, FIU, Miami, 31 January 2003 Proof of Concept Project (PoC) Target: Up to 5 radio telescope sites (not incl NL) connected in real-time to JIVE correlator Duration: Up to end of GÉANT contract (Nov 2004) Support: Best effort IP service transiting NRENs and GÉANT No significant upgrades to GÉANT (initially) Support 512M and 1G real time modes of operation Use existing NREN access ports Limited resilience

AMPATH, FIU, Miami, 31 January 2003 Proof of Concept Participants DANTE Pan-European Network GARR Italian NREN UKERNAUK NREN PSNC Polish NREN DFN German NREN SURFnetDutch NREN KTHNOC/NORDUnet Nordic NREN Manchester UniversityNetwork application software JIVEEVN Correlator Westerbork telescopeNetherlands Onsala Space Observatory Sweden MPIfRGermany Jodrell Bank UK TCfAPoland CNR IRA Italy

AMPATH, FIU, Miami, 31 January 2003 Success Criteria (for PoC) NREN/GÉANT point of view: Connect at least three sites to JIVE Observe significant usage (time and BW) EVN point of view: Same-day imaging of 12hour, 1Gb/s observation Real time network verification of transient phenomenon

AMPATH, FIU, Miami, 31 January 2003 Traffic Engineering on GÉANT...

GÉANT Nodes Affected by EVN Traffic UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #1 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #1 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #2 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #2 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Potential Upgrades to GÉANT UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #3 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #3 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #4 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

Failure Scenarios #4 UK SE FR NL BE DE1 DE2 CZ PL CH IT AT SURFnetJANETGARRPSNCDFN NORDUnet 2.5G 10G

AMPATH, FIU, Miami, 31 January 2003 Some Technical Issues Packet re-ordering due to architecture of Juniper M160 routers in GÉANT backbone. VLBI UDP application must fix this. VLBI data rates are restricted to “octaves” : - 256M, 512M, 1G “1Gbps” eVLBI flows are >1Gbps  two GE channels will be needed to support "1Gbps" from each telescope.

AMPATH, FIU, Miami, 31 January 2003 SURFnet GÉANT NREN1 NREN2 NREN3 IP1IP2IP3IP4 JIVE

AMPATH, FIU, Miami, 31 January 2003 PoC Design NREN SURFnet GE JIVE Corr. Separate, GE point-to-point connection for each telescope GÉANT GE link on /30 subnet between VLAN interface on the SURFnet access router and the GE NIC at Dwingeloo. Add second GE NIC for “1Gb/s” each carrying 512Mb/s

AMPATH, FIU, Miami, 31 January 2003 Local Loops - Status Italy: GARR : Backbone capacity - 2.5Gbps. Medicina radio telescope: 30km from Bologna POP. Good connectivity (dark fibre) expected by end of (3-4km dig to radio telescope.) Poland: PSNC: 10GE link from Torun to Poznan (location of GÉANT POP) ready in June Torun radio telescope: 15km, Torun to telescope in place. 100m and terminal equipment within 3 months. Germany: DFN: No backbone problems expected. Bonn ring on-line April’ km connection to DFN POP needed but not planned. Effelsberg Radio Telescope: Fibre to telescope in place bar 5km copper section. UK: UKERNA: no problems to support the eVLBI traffic on the SuperJANET backbone. 2.5G backup access port to GÉANT planned. Jodrell Bank: significant problem with local loop to JB (BT quoted £1M for the first year) Options being explored include “piggy-back” on eMERLIN, 30Gbit/s infrastructure ~June 2003 Sweden: NORDUnet: two, redundant 2.5Gbit/s links to GEANT in Stockholm available. SUNET: 10Gbit/s backbone with 2X2.5Gbit/sSRP links between Chalmer University and Stockholm. Onsala Space Observatory: 40km of fibre required. Installation costs of up to €150k have been quoted with €20k+ annual charges. The Netherlands: SURFnet: 2.5Gbit/s border connection will soon be upgraded to 10Gbit/s JIVE: 190km dark fibre link from JIVE to Amsterdam (SARA) in place. Westerbork: 24 fibre link directly to Dwingeloo by June‘03

AMPATH, FIU, Miami, 31 January 2003 Project Timeline

AMPATH, FIU, Miami, 31 January 2003 SURFnet GÉANT NREN1 NREN2 NREN3 IP1IP2IP3IP4 JIVE

AMPATH, FIU, Miami, 31 January 2003 UKLight Proposal Initiative to attract funding to create: UK point-of-access (PoA) to the international test-bed Intra UK optical test-bed Draft projects: Development of a common optical control plane Secure Optical Community Services A project to use a dark fibre infrastructure An applications demonstration based upon very long baseline interferometry (VLBI) for Radio Astronomy

AMPATH, FIU, Miami, 31 January 2003 Dedicated Lambda VLBI Network

AMPATH, FIU, Miami, 31 January nd eVLBI Workshop May 2003, Dwingeloo, Netherlands First Announcement10 th Jan ‘03 Second Announcement1 st March ‘03 Abstract Deadline19 th March ‘03 Final Announcement3 rd April ‘03 Registration Deadline20 th April ‘03 Hotel Deadline15 th April ‘03 Workshop15-16 th May ’03