EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Recent eVLBI developments at JIVE Arpad Szomoru Joint Institute for VLBI in Europe

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Outline Introduction Protocols, tuning issues Tests and results ftp-based Dual buffered Single buffered Real-time Jumbo frames Network monitoring Centralised control

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Introduction Move to disk-based operations nearly complete More reliable, data quality Cheaper to maintain Ease of use Direct access compared Unattended operations eVLBI: the future? No consumables High bandwidth Fast turn-around Disk based recording eVLBI using fiber

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE for the EVN and JIVE Feasibility of eVLBI:- Costs, timescales, logistics. Standards:- Protocols, parameter tuning, procedures at telescope and correlator. New Capabilities:- Higher data rates, improved reliability, quicker response. for GÉANT and the NRENs To see significant network usage with multiple Gbit streams converging on JIVE. Minimum is three telescopes (not including Westerbork) Must be seen to enable new science and not just solve an existing data-transport problem Real-time operation is seen as the ultimate aim, buffered operation accepted as a development stage. POC targets:

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Fibre pair from Amsterdam Optical Splitters GE lines to correlator LX Optics converters Cisco ONS Correlator Interface Current status: JIVE: 6 lambdas via Netherlight, each capable of 1 Gbps 1 Gbps connections to Westerbork, Torun, Onsala, (Metsahovi?), 2.5 to Jodrell 155 Mbps to Arecibo Connection of Medicina at 1 Gbps planned later this year Current status: JIVE: 6 lambdas via Netherlight, each capable of 1 Gbps 1 Gbps connections to Westerbork, Torun, Onsala, (Metsahovi?), 2.5 to Jodrell 155 Mbps to Arecibo Connection of Medicina at 1 Gbps planned later this year

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Protocols Use of existing protocols, available hardware TCP: maximal reliability, very sensitive to congestion UDP: connectionless, fast(er) but not reliable (enough?) Different protocols will be tested in the near future (tsunami?)

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Tuning issues Dramatic improvement of TCP performance possible by adjusting default settings: congestion window: product of roundtrip time and bandwidth size of queues between kernel and NIC SACK (Selective Acknowledgment) implementation MTU size (jumbo frames) Interrupt moderation: cpu may be bottleneck

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Tuning issues Dramatic improvement of TCP performance possible by adjusting default settings: congestion window: product of roundtrip time and bandwidth size of queues between kernel and NIC SACK (Selective Acknowledgment) implementation MTU size (jumbo frames) Interrupt moderation: cpu may be bottleneck

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Mem-mem transfer via patch

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Disk2net2disk via patch, jumbo

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Disk2net-net2disk Dwingeloo-Bologna

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Network stress test (iperf)

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Iperf (TCP) 30s intervals

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Intel(R) SE7501BR2 Intel(R) 7501 chip set Dual Intel(R) Xeon (single, 2.8G, 512K L2 cash installed) Dual channel DDR266 registered SDRAM (512MB) Dual peer PCIX 64/100, 2+2 slots 32/33 PCI, 2 slots On-board TX1000 and T10/100 NICs On-board Ultra320 SCSI On-board PATA Mark 5 Motherboard Upgrade

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Memory-memoryDisk2net-net2disk In2net- net2disk In2net- net2out UDPTCPUDPTCP Bench via patch Idem, jumbo frames Bench via Amsterdam Idem, jumbo frames Westerbork-JIVE Idem, jumbo frames Bologna-JIVE Jodrell-JIVE Arecibo-JIVE8832 Torun-JIVE Onsala-JIVE JIVE-Haystack61271 Test results

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Memory-memoryDisk2net-net2disk In2net- net2disk In2net- net2out UDPTCPUDPTCP Bench via patch Idem, jumbo frames Bench via Amsterdam Idem, jumbo frames Westerbork-JIVE Idem, jumbo frames Bologna-JIVE Jodrell-JIVE Arecibo-JIVE8832 Torun-JIVE Onsala-JIVE JIVE-Haystack61271 Test results

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Huygens descent tracking Data recorded at Mopra and Parkes, flown to Sydney, transported via dedicated 1Gbps lightpath to Jive Peak transfer rate ~400Mbps Use of modified tsunami protocol “slow start” algorithm Transfer freezes occasionally Quite buggy (but not everybody thinks so..)

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE eVLBI Transfer modes ftp-based: bandwidth not critical, has greatly improved response to technical problems at telescopes Dual buffered: recorded on Mk5 diskpack, transferred through disk2net and net2disk, played back at correlator Single buffered: streamed directly from formatters to diskpacks at JIVE Real-time: directly from formatters to correlator without disk buffering

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE eVLBI Transfer modes ftp-based: bandwidth not critical, has greatly improved response to technical problems at telescopes Dual buffered: recorded on Mk5 diskpack, transferred through disk2net and net2disk, played back at correlator Single buffered: streamed directly from formatters to diskpacks at JIVE Real-time: directly from formatters to correlator without disk buffering First real-time eVLBI Image

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE MERLIN, March 2002 eVLBI, September 2004 IRC+10420

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Jumbo frames Tested with: Results inconclusive, but… Transfer between Mk5’s (bench) UDP: <500 Mbps TCP: >500 Mbps, ONLY with jumbo frames Other hardware platform needed?

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Network monitoring BWCTL installed at most stations More measuring points along the way needed Script-based tools for inspection and visualisation of state of network

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Centralised control New operating model for EVN; correlator as integral part of network

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE

Centralised control New operating model for EVN; correlator as integral part of network Monitoring tools at stations Reliability of correlator, stability of Mark5 units Choice of platform?

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Conclusions Mark5 not very well suited for eVLBI Hardware limitations (interrupt conflicts, outdated motherboards, proprietary hardware) Software limitation (stuck to kernel 2.4 because of Jungo drivers, proprietary software) Next platform? support for 4Gbps, PCI Xpres? as much off-the-shelf as possible as simple as possible (no disks!) Dedicated lightpaths…