High Bandwidth Data Acquisition and Network Streaming in VLBI Jan Wagner, Guifré Molera et al. TKK / Metsähovi Radio Observatory

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #2 EXPReS: funds to connect stations at 1G/10G 1G fiber up to capacity with Tsunami UDP (2006) or JIVE vlbi_udp (2007) real-time transport protocols ToDo: new system for >1 Gbps rates! Basic e-VLBI Data Acquisition Introduction Basic: 8–16 stations, rates 128–512 Mbit/s; 24h 1–4 TB/station, all to correlator VLBI: observed galaxy’s noise recorded to disks, shipped to correlator e-VLBI: transfer station data via fast research networks, two alternatives: post-observation: record locally, transfer to correlator later real-time: stream to the correlator for temp storage or processing 1Apr08, JIVE vlbi_udp incoming aggregate rate record

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #3 Goals for High(er) Bandwidth DAQ Observe a broader radio spectrum broader spectrum => higher data rate correlation result and final image improves! high EXPReS goal 1/2/4/8 Gbps per station Higher rates => need at least: new analog-to-digital converter system (current 10+ years old) faster, higher capacity local/remote data storage systems suitable transport protocols - stations to correlator Ultimately e-VLBI data processing in real-time get rid of high capacity station data storage JIVE: hardware correlator successful in many ≤ 512 Mbps real-time runs distributed software correlators more natural with data streaming Introduction C. Phillips, e-VLBI Scientific Benefits, ATNF CSIRO memo, 2004

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #4 High Bandwidth DAQ Development DAQ steps: Digitize wide-band signal – Stream over 10G – Capture/store with PC(s) Development Environment

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #5 DAQ – Digitizing UC Berkeley’s iBOB FPGA hardware platform Virtex2 Pro FPGA with PowerPC405 two CX4 10G ports 1..2 Berkeley iADC expansion boards Atmel AT84AD001 two channels, 8-bit, 1 Gsample/s => max 8+8 Gbps from one iADC => 32 Gbps ADC data, 20 Gbps net Semi-ready component libraries from Berkeley for “faster” start of development Analog signal digitizing setup: iBOB FPGA reads raw 2-channel iADC data FPGA processes data, implements network streaming => … Basic Data Acquisition System

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #6 DAQ – Streaming TCP is not a good streaming choice No unreliable best-effort delivery, no delivery time constraints Real-time VLBI stream requirements: data loss f≤10% is tolerable (impact is ), occasional re-order ok data integrity non-critical: BER*2·10 -9 frame CRC32, checksumless UDP data must have time stamp: a 64-bit PSN sufficient, congestion control: none, data thinning, change sampling mode iBOB UDP streaming first implementation iBOB has only 14MB RAM Gbps) => no retransmissions => plain UDP packetizer, 1 stream per channel streaming configurable remotely via Telnet/ssh overall iBOB test design… =>. Basic Data Acquisition System

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #7 DAQ – iBOB RF-to-10G Test Design Basic Data Acquisition System

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #8 DAQ – Storage / RAID Performance Should use RAID-0 disk striping for best throughput Hardware RAID e.g. PCIe cards, XRaid, VTrak,.. too expensive and slow Software RAID baseline performance test on: ~1k€ workstation PC, Linux RAID-0 12 x SpinPoint F1, 9TB Test method: - source buffer ~90% RAM size - sequential fwrite() to a single file - not using 10G / UDP yet => Inexpensive 1–4 Gbps data storage system! Basic Data Acquisition System

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #9 DAQ – Stream Recording Wrote simple UDP/raw capture programs udp2raid, raw2raid Essential for performance in 10G stream recording: 10G NIC throughput: 4.7/9K jumbo frames, UDP checksums off prefer larger UDP packets, larger OS buffers => Myri, Chelsio NIC PC iperf : 9.9 Gbps UDP, 0% loss Application and disk throughput: use multi-core system, try to utilize (nearly) all cores  separate network and disk I/O into own threads => udp2raid, raw2raid currently min. 4 Gbps from 10G to RAID-0 => we have 1–4 Gbps iBOB stream to Abidal PC disk recording Basic Data Acquisition System

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #10 Fast “off-line” e-VLBI File Transfer Transfer station files to where you process the data, one option: Tsunami UDP v1.1 fast ‘FTP’ and real-time ( Weekly used in geodesic VLBI for ≤1 Gbps file transfer, occasional real-time Current v1.1 version, first result on 10G LAN: high CPU load, 3 – 3.5 Gbps to RAID, protocol rate limit 4 Gbps Started v1.2 of protocol to use udp2raid improvements: 4 – 4.4 Gbps to Abidal PC RAID 7 Gbps to RAM (vs. UDT4-protocol 5.2 Gbps to RAM) Basic Data Acquisition System => Improved rates for “off-line” e-VLBI

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #11 Summary Many European VLBI stations start to reach ~1 Gbps streaming with the old hardware Basic iBOB design can further improve the rate: digitize and stream 1 – 20 Gbps UDP A consumer ~1 k€ PC + 12 disks is capable of 4 Gbps storage: can capture the iBOB stream(s) to PC RAID Tsunami v1.2 has improved file transfer rates Can transfer from station to similar ≥4 Gbps PC elsewhere ToDo: port real-time Tsunami server onto iBOB board

10 April 2008EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBISlide #12 Questions/Answers Contact information Jan Wagner M.Sc./Researcher EXPReS (Metsähovi) Additional Information [note: only one “s”] EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #026642