iBOB Data Acquisition and Network Streaming Jan Wagner et al. TKK / Metsähovi Radio Observatory 7th International eVLBI Workshop Shanghai, 2008

Current e-VLBI Data Acquisition Introduction Current e-VLBI Data Acquisition Telescope data generated by Data Acquisition (DAQ) hardware e-VLBI : telescope data is sent via networks to correlator capture DAQ data locally to RAID disk array, send afterwards send data in real-time, capture and store or real-time process remotely Current DAQ situation in the EVN: Receiver and RF front-end : »512 MHz RF bandwidth VLBA rack downconverters : VCs & BBCs, ≤8 MHz base-band / ch Mark4-style A/D digitizers : max 32 Ms/s/channel 2-bit → 64 Mb/s / ch max theoretical rate : 16 x 2-ch BBCs → 2 Gbit/s (512 MHz RF) max rate in practice : formatting, Mark5 disk&NIC → ≤ 1 Gbit/s => Stuck at 256/512 MHz, 1 Gbit/s? 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Towards More DAQ Bandwidth Introduction Towards More DAQ Bandwidth Today’s networks and PC systems easily support over 1G rates => just a matter of providing antenna data samples faster => need a DAQ with more bandwidth More RF bandwidth is beneficial for astro VLBI Correlation/synthesis yields better sensitivity, noise: improved images Observation time per object could be reduced, more objects per session More bandwidth from bit reso allows AGC, band equalizer, RFI mitigation, … To improve on the current DAQ: Input bandwidth: off-the shelf gigasample A/D converter(s) Processing: keep as wideband for astro, fine channelization for geo(?) Output interface: copper Ethernet, UDP packet streams For Metsähovi DAQ development and tests, used iBOB platform C. Phillips, e-VLBI Scientific Benefits, ATNF CSIRO memo, 2004 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

iBOB DAQ Hardware Platform iBOB DAQ Description iBOB DAQ Hardware Platform UC Berkeley’s iBOB FPGA board Virtex II Pro 2vp50 with PPC CPUs two copper 10GbE ports (CX4) two expansion ports (Z-DOK) Berkeley iADC board Atmel AT84AD001 dual-channel, 8-bit, 1 Gs/s/ch => 1 GHz MHz BW, 16 Gbit/s / board Nat LMX2531 or ADI ADF4360 ~950..1080 MHz PLL for ADC fs Similar board ROACH (meerKAT) available “soon”, has 4 x 10GbE, iBOB designs should drop in For iBOB, created several DAQ “firmware” test designs 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

iBOB IF-to-Ethernet Design Example iBOB DAQ Description iBOB IF-to-Ethernet Design Example 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

iBOB DAQ and Protocol for 10GbE UDP Packet Format iBOB DAQ and Protocol for 10GbE Real-time e-VLBI: data integrity less important than timely send data loss factors f≤10% tolerable, image degrades <=> packet loss and reorder ok, payload integrity non-critical <=> don’t bother to retransmit lost packets <=> UDP checksums unnecessary: frame CRC32->2·10-9*BER congestion control: none - or client-specified sample mode changes The only essential points – implemented in Test-DAQ UDP must know which RF band/channel a sample comes from: sample series from same channel placed into same UDP packet for <60k channels can just send channel X to IPv4/6 port# Y samples must be time stamped: trigger acquisition at 1PPS, give packets a 64-bit packet sequence# assume 4kB UDP => ambiguity after 4.5e15 samples : 245 years @ 10Ts/s rest is “constant” a priori info, doesn’t belong into the stream (YMMV) 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

iBOB DAQ Tests Local 10G LAN tests iBOB Streaming Tests iBOB DAQ Tests Local 10G LAN tests Analog signal sampled with iBOB, streamed as UDP to 10G LAN UDP stream recording with ‘udp2raid’, ‘raw2raid’ (~tcpdump, gulp) udp2raid’ed to RAID-0 at up to 4 Gbps (current RAID ~4.3G, XFS) Data, spectrum checks with Matlab and swspectrometer 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Mh-On Streaming/Network Tests iBOB Streaming Tests Mh-On Streaming/Network Tests First tests on 14/15th May Supplied Onsala with test firmware 100 MHz clock, 1024 Ms/s sampling I/Q: VLBI IF 90..540 MHz, -10dBm On streamed to Mh RAID-PC Mh PC busy with other computations… Recorded 1.6, rest with slow-PC loss Checked for and found PCAL Second test 11th June used ”adc2tsunami”, proto/test design sent 2 channels (I/Q) to On IBOB % regwrite dest_ip_ich 0x82F20A05 # 130.242.10.5, Onsala IBOB % regwrite dest_port_ich 46330 # I-channel to port 46330 IBOB % regwrite dest_ip_qch 0x82F20A05 # … IBOB % regwrite dest_port_qch 46331 # … IBOB % regwrite teng_wordsperframe 512 # 512x8 : 4kB UDP IBOB % regwrite teng_decfactor 0 # sample rate reduction 1/(1+n) 3.2G, (6.4G), 1.6 Gbps; 2 channels © NORDUnet (http://www.nordu.net/stats/) 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Mh-On Streaming Test - Route iBOB Streaming Tests Mh-On Streaming Test - Route © NORDUnet (http://www.nordu.net/stats/) Funet→Nordunet→Sunet Mh On © OptoSUNET (http://stats.sunet.se/) 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Mh-On Streaming Test - Traffic iBOB Streaming Tests Mh-On Streaming Test - Traffic 2G, 4G, 7.8G 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Mh-On Streaming Test - Errors iBOB Streaming Tests Mh-On Streaming Test - Errors 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Mh-On Streaming Test - Comments iBOB Streaming Tests Mh-On Streaming Test - Comments Borje Josefsson (Sunet chief tech): “AFAIK the tests started at 10AM Swedish time, why can't I ever go and have a beer at that time :-)” BJ made Internet Land Speed record in 2004, 4.3 Gbps over 42 hops of shared networks from commercial ISP http://proj.sunet.se/LSR3-s/ Will upgrade Sunet-Nordunet to 40G this week Roy Molini (Funet/CSC): “It is definitely exciting to see data streaming from Finland at such incredible rates across infrastructures built specifically for meeting just those demands. The idea of Research Networking is clearly shown.” 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Conclusions Basic Mh iBOB designs allow to digitize antenna signal, different rates stream as UDP to local or remote IP On->Mh RAID recording was lossless “only” at 1.6 Gbps try again when PC really free - local tests give 4G to disk Mh->On network streaming No problems with 0 – 7.8 Gbps traffic Credit goes to Nordunet, Sunet, Funet: it is straightforward and welcome to stream multi-Gbps over Internet/RN Future test or demo: stream and record a maser source with 2-bit / 8-bit, 2 x 512 MHz BW, analyze in swspectrometer 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Conclusion Questions/Answers Contact information Jan Wagner M.Sc./Researcher EXPReS (Metsähovi) jwagner@kurp.hut.fi Additional Information http://expres-eu.org/ [note: only one “s”] http://www.metsahovi.fi/en/vlbi/ EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #026642 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

Conclusion Refs Comparison of two frequency synthesizers: LMX2531 and ADF4360 http://www.metsahovi.fi/en/vlbi/ibob/Comparing-ADI-NAT.pdf Multiplier-Free Band-Selectable Digital Filters Arnaud Santraine, Sébastien Leprince, Fred Taylor, ICASSP 2004 Programmable, multiplier free channelizer Arnaud Santraine, Sébastien Leprince 12 June 2008 iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai