1. iBOB Data Acquisition and Network Streaming
Jan Wagner et al.
TKK / Metsähovi Radio Observatory
7th International eVLBI Workshop
Shanghai, 2008

2. Current e-VLBI Data Acquisition
Introduction
Current e-VLBI Data Acquisition
Telescope data generated by Data Acquisition (DAQ) hardware
e-VLBI : telescope data is sent via networks to correlator
capture DAQ data locally to RAID disk array, send afterwards
send data in real-time, capture and store or real-time process remotely
Current DAQ situation in the EVN:
Receiver and RF front-end : »512 MHz RF bandwidth
VLBA rack downconverters : VCs & BBCs, ≤8 MHz base-band / ch
Mark4-style A/D digitizers : max 32 Ms/s/channel 2-bit → 64 Mb/s / ch
max theoretical rate : 16 x 2-ch BBCs → 2 Gbit/s (512 MHz RF)
max rate in practice : formatting, Mark5 disk&NIC → ≤ 1 Gbit/s
=> Stuck at 256/512 MHz, 1 Gbit/s?
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

3. Towards More DAQ Bandwidth
Introduction
Towards More DAQ Bandwidth
Today’s networks and PC systems easily support over 1G rates
=> just a matter of providing antenna data samples faster
=> need a DAQ with more bandwidth
More RF bandwidth is beneficial for astro VLBI
Correlation/synthesis yields better sensitivity, noise: improved images
Observation time per object could be reduced, more objects per session
More bandwidth from bit reso allows AGC, band equalizer, RFI mitigation, …
To improve on the current DAQ:
Input bandwidth: off-the shelf gigasample A/D converter(s)
Processing: keep as wideband for astro, fine channelization for geo(?)
Output interface: copper Ethernet, UDP packet streams
For Metsähovi DAQ development and tests, used iBOB platform
C. Phillips, e-VLBI Scientific Benefits, ATNF CSIRO memo, 2004
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

4. iBOB DAQ Hardware Platform
iBOB DAQ Description
iBOB DAQ Hardware Platform
UC Berkeley’s iBOB FPGA board
Virtex II Pro 2vp50 with PPC CPUs
two copper 10GbE ports (CX4)
two expansion ports (Z-DOK)
Berkeley iADC board
Atmel AT84AD001
dual-channel, 8-bit, 1 Gs/s/ch
=> 1 GHz MHz BW, 16 Gbit/s / board
Nat LMX2531 or ADI ADF4360
~ MHz PLL for ADC fs
Similar board ROACH (meerKAT) available “soon”, has 4 x 10GbE, iBOB designs should drop in
For iBOB, created several DAQ “firmware” test designs
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

5. iBOB IF-to-Ethernet Design Example
iBOB DAQ Description
iBOB IF-to-Ethernet Design Example
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

6. iBOB DAQ and Protocol for 10GbE
UDP Packet Format
iBOB DAQ and Protocol for 10GbE
Real-time e-VLBI: data integrity less important than timely send
data loss factors f≤10% tolerable, image degrades
<=> packet loss and reorder ok, payload integrity non-critical
<=> don’t bother to retransmit lost packets
<=> UDP checksums unnecessary: frame CRC32->2·10-9*BER
congestion control: none - or client-specified sample mode changes
The only essential points – implemented in Test-DAQ UDP
must know which RF band/channel a sample comes from:
sample series from same channel placed into same UDP packet
for <60k channels can just send channel X to IPv4/6 port# Y
samples must be time stamped:
trigger acquisition at 1PPS, give packets a 64-bit packet sequence#
assume 4kB UDP => ambiguity after 4.5e15 samples : Ts/s
rest is “constant” a priori info, doesn’t belong into the stream (YMMV)
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

7. iBOB DAQ Tests Local 10G LAN tests
iBOB Streaming Tests
iBOB DAQ Tests
Local 10G LAN tests
Analog signal sampled with iBOB, streamed as UDP to 10G LAN
UDP stream recording with ‘udp2raid’, ‘raw2raid’ (~tcpdump, gulp)
udp2raid’ed to RAID-0 at up to 4 Gbps (current RAID ~4.3G, XFS)
Data, spectrum checks with Matlab and swspectrometer
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

8. Mh-On Streaming/Network Tests
iBOB Streaming Tests
Mh-On Streaming/Network Tests
First tests on 14/15th May
Supplied Onsala with test firmware 100 MHz clock, 1024 Ms/s sampling
I/Q: VLBI IF MHz, -10dBm
On streamed to Mh RAID-PC
Mh PC busy with other computations…
Recorded 1.6, rest with slow-PC loss
Checked for and found PCAL
Second test 11th June
used ”adc2tsunami”, proto/test design
sent 2 channels (I/Q) to On
IBOB % regwrite dest_ip_ich 0x82F20A05 # , Onsala
IBOB % regwrite dest_port_ich # I-channel to port 46330
IBOB % regwrite dest_ip_qch 0x82F20A05 # …
IBOB % regwrite dest_port_qch # …
IBOB % regwrite teng_wordsperframe 512 # 512x8 : 4kB UDP
IBOB % regwrite teng_decfactor # sample rate reduction 1/(1+n)
3.2G, (6.4G), 1.6 Gbps; 2 channels
© NORDUnet (
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

9. Mh-On Streaming Test - Route
iBOB Streaming Tests
Mh-On Streaming Test - Route
© NORDUnet (
Funet→Nordunet→Sunet Mh On
© OptoSUNET (
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

10. Mh-On Streaming Test - Traffic
iBOB Streaming Tests
Mh-On Streaming Test - Traffic
2G, 4G, 7.8G
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

11. Mh-On Streaming Test - Errors
iBOB Streaming Tests
Mh-On Streaming Test - Errors
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

12. Mh-On Streaming Test - Comments
iBOB Streaming Tests
Mh-On Streaming Test - Comments
Borje Josefsson (Sunet chief tech): “AFAIK the tests started at 10AM Swedish time, why can't I ever go and have a beer at that time :-)”
BJ made Internet Land Speed record in 2004, 4.3 Gbps over 42 hops of shared networks from commercial ISP
Will upgrade Sunet-Nordunet to 40G this week
Roy Molini (Funet/CSC): “It is definitely exciting to see data streaming from Finland at such incredible rates across infrastructures built specifically for meeting just those demands. The idea of Research Networking is clearly shown.”
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

13. Conclusions Basic Mh iBOB designs allow to
digitize antenna signal, different rates
stream as UDP to local or remote IP
On->Mh RAID recording was lossless “only” at 1.6 Gbps
try again when PC really free - local tests give 4G to disk
Mh->On network streaming
No problems with 0 – 7.8 Gbps traffic
Credit goes to Nordunet, Sunet, Funet: it is straightforward and welcome to stream multi-Gbps over Internet/RN
Future test or demo: stream and record a maser source with 2-bit / 8-bit, 2 x 512 MHz BW, analyze in swspectrometer
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

14. Conclusion
Questions/Answers
Contact information
Jan Wagner
M.Sc./Researcher
EXPReS (Metsähovi)
Additional Information
[note: only one “s”]
EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #026642
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai

15. Conclusion
Refs
Comparison of two frequency synthesizers: LMX2531 and ADF
Multiplier-Free Band-Selectable Digital Filters Arnaud Santraine, Sébastien Leprince, Fred Taylor, ICASSP 2004
Programmable, multiplier free channelizer Arnaud Santraine, Sébastien Leprince
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai