1. Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit Development of Real Time eVLBI at Jodrell Bank Observatory 7 th International eVLBI Workshop, Shanghai, 16-17 th June 2008 Richard Hughes-Jones DANTE Jonathan Hargreaves JBO

2. Connect. Communicate. Collaborate The 4 Gig Path for EXPReS Onsala – Jodrell Multi: Domain Vendor Technology

3. Connect. Communicate. Collaborate First Test Path 4 Gig TDM/SDH Lighpath over GÉANT2+ 10 Gigabit Ethernet Lambda over NORDUnet

4. Connect. Communicate. Collaborate UDP Throughput Stoc-Lon Alcatel Metro Core Connect MCC Flow control OFF Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes Max throughput 4.05 Gbit/s Packet loss as expected Sending host, 3 CPUs idle For 90% in kernel mode Receiving host 3 CPUs idle For <20 µs packets, 1 CPU is ~30% in kernel mode

5. Connect. Communicate. Collaborate UDP Jitter Stoc-Lon Alcatel Metro Core Connect MCC Flow control OFF Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=0 Coalescence OFF 8972 byte packets 16.5 μs spacing Peak in packet spacing at 16 μs as expected Other peaks at 19 and 51 µs

6. Connect. Communicate. Collaborate UDP Throughput Lon-Stoc Alcatel Metro Core Connect MCC Flow control ON Kernel 2.6.20-web100_pktd Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes Max throughput 4.05 Gbit/s No Packet loss – flow control Sending host, 3 CPUs idle For 60% in kernel mode Receiving host 3 CPUs idle For <20 µs packets, 1 CPU is ~30% in kernel mode

7. Connect. Communicate. Collaborate TCP reno Throughput Lon-Stoc Alcatel Metro Core Connect MCC Flow control ON 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 3.85 Gbit/s iperf Max throughput 4.02 Gbit/s 3 re-transmits Due to “other reductions”

8. Connect. Communicate. Collaborate TCP reno Throughput Stoc-Lon MCC NO Flow control 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 192 Mbit/s iperf Max throughput 465 Mbit/s TCP Cwnd to 0 Many re-transmits

9. Connect. Communicate. Collaborate Alcatel MCC Buffer size Classic Bottleneck 10 Gbit/s input 4 Gbit/s output Use udpmon to send a stream of spaced UDP packets Measure packet number of first lost frame as function of w packet spacing Slope gives buffer size ~57 kBytes

10. Connect. Communicate. Collaborate Use UDP to emulate TCP slowstart udpmon sends bursts of spaced packets: –32 packets –Jumbo 8000 bytes –back2back –4 ms between bursts Path Lon-Ams_FF-Prague-Paris- Lon Rtt 55.5 ms See 13 packets then loose 1 in 3 Confirm the TCP problem!

11. Connect. Communicate. Collaborate iNetTest: iBoB FPGA with 10 GE

12. Connect. Communicate. Collaborate iNetTest: Control & Operation 10/100 Ethernet 10Gb Port 0 iNetTest iBOB 1 Send and receive packets of selected length and spacing Count packets transmitted and received, calculate histograms of arrival times Control PC IP control of multiple iBOBs 10Gb Port 1 10Gb Port 0 10Gb Port 1 Ten Gb Network iNetTest iBOB n Expansion to more than two iBOBs iNetTest iBOB 2 Send and receive packets of selected length and spacing Count packets transmitted and received, calculate histograms of arrival times

13. Connect. Communicate. Collaborate iNetTest: Simulink design Jonathan Hargreaves JBO Network test parameter CSRs 10GE MAC core Inter-packet time histogram Transmit & receive event time log

14. Connect. Communicate. Collaborate iNetTest: Details iNetTest FPGA runs with a 200 MHz (5 ns) clocks Two iNetTest units can be controlled over ethernet IP from a PC iNetTest Ethernet IP address and MAC preset in code – last digit selected by jumper Ten GE IP address, Port and Gateway can be user configured Automatic or manual ARPing. ARP tables for each port can be examined iNetTest responds to and generates PING UDP packets can be sent by firmware between two iNetTests or between iNetTest and PCs User selects the number of packets to send, their length, and the time between them iBOBs count received and transmitted packets, and store up to 2048 time stamped events per port iBOBs generate histograms of the arrival time distribution for received packets

15. Connect. Communicate. Collaborate UDP Throughput vs. Packet Spacing PC Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R CX4 –rx-usecs=25 Coalescence ON MTU 9000 bytes UDP Packets Max throughput 9.4 Gbit/s iBoB Packet 8234 Data: 8192+ Header: 42 100 MHz clock Max rate 6.6 Gbit/s See 6.44Gbit/s 200 MHz clock - Linespeed

16. Connect. Communicate. Collaborate PC to iNetTest Packet Jitter Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre Packet separation 100 µs iBoB bins 2320 ns Width ± 34 µs Similar to PC-PC but no extra peaks Structure caused by PC The PC – PC plots look like:

17. Connect. Communicate. Collaborate iNetTest – iBOB to iBOB Results 10 GE 3m CX4 cable link 8192 byte packets 1 million transmitted at line rate between two iNetTest units. 5 ns bins B2B 10 ns FWHM With Switch: –Fujutsu XG2000 20 ns –HP 6400cl 35 ns long tail Number of Packets Time between packet arrivals (ns)

18. Connect. Communicate. Collaborate Back to the 4 Gig Path

19. Connect. Communicate. Collaborate Stockholm – Manc. with TSS 4 Gig TDM/SDH Lighpath over GÉANT2+ 4 Gigabit Ethernet Path over TSS NORDUnet 4 Gig on Ethernet Optical Transmission JANET & NNW

20. Connect. Communicate. Collaborate UDP Throughput Stoc-Man with TSS Alcatel TSS – 10GE – MCC Kernel 2.6.20-web100_pktd Myricom 10G-PCIE-8A-R Fibre –rx-usecs=75 Coalescence ON MTU 9000 bytes Max throughput 3.78 Gbit/s (Previously was 4.05 Gbit/s) Packet loss 10 % at 4.096 Gbit/s for 8192 Bytes Man-Stoc: No packet loss @ 4.096 Gbit/s

21. Connect. Communicate. Collaborate Packet loss Stoc-Man with TSS Alcatel TSS – 10GE – MCC Send 1M packets from PC in Stockholm 8192 bytes All cross the TSS cloud All enter Alcatel MCC as 10 GE frames Packet loss in MCC Card ~10% at 4096 Mbit/s Classic “bottleneck” performance But WHY???

22. Connect. Communicate. Collaborate UDP Jitter Man-Stoc with TSS Alcatel MCC – 10GE – TSS – PC Kernel 2.6.20-web100_pktd Myricom 10G-PCIE-8A-R Fibre –rx-usecs=0 Coalescence OFF 8192 byte packets Alcatel MCC is TDM 28 VC-4 Can only send packets at 16 µs Not at 10 Gbit/s Peak packet spacing ~ 6 µs ie packets arriving at 10 Gbit/s Suggests packet bunching in the network Sto-Man no TSS 16 µs

23. Connect. Communicate. Collaborate UDP 1-way delay Man-Stoc with TSS Alcatel MCC – 10GE – TSS – PC Kernel 2.6.20-web100_pktd Myricom 10G-PCIE-8A-R Fibre –rx-usecs=0 Coalescence OFF 8192 byte packets Structure similar to interrupt coalescence but this is OFF As function of recv time see gaps of ~ 130 us packets at ~ 10Gbit/s Again suggests packet bunching in the network

24. Connect. Communicate. Collaborate Last week: Onsala – Jodrell iNetTest - iNetTest 4 Gig TDM/SDH Lighpath over GÉANT2+ 4 Gigabit Ethernet Path over TSS NORDUnet 4 Gig on Ethernet Optical Transmission JANET & NNW

25. Connect. Communicate. Collaborate iNetTest: Onsala  JBO UDP Packet Jitter 1M packets 8192 byte packets 16 µs spacing 4.096 Gbit/s 10 % packet loss Peaks 14 & 15.7 µs Tail to 130 µs Cyclic variations spacing 3.25 µs 45 packets at 6.5 µs (line rate)

26. Connect. Communicate. Collaborate Development of Real Time eVLBI at JBO eMERLIN Import – Onsala to Jodrell eMERLIN Export – Jodrell to JIVE

27. Connect. Communicate. Collaborate Overview of eVLBI at JBO eMERLIN CORRELATOR JBOJIVE 4Gbps light path Onsala JBO CX4 4Gbps Or fibre if > 15m CX4 4Gbps iBOB 5SwitchiBOB 4SwitchADCStation BoardVSI 4Gbps light path CX4 1Gbps iBOB 0SwitchStation BoardVSI VSI to ZDOK Switch iBOB 1Station BoardVSI VSI to ZDOK iBOB 2Station BoardVSI VSI to ZDOK iBOB 3Station BoardVSI VSI to ZDOK VLBI Mk V b receiv ers

28. Connect. Communicate. Collaborate Onsala to JBO: Transmit eMERLIN Import ZDOK0ZDOK0 iADC 2 channels 1024MSPS per channel 8 bit sampling 10GB PORT. 4GB/s out ASSEMBLE PACKETS Divide data into 8192 byte packets plus 16 byte MkVC compatible header Alternatively generate test packets as per iNetTest Count packets transmitted DATA Left Pol ADCCLK ONEPPS Convert data to 2 bit Measure signal power on both channels Generate real time clock based on ADC clock Pass data & sync from ADC clock to iBOB 200MHz clock domain Power PC iBOB control and monitoring via registers on the OBP bus 10/100 ethernet Measured RMS Signal Power Read and Set Real Time Clock Set up test mode parameters Control Data Flow ON/OFF Set Network Source and Destination DATA Right Pol iBOB

29. Connect. Communicate. Collaborate Onsala to JBO: Receive ZDOK0ZDOK0 10GB PORT. 4GB/s in RE-ORDER PACKETS Store data in SRAM location according to its time stamp and sequence number Regenerate Sync and Data Valid signals FIFO 16k deep Smooth out bunching due to network delay variations Power PC iBOB control and monitoring via registers on the OBP bus 10/100 ethernet iBOB STREAM OUT Clock data out of SRAM on 128MHz correlator clock Stream the left and right polarisations to the Station Board VSI chips ZDOK1ZDOK1 VSI Chip 0 Route left polarisation data to input stage of Station Board Send correlator clock to iBOB VSI Chip 1 Route right polarisation data to input stage of Station Board Station Board SRAM0 512 packets L pol. data SRAM1 512 packets R pol. data DATA 32 bit 128MHz clock

30. Connect. Communicate. Collaborate Onsala to JBO - Status iADC and transmit iBOB tested with ADC clocked at 1GS/s in diagnostic mode – ADC is set to output a fixed bit pattern iBOB receiver and VSI chip coded in Simulink and VHDL Next Steps Test iADC and transmit iBOB with sinusoidal test signal Generated deliberately out-of-order packets and bursts of greater than 4Gbps to test receiver buffering

31. Connect. Communicate. Collaborate JBO to JIVE: Transmit – Overview eMERLIN Export ZDOK0ZDOK0 10GB PORT. 1 Gb/s out REMOVE DELAY MODEL (COURSE) Use 72 bit wide bt 512k deep SRAM to remove eMERLIN delay model with 16 microsecond resolution FORMAT Generate 10000 byte MkVb frames Power PC iBOB control and monitoring via registers on the OBP bus 10/100 ethernet iBOB ZDOK1ZDOK1 VSI Chip 0 Process left polarisation data Detail on next slide VSI Chip 1 Process right polarisation data Detail on next slide Station Board DATA 32 bit CLOCK VALID PDATA CLOCK VALID PACKETISE Divide each MkVb frame into two 5008 byte packets (incl header) Add Mk5C header to each packet

32. Connect. Communicate. Collaborate JBO to JIVE: Transmit -VSI Chip Detail VSI Chip 0 - Process left polarisation data Monitor & Control Bus (MCB) Station Board DATA 32 bit CLOCK VALID PDATA Select a 128MHz band from the filter bank Subtract Merlin Delay (Fine) Variable coefficient FIR filter changes delay in steps of 1/16th of original sample period Reclock for delays up to 16ns Mixer 0 FIR Low pass filter Decimate by 8/16 128 taps Convert 4/8 bit data to 2 bit VLBI Mixer 1 Mixer 8/16 Stream data out to iBOB over VSI cable MCB Interface Load band selection, delay model and filter parameters 0 1 15 Mixer Bank Select 8 or 16 VLBI bands from 128MHz input Remove eMERLIN offset (a multiple of 10kHz)

33. Connect. Communicate. Collaborate JBO to JIVE: Status VSI chip DSP functions currently under development and simulation Transmit iBOB re-uses packetising code from iNetTest Next Steps VSI chip MCB interface – based on Verilog code provided by Dave Fort at Penticton Transmit iBOB needs coarse delay and MkVb formatting

34. Connect. Communicate. Collaborate Summary UDP on a 28 VC-4, 4.2 Gigabit lightpath provides the performance required for EXPReS TCP performance poor due to small Ethernet buffer size on the equipment connecting 10GE to the 4 Gig Lightpath Ethernet flow-control with short tails helps TCP The network behaves correctly. Care needed on choice of PC for 10 Gigabit. FPGA iNetTest solutions work very well. Onsala – Jodrell 4 Gigabit path now in place. Packet loss and packet spacing need more understanding. VLBI FPGA application progressing well.

35. Connect. Communicate. Collaborate Any Questions ?

36. Connect. Communicate. Collaborate For More Information www.geant2.net www.dante.net For latest news and factsheets http://www.geant2.net/mediahttp://www.geant2.net/media For research activities http://www.geant2.net/researchhttp://www.geant2.net/research http://expres-eu.org/ [note: only one “s”]http://expres-eu.org/ http://www.jive.nl/ Contact information: Richard Hughes-Jones, DANTE, Richard.Hughes-Jones@dante.org.uk Jonathan Hargreaves, JBO, jh@jb.man.ac.ukRichard.Hughes-Jones@dante.org.ukjh@jb.man.ac.uk EXPReS is made possible through the support of the European Commission (DG- INFSO), Sixth Framework Programme, Contract #02664

37. Connect. Communicate. Collaborate GÉANT2 Topology November 2006

38. Connect. Communicate. Collaborate TCP BIC Throughput Lon-Stoc Alcatel Metro Core Connect MCC Flow control ON 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 3.98 Gbit/s iperf Max throughput 4.02 Gbit/s Several re-transmits

39. Connect. Communicate. Collaborate TCP BIC Throughput Stoc-Lon MCC NO Flow control 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 71.6 Mbit/s iperf Max throughput 192 Mbit/s TCP Cwnd rapidly opens Many more re-transmits

40. Connect. Communicate. Collaborate TCP BIC Throughput Lon-Stoc But Faster and Longer MCC Flow control 48 VC-4 RTT 60 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 59 MByte 1 * BDP txquelen 1300 … iperf Ave throughput 6.5 Gbit/s iperf Max throughput ~6.9 Gbit/s Some glitches & re-transmits