1. Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI Richard Hughes-Jones

2. Connect. Communicate. Collaborate Progress [1] Oct Nov 07 EXPReS Requirements doc produced Path formally requested from JANET Meetings with SUNET & NORDUnet –Agree 4 Gigabit path initial priority –Organise wiki –Discuss other RAst. needs: Onsala-JIVE, Met-Onsala, Met-JBO LOFAR SUNET 10 GE Onsala-Stockholm installed Nov 07 NORDUnet install connectivity to Copenhagen –Request GÉANT2+ connection

3. Connect. Communicate. Collaborate Progress [2] Dec 07 JANET indicate insufficient GÉANT2+ access BW to UK –10Gig packet switched + 2.5 Gig backup –(10-2.5) Gig SDH Lightpath DANTE Provision 4 Gig path London PoP – Copenhagen PC shipped to Stockholm NORDUnet Install and connect PC Tests from EXPReS1 in London over GÉANT2+ –With internal loopback packets get to Copenhagen and back NORDUnet cant see light from DANTE MCC Blade failed Alcatel called to Stockholm JANET Propose direct 10 GE Lambda Manchester-London

4. Connect. Communicate. Collaborate Current Test Path 10 GE Lambda over NORDUnet 4 Gig Lighpath over GÉANT2+

5. Connect. Communicate. Collaborate 4 Gig Path JANET Optical

6. Connect. Communicate. Collaborate Tests on GÉANT2 Testbed UDP Fine at 4.2 Gigabit TCP performance poor due to small Ethernet buffer size on Alcatel 1678 Metro Core Connect TCP slow start sends bursts at 10 Gbit/s MCC cannot buffer these with 10 Gbit/s in and 4 Gbit/s out Evaluate new firmware v4.2 Ethernet flow-control can now be enabled London new firmware v4.2 Copenhagen firmware v4.1

7. Connect. Communicate. Collaborate UDP Throughput Lon-Stoc MCC Flow control Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes Max throughput 4.05 Gbit/s No Packet loss Sending host, 3 CPUs idle For 60% in kernel mode Receiving host 3 CPUs idle For <20 µs packets, 1 CPU is ~30% in kernel mode

8. Connect. Communicate. Collaborate UDP Throughput Stoc-Lon MCC Flow control Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes Max throughput 4.05 Gbit/s Packet loss as expected Sending host, 3 CPUs idle For 90% in kernel mode Receiving host 3 CPUs idle For <20 µs packets, 1 CPU is ~30% in kernel mode

9. Connect. Communicate. Collaborate UDP 1-way Delay Lon-Stoc MCC Flow control ON Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=0 Coalescence OFF MTU 8000 bytes Wait time 0 ie linespeed Packets Q with 8.5µs spacing Packets Leave Q 16 µs spacing

10. Connect. Communicate. Collaborate TCP reno Throughput Lon-Stoc MCC Flow control 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 3.85 Gbit/s iperf Max throughput 4.02 Gbit/s 3 re-transmits

11. Connect. Communicate. Collaborate TCP BIC Throughput Lon-Stoc MCC Flow control 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 3.98 Gbit/s iperf Max throughput 4.02 Gbit/s Several re-transmits

12. Connect. Communicate. Collaborate TCP reno Throughput Stoc-Lon MCC NO Flow control 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 192 Mbit/s iperf Max throughput 465 Mbit/s TCP Cwnd to 0 Many re-transmits

13. Connect. Communicate. Collaborate TCP BIC Throughput Stoc-Lon MCC NO Flow control 27 VC-4 RTT 31 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 32 MByte 2*BDP iperf Ave throughput 71.6 Mbit/s iperf Max throughput 192 Mbit/s TCP Cwnd rapidly opens Many more re-transmits

14. Connect. Communicate. Collaborate TCP BIC Throughput Lon-Stoc But Faster and Longer MCC Flow control 48 VC-4 RTT 60 ms Kernel 2.6.20-web100_pktd-plus Myricom 10G-PCIE-8A-R Fibre –rx-usecs=25 Coalescence ON MTU 9000 bytes TCP buffer 59 MByte 1 * BDP txquelen 1300 … iperf Ave throughput 6.5 Gbit/s iperf Max throughput ~6.9 Gbit/s Some glitches & re-transmits

15. Connect. Communicate. Collaborate Summary MCCs will be upgraded to v4.2 General case – e.g. with trunking i/f with multiple Qs / VLANs users may have to perform traffic shaping. Ethernet flow-control with short tails helps TCP UDP performance provides that required for EXPReS 28 VC-4 will be required. Do we wish to move PC to Onsala when convenient ? What are our timescales?

16. Connect. Communicate. Collaborate iBoB hot news

17. Connect. Communicate. Collaborate iNetTest Progress iBoB FPGA running with 200 MHz clock Statistics, histograms and timing events implemented Ethernet packet jitter measured 10GE_port0  10GE_port1 Two iNetTest units tested Back-2-Back UDP performance at 10 Gigabit linespeed

18. Connect. Communicate. Collaborate iNetTest B2B UDP Packet Jitter 10Gig Ethernet CX4 links 8000 byte packets 5 ns bins FWHM 10 ns No shape variation with packet size 8192 Bytes 3000 Bytes

19. Connect. Communicate. Collaborate iNetTest B2B UDP Packet size 10Gig Ethernet CX4 links 8000 byte packets Slope is 1 for 200MHz So we are really running at that speed!!

20. Connect. Communicate. Collaborate Any Questions ?

21. Connect. Communicate. Collaborate For More Information www.geant2.net www.dante.net For latest news and factsheets http://www.geant2.net/mediahttp://www.geant2.net/media For research activities http://www.geant2.net/researchhttp://www.geant2.net/research ALICE http://alice.dante.net/ http://alice.dante.net/ RedCLARA http://www.redclara.nethttp://www.redclara.net REUNA http://www.reuna.cl/joomla/index.php http://www.reuna.cl/joomla/index.php

22. Connect. Communicate. Collaborate GÉANT2 Topology November 2006