1. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 1 FABRIC 4 Gigabit Work & VLBI-UDP Performance and Stability. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 2 uWhat is inside GÉANT2 u What is the collaboration interesting? 10 Gigabit Ethernet UDP memory-2-memory flows TCP flows with allocated Bandwidth uOptions using GÉANT Development Network 10 Gbit SDH Network uOptions Using the GÉANT LightPath Service PoP Location for Network tests Multi-Gigabit Trials on GEANT Collaboration with Dante.

3. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 3 GÉANT2 Topology

4. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 4 GÉANT2: The Convergence Solution NREN Access Existing IP Router Existing IP Router GÉANT2 POP B GÉANT2 POP A Managed Lambda’s 1626 LM L2 Matrix L2 TDM Matrix TDM 1678 MCC Dark Fiber EXPReS PC 10 GE EXPReS PC 10 GE

5. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 5 What do we want to do? uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in their PoPs with 10 Gigabit NICs we FABRIC provide the SX XFPs, Dante the MCC ports uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems uDante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP when using BW limited Lightpaths uNeed A Collaboration Agreement

6. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 6 Generic solution: Rate adaptive mapping into VC4-nv with flow ctrl. and LCAS, n=1..64; up to 4 ports per board Full rate option: Ethernet full rate up to VC4- 68v, flow ctrl. and LCAS for n<68 Optical Transport Hierarchy option: Mapping into OpticalDataUnit1/2 with flow ctrl. O/E Data Mapper 4x 10GE LAN board in 1678MCC Data Mapper MAC O/E BKP dr Data Mapper Data Mapper MAC O/E Data Mapper Device - GFP-F into VC-4-nv - GFP-F into ODU1/2 4x10GE LAN interface ports To matrix 10GE-LAN interface board in 1678 MCC Dante interested in MCC performance

7. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 7 Options Using the GÉANT Development Network u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: London Amsterdam Paris Prague Frankfurt uCan do traffic routing so make long rtt paths uAvailable Feb 07 uLess Pressure for long term tests

8. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 8 Options Using the GÉANT LightPaths uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in Dante PoPs u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna uCan do traffic routing so make long rtt paths uIdeal: London Copenhagen

9. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 9 Test PCs Have Arrived u Boston/Supermicro X7DBE u Two Dual Core Intel Xeon Woodcrest 5130 2 GHz Independent 1.33GHz FSBuses u 530 MHz FD Memory (serial) uChipsets: Intel 5000P MCH – PCIe & Memory ESB2 – PCI-X GE etc. u PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X u 2 Gigabit Ethernet u SATA

10. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 10 Lab Tests 10 Gigabit Ethernet u10 Gigabit Test Lab set up in Manchester Cisco 7600 Cross Campus λ <1ms Server quality PCs Neterion NICs 4 Myricom 10 Gbit NICs – delivery this week Chelsio being purchased uB2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing uInstalled Fedora Core5 Kernels 2.6.17 & 2.6.18 (+web100 + packet drop) on the Intel dual-core PCs (Stephen Kershaw)

11. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 11 iBOB

12. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 12 Prototype iBOB with two sampler boards attached FPGA based signal processing board from UC Berkeley See Ralph’s Talk this afternoon

13. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 13 Bryan Anderson Station Board RAM iBOB 10GE VSI 10GE CX4 system based Disk VSI or headstack CX4 - fibre media converter iBOB block diagram u10 Gigabit Ethernet now available uUDP/IP module exists uUse for Demonstration of FPGA driven IP networking uLink iBoB to PC NIC – basic tests uTest over GÉANT u4 Gigbit flow Onsala - Jodrell

14. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 14 vlbi_udp Multiple stream tests over the Academic network

15. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 15 vlbi_udp Flow Diagram Ring buffer Input thread Ring buffer Send thread Disk Memory Receive thread Output thread Disk Memory From Simon Casey UDP Data TCP Control Control thread Vlbi_control vlbi_send vlbi_recv

16. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 16 Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Torun Poland e-VLBI at the GÉANT2 Launch Jun 2005

17. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 17 Geant2 Launch event, June 2005

18. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 18 UDP Performance: 3 Flows on GÉANT uThroughput: 5 Hour run uJodrell: JIVE 2.0 GHz dual Xeon – 2.4 GHz dual Xeon 670-840 Mbit/s uMedicina (Bologna): JIVE 800 MHz PIII – mark623 1.2 GHz PIII 330 Mbit/s limited by sending PC uTorun: JIVE 2.4 GHz dual Xeon – mark575 1.2 GHz PIII 245-325 Mbit/s limited by security policing (>400Mbit/s  20 Mbit/s) ? uThroughput: 50 min period uPeriod is ~17 min

19. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 19 UDP Performance: Flows on GÉANT Packet Loss & Re-ordering uJodrell: 2.0 GHz Xeon Loss 0 – 12% Reordering significant uTorun: 2.4 GHz Xeon Loss 6 - 12% Reordering in-significant

20. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 20 Instability: 2 streams into JIVE, March 2006 From Simon Casey

21. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 21 Dedicated Gbit link EVN-NREN Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Chalmers University of Technology, Gothenburg Torun Poland Gbit link Metsähovi

22. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 22 Any Questions?

23. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 23 Backup Slides

24. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 24 From PoS to Ethernet Connect. Communicate. Collaborate More Economical Architecture Highest Overall Network Availability Flexibility (VLAN management) Highest Network Performance (Latency) Router IP Links 1/10 Gigabit Ethernet VC-4-nv Channels L2 Matrix TDM Matrix 1678 MCC Transport Node VLANs

25. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 25 10 Gigabit Ethernet: UDP Throughput u1500 byte MTU gives ~ 2 Gbit/s uUsed 16144 byte MTU max user length 16080 uDataTAG Supermicro PCs uDual 2.2 GHz Xenon CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate throughput of 2.9 Gbit/s uCERN OpenLab HP Itanium PCs uDual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.7 Gbit/s uSLAC Dell PCs giving a uDual 3.0 GHz Xenon CPU FSB 533 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.4 Gbit/s

26. EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 26 10 Gigabit Ethernet: Tuning PCI-X u16080 byte packets every 200 µs uIntel PRO/10GbE LR Adapter uPCI-X bus occupancy vs mmrbc Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s mmrbc 1024 bytes mmrbc 2048 bytes mmrbc 4096 bytes 5.7Gbit/s mmrbc 512 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update