1. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 1 Collaborations in Networking and Protocols HEP and Radio Astronomy Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 2 VLBI Proof of Concept at iGrid2002 European Topology: NRNs, Geant, Sites SuperJANET4 iGrid 2002 Manchester Jodrell SURFnet JIVE

3. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 3 uCollaboration HEP, Radio Astronomy, Dante the NRNs, and Campus folks uSome results of the e-VLBI Proof of Concept

4. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 4 Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Torun Poland e-VLBI at the GÉANT2 Launch Jun 2005

5. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 5 e-VLBI UDP Data Streams uCollaboration HEP, Radio Astronomy, Dante the NRNs, and Campus folks uGood opportunity to test UDP Throughput: 5 Hour run

6. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 6 ESLEA and UKLight uExploiting Switched Lightpaths for e-Science Applications uEPSRC e-Science project £1.1M 11.5 FTE Core Technologies: Protocols Control plane HEP data transfers – ATLAS and D0 e-VLBI Medical Applications High Performance Computing uInvolved with Protocols, HEP and e-VLBI uStephen Kershaw appointed as RA (joint with EXPReS) uInvestigate how well the protocol implementations work UDP flows, TCP advanced stacks, DCCP (developed by UCL partners) Also examine how the Applications “use” the protocols Also the effect of the transport protocol on what the Application intended! uDevelop real-time UDP transport for e-VLBI – vlbi_udp

7. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 7 ESLEA and UKLight u6 * 1 Gbit transatlantic Ethernet layer 2 paths UKLight + NLR uDisk-to-disk transfers with bbcp Seattle to UK Set TCP buffer and application to give ~850Mbit/s One stream of data 840-620 Mbit/s uStream UDP VLBI data UK to Seattle 620 Mbit/s Reverse TCP

8. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 8 tcpmon: TCP Activity for remote Farms:Manc-CERN Req-Resp Web100 hooks for TCP status Round trip time 20 ms 64 byte Request green 1 Mbyte Response blue TCP in slow start 1st event takes 19 rtt or ~ 380 ms TCP Congestion window gets re-set on each Request TCP stack RFC 2581 & RFC 2861 reduction of Cwnd after inactivity Even after 10s, each response takes 13 rtt or ~260 ms Transfer achievable throughput 120 Mbit/s Event rate very low Application not happy!

9. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 9 ESLEA: ATLAS on UKLight 1 Gbit Lightpath Lancaster-Manchester Disk 2 Disk Transfers Storage Element with SRM using distributed disk pools dCache & xrootd

10. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 10 udpmon: Lanc-Manc Throughput not quite what we expected !! Lanc  Manc Plateau ~640 Mbit/s wire rate No packet Loss Manc  Lanc ~800 Mbit/s but packet loss Send times Pause 695 μs every 1.7ms So expect ~600 Mbit/s Receive times (Manc end) No corresponding gaps

11. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 11 EXPReS & FABRIC uEU Project to realise the current potential of eVLBI and investigate the Next Generation capabilities. uSSA Use of GRID Farms for distributed correlation. Linking Merlin telescopes to JIVE (present correlator) 4 * 1 Gigabit from Jodrell Links to 10 Service Challenge work. Interface to eMERLIN – data at 30 Gbit/s uJRA - FABRIC Investigate use of different IP Protocols 10 Gigabit Onsala to Jodrell Links to 10 Gbit HEP work. Investigate 4 Gigabit over GEANT2 Switched Lightpaths UDP and TCP Links to Remote Compute Farm HEP work. Develop 1 and 10 Gbit Ethernet end systems using FPGAs Links to CALICE HEP work.

12. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 12 FABRIC 4 Gigabit Demo Will use a 4 Gbit Lightpath between two GÉANT PoPs Collaboration with Dante – Discussions in progress Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP tests

13. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 13 10 Gigabit Ethernet: UDP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 2048 bytes 66 MHz uOne 8000 byte packets 2.8us for CSRs 24.2 us data transfer effective rate 2.6 Gbit/s u2000 byte packet, wait 0us ~200ms pauses u8000 byte packet, wait 0us ~15ms between data blocks CSR Access 2.8us Data Transfer

14. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 14 Calice uVirtex 4 board from pld Applications uPCI-express development card uUsing the FPGA to send and receive raw Ethernet frames at 1 Gigabit uPackage data from internal memory or external source into Ethernet uConsidering building a 10 Gigabit Ethernet add-on card Take data in on the1Gig links, processing it, send results out on 10Gig link. uUsing 2 boards (2nd is a data generator) we could produce a small scale Calice DAQ, take data in, buffer it to the DDR2 ram, and then read it out, Ethernet frame it and ship to PCs. uIdeas for an Ethernet packet monitor. From Slides of Marc Kelly

15. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 15 Backup Slides Further network & end host investigations

16. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 16 VLBI Work TCP Delay and VLBI Transfers Manchester 4th Year MPhys Project by Stephen Kershaw & James Keenan

17. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 17 VLBI Network Topology

18. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 18 VLBI Application Protocol u VLBI data is Constant Bit Rate u tcpdelay instrumented TCP program emulates sending CBR Data. Records relative 1-way delay Data1 ●●● Timestamp1 Time TCP & Network Receiver Timestamp2 Sender Data2 Timestamp4 Timestamp5 Data4 Timestamp3 Data3 Packet loss RTT Time Sender Receiver ACK Segment time on wire = bits in segment/BW uRemember Bandwidth*Delay Product BDP = RTT*BW

19. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 19 1 way delay – 10000 packets 1-Way Delay 1 way delay 100 ms Message number 100 ms u10,000 Messages uMessage size: 1448 Bytes uWait time: 0 uTCP buffer 64k uRoute: Man-ukl-JIVE-prod-Man uRTT ~26 ms

20. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 20 = 1.5 x RTT = 1 x RTT 26 ms Message number ≠ 0.5 x RTT 1 way delay 10 ms 10 ms uWhy not just 1 RTT? uAfter SlowStart TCP Buffer Full uMessages at front of TCP Send Buffer have to wait for next burst of ACKs – 1 RTT later uMessages further back in the TCP Send Buffer wait for 2 RTT 1-Way Delay Detail

21. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 21 Recent RAID Tests Manchester HEP Server

22. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 22 “Server Quality” Motherboards u Boston/Supermicro H8DCi u Two Dual Core Opterons 1.8 GHz u 550 MHz DDR Memory u HyperTransport uChipset: nVidia nForce Pro 2200/2050 u AMD 8132 PCI-X Bridge u PCI 2 16 lane PCIe buses 1 4 lane PCIe 133 MHz PCI-X u 2 Gigabit Ethernet u SATA

23. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 23 Disk_test: areca PCI-Express 8 port Maxtor 300 GB Sata disks RAID0 5 disks Read 2.5 G bit/s Write 1.8 Gbit/s RAID5 5 data disks Read 1.7 Gbit/s Write 1.48 Gbit/s RAID6 5 data disks Read 2.1 Gbit/s Write 1.0 Gbit/s

24. Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 24 UDP Performance: 3 Flows on GÉANT uThroughput: 5 Hour run uJodrell: JIVE 2.0 GHz dual Xeon – 2.4 GHz dual Xeon 670-840 Mbit/s uMedicina (Bologna): JIVE 800 MHz PIII – mark623 1.2 GHz PIII 330 Mbit/s limited by sending PC uTorun: JIVE 2.4 GHz dual Xeon – mark575 1.2 GHz PIII 245-325 Mbit/s limited by security policing (>400Mbit/s  20 Mbit/s) ? uThroughput: 50 min period uPeriod is ~17 min