1. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 2 Remote Computing Farms uDiscussion at CERN to establish a work-plan for 2006 Valuable for Monitoring and Calibration MOU Alberta CERN Krakow Manchester New Network Topology with all links carried by GÉANT and NRNs uPlanned Investigations Characterise the new network links and end host performance Tools:iperf udpmon thrulay yatm Measure the ATLAS request-response behaviour Tools: tcpmon, web100 tcpdump Setup the WAN emulator with the measured conditions Compare network and ATLAS traffic observations Install and test ATLAS application gateway (as used at the pit) Test deployment of Online TDAQ HLT releases Measure performance of Online TDAQ HLT releases Consider how to link Real-Time T/DAQ to remote Grid farms uFirst draft of Work Plan document circulated

3. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 3 Network Operation & Performance uAnalysis of Fault Tolerance in ATLAS T/DAQ Networks Document the action of the switches Fate of the packets Effect on T/DAQ applications Networks Considered: Front End (DataFlow) Network BackEnd Network Controls Network (Run control, services, some monitoring) Consider questions like: “Failure of a link between the ROS and the ROS Concentrator Switch” Draft Document being discussed uPerformance tests discussed The PCI-e 4* 1GE PEG4 NIC Silicom. Simple and trunking Throughput ROS SuperMicro Motherboard 6 PCI, 1 4 lane PCI-e, one 3.4 GHz Xeon (dual socket)

4. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 4 Network Monitoring in ATLAS T/DAQ uLevels of Monitoring SNMP Statistics MRTG, RRD, YATM higher sample rate Traffic patterns, bytes, packets NOT dropped packets Network test programs udpmon, iperf Throughput loss 1-way delay rtt Standalone ATLAS test programs speaking the TDAQ application protocol. Richard ATLAS test programs speaking the TDAQ application protocol using TDAQ APIs Stefan Monitoring by the TDAQ application itself uIntegration of Message Passing Libraries DataFLow (Reiner) and EF (Mario) main difference in substantiation of buffers Integrate over common thin shim over the socket calls uIdea to put monitoring into (common) message passing layer What can be observed? Question of keeping state – Application would be the best place !

5. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 5 Related Work: RAID, ATLAS Grid uRAID0 and RAID5 tests 4 th Year MPhys project last semester Throughput and CPU load Different RAID parameters Number of disks Stripe size User read / write size Different file systems Ext2 ext3 XSF Sequential File Write, Read Sequential File Write, Read with continuous background read or write uStatus Need to check some results & document Independent RAID controller tests planned.

6. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 6 ESLEA: ATLAS Grid on UKLight uDemonstration of benefits of Dedicated links 1 Gbit Lightpath Lancaster-Manchester Disk 2 Disk Transfers Storage Element with SRM using distributed disk pools dCache & xrootd

7. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 7 Check out the end host: bbftp uWhat is the end-host doing with your application protocol? uTransatlantic bbftp over TCP/IP uLook at the PCI-X buses u3Ware 9000 controller RAID0 u1 Gbit Ethernet link u2.4 GHz dual Xeon u~660 Mbit/s PCI-X bus with RAID Controller PCI-X bus with Ethernet NIC Read from disk for 44 ms every 100ms Write to Network for 72 ms

8. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 8 Any Questions?

9. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 9 Backup Slides

10. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 10 TCP Stacks & CPU Load uReal User problem! uEnd host TCP flow at 960 Mbit/s with rtt 1 ms falls to 770 Mbit/s when rtt 15 ms u1.2GHz PIII rtt 1 ms TCP iperf 980 Mbit/s Kernel mode 95% Idle 1.3 % CPULoad with nice priority Throughput falls as priority increases No Loss No Timeouts uNot enough CPU power u2.8 GHz Xeon rtt 1 ms TCP iperf 916 Mbit/s Kernel mode 43% Idle 55% CPULoad with nice priority Throughput constant as priority increases No Loss No Timeouts uKernel mode includes TCP stack and Ethernet driver

11. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 11 A Few Items for Discussion uAchievable Throughput uSharing link Capacity (OK what is sharing?) uConvergence time uResponsiveness urtt fairness (OK what is fairness?) umtu fairness uTCP friendliness uLink utilisation (by this flow or all flows) uStability of Achievable Throughput uBurst behaviour uPacket loss behaviour uPacket re-ordering behaviour uTopology – maybe some “simple” setups uBackground or cross traffic - how realistic is needed? – what protocol mix? uReverse traffic uImpact on the end host – CPU load, bus utilisation, Offload uMethodology – simulation, emulation and Real links ALL help

12. T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 12 More Information Some URLs 1 uUKLight web site: http://www.uklight.ac.uk uMB-NG project web site: http://www.mb-ng.net/ uDataTAG project web site: http://www.datatag.org/ uUDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net uMotherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt & http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ uTCP tuning information may be found at: http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ uDante PERT http://www.geant2.net/server/show/nav.00d00h002