1. Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones University of Manchester In Collaboration with: Bryan Caron University of Alberta Krzysztof Korcyl IFJ PAN Krakow Catalin Meirosu TERANA Jakob Langgard Nielsen Niels Bohr Institute

2. Slide: 2 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 2 Tests from Krakow u Configure/Test/Operate Krakow – CERN ANT GPS box Learn many details !!! Successful GPS and ANT operation using GUI at CERN Histograms still to be done u Synchronized Manchester – Krakow clocks with GPS u Configured ANT test Krakow-Manchester BUT … Manc Alteon card 66 MHz Manc Netgear card 0.5Mbyte Exchange at CERN next week u Online help from Catalin u udpmon tests Krakow – CERN CERN – Krakow Manc – CERN

3. Slide: 3 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 3 GÉANT2 Topology

4. Slide: 4 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 4 GÉANT2: The Convergence Solution NREN Access Existing IP Router Existing IP Router GÉANT2 POP B GÉANT2 POP A Managed Lambda’s 1626 LM L2 Matrix L2 TDM Matrix TDM 1678 MCC Dark Fiber EXPReS PC 10 GE EXPReS PC 10 GE

5. Slide: 5 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 5 Pionier Weather Map

6. Slide: 6 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 6 Test Traffic on Pionier

7. Slide: 7 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 7 CERN  Krakow: Throughput CERN  Krakow Krakow  CERN u800Mbit/s u20% Loss in the network u600-800Mbit/s u40% Loss in the network u~3 * more re-ordering

8. Slide: 8 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 8 CERN  Krakow: Latency CERN  Krakow Krakow  CERN uFwhm ~10us uPeak separation 90-120 us uFwhm 50 us uPeak separation ~135 us

9. Slide: 9 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 9 CERN  Krakow: 1-way delay CERN  Krakow Krakow  CERN u20 us packet spacing uInterrupt Coalescence

10. Slide: 10 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 10 Thanks to all who helped, including: uNational Research Networks Canarie, Dante, DARENET, Netera, PSNC and UKERNA u“ATLAS remote farms” J. Beck Hansen, R. Moore, R. Soluk, G. Fairey, T. Bold, A. Waananen, S. Wheeler, C. Bee u“ATLAS online and dataflow software” S. Kolos, S. Gadomski, A. Negri, A. Kazarov, M. Dobson, M. Caprini, P. Conde, C. Haeberli, M. Wiesmann, E. Pasqualucci, A. Radu

11. Slide: 11 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 11 Any Questions?

12. Slide: 12 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 12 Backup Slides

13. Slide: 13 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 13 Remote Computing Concepts ROB L2PU SFI PF Local Event Processing Farms ATLAS Detectors – Level 1 Trigger SFOs Mass storage Experimental Area CERN B513 Copenhagen Edmonton Krakow Manchester PF Remote Event Processing Farms PF lightpaths PF Data Collection Network Back End Network GÉANT Switch Level 2 Trigger Event Builders

14. Slide: 14 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 14 ATLAS Remote Farms – Network Connectivity

15. Slide: 15 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 15 ATLAS Application Protocol u Event Request EFD requests an event from SFI SFI replies with the event ~2Mbytes u Processing of event u Return of computation EF asks SFO for buffer space SFO sends OK EF transfers results of the computation u Tcpmon - instrumented tcp request-response program emulates the Event Filter EFD to SFI communication. Send OK Send event data Request event ●●● Request Buffer Send processed event Process event Time Request-Response time (Histogram) Event Filter Daemon EFD SFI and SFO

16. Slide: 16 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 16 End Hosts & NICs CERN-Manc. Request-response Latency Throughput Packet Loss Re-Order uUse UDP packets to characterise Host & NIC SuperMicro P4DP8 motherboard Dual Xenon 2.2GHz CPU 400 MHz System bus 66 MHz 64 bit PCI bus

17. Slide: 17 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 17 TCP (Reno) – Details uTime for TCP to recover its throughput from 1 lost packet given by: u for rtt of ~200 ms: 2 min UK 6 ms Europe 20 ms USA 150 ms