Network performance issues recently raised at IN2P3-CC Guillaume.Cessieux@cc.in2p3.fr 2010-06-17 All units are in bit/sec

LHCOPN related (1/1) 2010-05-31: Performance problem with US-T1-BNL or CH-CERN (software report 40M) Paths are 10G (CERN is direct, BNL is reached through CERN) iperf tests, worst case results achieved both ways: FR-CCIN2P3 ↔ CH-CERN 0.95G on a 1G interface 1.83G on a 2G trunk FR-CCIN2P3 ↔ US-T1-BNL 550M on a 1G interface Not maximal maybe due to some other traffics interfering Far above throughput experienced by software No network issue 19 e-mails, case closed 2010-06-07 for network teams GCX 2010-06-17

Generic IP (1/2) IHEP (CN, Beijing), 2010-05-14 Max bandwidth expected: 90M While iperf achieved: IHEP → IN2P3-CC = 1.58M IN2P3-CC → IHEP = 60M Seems a mix of: Sporadic router overload or traffic-engineering experienced around Beijing’s peering (TEIN3/GÉANT) GÉANT’s ticket stillborn (unable to reproduce…) End hosts’ TCP stacks behaving differently during error recovery Two different hosts, in the same IN2P3-CC’s subnet, tested at the same time can give very different results (Sun Solaris vs Linux SL4) IHEP figured out theirs TCP parameters on hosts were mistakenly changed Reverted back on 2010-06-17 to those from 2010-06-04 Then FTS transfers performance back (~60M both way) 22 e-mails, case closed 2010-06-17 GCX 2010-06-17

Generic IP (2/2) KEK (JP, Tokyo), 2010-02-16 ~1G expected, but iperf reported: IN2P3-CC → KEK = 800M KEK → IN2P3-CC = 300M Good results between IN2P3-CC ↔ ICEPP (University of Tokyo) = ~1G And ICEPP and KEK have a very similar path to IN2P3-CC Thought to be a TCP stack issue Bad behaviour of SL3 TCP stack on high latency networks We figured out no production traffic is exchanged between KEK and IN2P3-CC, so no problem to solve… 17 e-mails, case closed 2010-04-14 Future hosts at KEK will be SL4 or SL5 GCX 2010-06-17

Conclusion Problem often lies on sender’s side Hard to clearly discriminate or not networks Sporadic and temporary issues Iperf tests are good but proof can only be obtained on end-hosts really used Killing generic measurement points on sites... Coordinated efforts required on both sides Too much time spent on administrative issues (get access, open ports etc.) Upcoming Clear process to better manage such regular issues at IN2P3-CC Clear actors, roles, responsibilities and tools Spanning across several teams: support, storage, system and network perfSONAR box at IN2P3-CC Regular bandwidth measurements GCX 2010-06-17