FTS Issue in Beijing Erming PEI 2010/06/18

Failures IN2P3BEIJING experienced many transfer failures since the middle of May. Almost all large files failed, only small files passed. Only ~500KB/s per file in average. FTS Channel configuration: 10files and 10streams shared by ATLAS and CMS ~1Gb/s bandwidth in total (iperf result: ~90MB) ATLAS failures is proportional to CMS transfers CMS had aggressive transfers from all over the world with multiple channels and 2 routes, while ATLAS transfers only from CC-IN2P3 to Beijing in single channel.

Network Routes Europe America CSTNET IHEP 2.5Gbps 622Mbps 1Gbps CMS ATLAS IHEP

Reason/Solution There’s a network bottleneck in CSTNET!! No matter how we tuned all the parameters, the peak of the total bandwidth can only reach 550Mbps. Even when we tried to get CMS activities stopped! After negotiation CNIC, the total bandwidth was recovered to ~1Gbps. Both ATLAS/CMS file transfers had better performance later. All transfers passed.

Total Throughput

Intermediation ATLAS and CMS still compete the bandwidth CMS transfers files from all the T1s and many T2s to Beijing, which ate most of the bandwidth. In order to intermediate the two experiments, we had such Tuned FTS channel to 40 concurrent files and 16 streams Tuned TCP maximum window to 32MB Set the concurrent number of files and VO share specifically for CMS to limit its activities. CMS channel limit on IN2P3-BEIJING and STAR-BEIJING: 5 Reduce CMS VO share on IN2P3-BEIJING to 20%

Now, the files transfer rate is very good. Big files all got successfully transferred.

BEIJINGIN2P3 Since June 4th, BEJINGIN2P3 had very low performance. It turned out that the fault of bad TCP window default size: only ~80KB

Individual Throughput <50KB/s

TCP Window parameters: After tuned the default TCP window size to 4MB, the situation got improved. TCP Window parameters: net.ipv4.tcp_rmem = 1048576 4194304 16777216 net.ipv4.tcp_wmem = 1048576 4194304 16777216

Conclusion Found a network bottleneck and finally got it eliminated Tuning of TCP window size has good effect on improving the performance. Intermediation between ATLAS and CMS is necessary. Thanks to Eric for keeping eyes on this issue…