1. Roberto Barbera Prague, 12.12.2002 ALICE Multi-site Data Transfer Tests on a Wide Area Network Giuseppe Lo Re Roberto Barbera Work in collaboration with: P. Cerello, D. Di Bari, G. Donvito (CMS), E. Fragiacomo, A. Fritz, M. Luvisetto, M. Masera, F. Minafra, D. Mura, S. Piano, M. Sitta, J. Švec, R. Turrisi Contributions from GARR and INFN NetGroup: C. Allocchio, M. Campanella, L. Gaido, S. Lusso, M. Michelotto, S. Spanu, S. Zani, D. De Girolamo CHEP 2004, 30 Sep 2004

2. Roberto Barbera Prague, 12.12.2002 Outline Objectives Preparation and benchmark Testbed layout and test results Conclusions CHEP 2004, 30 Sep 2004

3. Prague, 12.12.2002 Objectives See if the actual bandwidths can cope with the ALICE needs Spot possible bottle-necks out in the point-to-point transfers (I/O  LAN  WAN  LAN  I/O) Check, with “real” numbers of “real” use cases, if bandwidth attributions foreseen in the next future are adequate I/O server Front-end router Front-end router WANLAN disk I/O (W/R block size) TCP windows # streams BDP = BW*RTT CHEP 2004, 30 Sep 2004

4. Roberto Barbera Prague, 12.12.2002 Preparation and Benchmark Standard configuration of both the TCP stack and disk I/O parameters in Linux SSH keys exchanged among all machines to “secure” file transfers without typing passwords Automatic procedure installed on all machines with both Flat and Multi-Tier configurations. CHEP 2004, 30 Sep 2004

5. Roberto Barbera Prague, 12.12.2002 Testbed layout and “numbers” BA: 3 servers (2 ALICE, 1 CMS) BO: 6 servers CA: 2 servers CNAF: 2 servers CT: 2 servers PD: 6 servers TO: 2 servers TS: 1 server Prague: 1 server Houston: 1 server CNAF Padova Houston Prague CHEP 2004, 30 Sep 2004

6. Roberto Barbera Prague, 12.12.2002 Disk access measurements (non reserved access, local disk) MachineWrite (MBytes/s)Read (MBytes/s) boalice8.bo.infn.it55 server3.ca.infn.it4561 aliserv10.ct.infn.it2734 alifarm02.to.infn.it4059 alifarm.ts.infn.it2836 MachineWrite (MBytes/s)Read (MBytes/s) boalice8.bo.infn.it53 server3.ca.infn.it4332 aliserv10.ct.infn.it5725 pcalice19.pd.infn.it55 alifarm02.to.infn.it3153 alifarm.ts.infn.it2734 Bonnie++1.10 IOzone-3.164 CHEP 2004, 30 Sep 2004

7. Roberto Barbera Prague, 12.12.2002 Bandwidth measurements MachineBW1(Mb/s)BW2(Mb/s)BW4 (Mb/s)BW8(Mb/s)BW16(Mb/s)BW32(Mb/s) boalice8.bo.infn.it767779848687 server3.ca.infn.it12212221 22 aliserv10.ct.infn.it91518 1920 pcalice19.pd.infn.it265187929394 alifarm02.to.infn.it275057616469 alifarm.ts.infn.it1418 19 Iperf-1.6.3 Netperf-2.1 MachineBW1(Mb/s)BW2(Mb/s)BW4 (Mb/s)BW8(Mb/s)BW16(Mb/s)BW32(Mb/s) boalice8.bo.infn.it304465808186 server3.ca.infn.it131822 23 aliserv10.ct.infn.it916192022 pcalice19.pd.infn.it265187929397 alifarm02.to.infn.it284146556165 alifarm.ts.infn.it141718 1719 CHEP 2004, 30 Sep 2004

8. Roberto Barbera Prague, 12.12.2002 GARR network status at the beginning Bari: 28 Mb/s (BGA: 16 Mb/s) Bologna: 32 Mb/s Cagliari: 8 Mb/s Catania: 34 Mb/s CNAF: 1024 Mb/s Padova: 155 Mb/s Torino: 155 Mb/s (BGA: 70 Mb/s) Trieste: 16 Mb/s CHEP 2004, 30 Sep 2004

9. Roberto Barbera Prague, 12.12.2002 Network bandwidths (after the tests) Bari: 28 Mb/s (BGA: 16 Mb/s) Bologna: 100 Mb/s (BGA: 32 Mb/s) Cagliari: 32 Mb/s Catania: 34 Mb/s (direct connection to GARR-G in 6 months, up to 2.5 Gb/s) CNAF: 1024 Mb/s Padova: 155 Mb/s Torino: 155 Mb/s (BGA: 70 Mb/s) Trieste: 24 Mb/s CHEP 2004, 30 Sep 2004

10. Roberto Barbera Prague, 12.12.2002 Flat test CHEP 2004, 30 Sep 2004 Each server transfer files from/to any other servers waits a random time uniformly choosen between 0 and customizable maximum (1 min and 5 mins tried so far) chooses at random on of the other N-1 servers (with a weight proportional to the maximum bandwith of the site that server belongs to) chooses at random one of three files with different sizes (1.6 GB, 0.8 GB, and 0.3 GB) sends back and forth the file using bbFTP with a customizable number of parallel streams (16 and 8 tried) checks if any bits got lost and fills a detailed log file

11. Roberto Barbera Selected results (Bologna) saturated ! Official GARR NOC statistics CHEP 2004, 30 Sep 2004

12. Roberto Barbera Selected results (Cagliari) saturated ! Official GARR NOC statistics

13. Roberto Barbera Selected results (Catania) heavy traffic ! Official GARR NOC statistics CHEP 2004, 30 Sep 2004

14. Roberto Barbera Prague, 12.12.2002 Multi-tier use-case (HBT prod., 5000 evts., 9 TB) CNAF 60% Tier-1 CT 20% TO 20% Tier-2 1.8 TB 1 MB in 50 MB out Tier-3/4 BABOCAPDTS CHEP 2004, 30 Sep 2004

15. Roberto Barbera Results (Official GARR NOC stats.) Tier1@CNAF Tier2@Torino Tier2@CataniaTier3@Cagliari

16. Roberto Barbera Prague, 12.12.2002 Latest developments TCP tuning to improve throughput The participation of non Italian sites (especially with large RTT’s) like Prague and Houston has been useful to verify the effect of TCP tuning. SiteRTT (msec) from CNAFBW (Mb/s) from CNAFBDP (MB) Houston140701.2 Prague202500.6 Catania25 0.08 CHEP 2004, 30 Sep 2004

17. Roberto Barbera Prague, 12.12.2002 bbFTP vs # streams and TCP windows Catania-CNAF Max bw measured (iperf) = 25 Mb/s 1 streams 2 streams 4 streams Saturated also for small files saturated CHEP 2004, 30 Sep 2004

18. Houston-CNAF, Max bw measured (iperf) = 50 Mb/s Roberto Barbera Prague, 12.12.2002 bbFTP vs # streams and TCP windows 1 streams 2 streams 4 streams 6 streams saturated CHEP 2004, 30 Sep 2004

19. Prague-CNAF, Max bw measured (iperf) = 250 Mb/s Roberto Barbera Prague, 12.12.2002 Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration bbFTP vs no streams and TCP windows 1 streams 2 streams With an very high maximum buffer size (130KB->8 MB) 1 streams 2 streams CHEP 2004, 30 Sep 2004

20. Prague-CNAF with 1, 2, 4 and 6 streams with a very large maximum buffer size (8MB » BDP). [Ref: http://www-didc.lbl.gov/TCP-tuning/] Roberto Barbera Prague, 12.12.2002 2 streams 1 streams 4 streams6 streams Bottleneck at I/O level CHEP 2004, 30 Sep 2004

21. Roberto Barbera Prague, 12.12.2002 Conclusions First “real” multi-site/multi-server stress-test of the Italian GARR network Actual bandwidths resulted strongly inadequate if we especially consider all ALICE sites “as a whole” and the present number of servers already available by now Useful information on the actual farm architecture (limits of NFS in case of many parallel threads and big files) Big “perturbation” and interest inside both INFN NetGroup and GARR with prompt and excellent feed-back and support Strong and “incredibly” fast bandwith upgrades in many sites made by the GARR NOC Mapping of the testbed on a multi-tier topology does not seem to pose major problems for Tier-3’s CHEP 2004, 30 Sep 2004