1. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Progress at GridKa Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany http://www.gridka.de Bruno Hoeft

2. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Outline LAN of GridKa –Struckure –Prevision of installation in 2008 WAN –History (1G/2003) –Current 10G –Testbed –Challenges crossing multi NREN –Quality and qantity evalutation of network between GridKa and openlap

3. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft GridKa – Planning of WAN connectivity (projection from 2003, Reinhard Maschuw) 20012002200320042005200620072008 34 Mbps 155 Mbps 2 Gbps 10 Gbps 20 Gbps start discussions with DFN ! start discussions with Dante !

4. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft GridKa – Planning of WAN connectivity 20012002200320042005200620072008 34 Mbps 155 Mbps 2 Gbps 10 Gbps 20 Gbps start discussions with DFN ! start discussions with Dante / DFN!

5. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft router private network Storage direct internet access 1 Gbit/s … … C ompute N ode … Ethernet 320 Mbit Ethernet 100 Mbit Ethernet 1 Gbit FiberChannel 2Gbit Ethernet 10 Gbit 1Gbit/s GridKa PIX L ogin S erver … DFN L ogin S erver router Network installation FS … … FS NAS FS SAN … … switch

6. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft router 1Gbit/s GridKa PIX L ogin S erver … FS … … private network Storage direct internet access 1 Gbit/s … FS NAS FS DFN … … C ompute N ode … … Ethernet 320 Mbit Ethernet 100 Mbit Ethernet 1 Gbit FiberChannel 2Gbit Ethernet 10 Gbit L ogin S erver switch router Network installation incl. Management Network SAN Master Controller RM -Ganglia -Nagios -Cacti switch

7. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft CN - Rack … … … … FS … … Block Adminis- tration Block A Backbone Router completed end 2005 CN - Rack … … … … FS … … Block Adminis- tration Block C Backbone Router 2006 2007 ?? 10 Gbit 10 Gbit Internet 10 Gbit CN - Rack … … … … FS … … Block Adminis- tration Block B Backbone Router CN - Rack … … … … FS … … Block Adminis- tration Block D Backbone Router

8. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft CERN GridFTP server GridFTP server WAN connection and Gigabit test with CERN Géant 10 Gbps DFN 2.4 Gbps GridFTP tested over 1 Gbps Karlsruhe Frankfurt 2x 1 Gbps 98% of 1Gbit Geneva

9. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft CERN GridFTP server GridFTP server WAN connection and 10Gigabit test with CERN Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt Geneva ar-karlsruhe1-ge5-2-700.g-win.dfn.de r-internet.fzk.de cr-karlsruhe1-po12-0.g-win.dfn.de dfn.de1.de.geant.net de.it1.it.geant.net it.ch1.ch.geant.net swiCE2-P6-1.switch.ch swiCE3-G4-3.switch.ch

10. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft CERN GridFTP server GridFTP server WAN connection and 10Gigabit test with CERN Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt ar-karlsruhe1-ge5-2-700.g-win.dfn.de r-internet.fzk.de cr-karlsruhe1-po12-0.g-win.dfn.de dfn.de1.de.geant.net swiCE3-G4-3.switch.ch de.fr1.fr.geant.net fr.ch1.ch.geant.net LSP Routing Geneva

11. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft CERN GridFTP server GridFTP server WAN connectivity and 10Gigabit test with CERN Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt -Bandwidth evaluation (tcp/udp) - MPLS via France - LBE - GridFTP server pool HD to HD Storage to Storage - SRM Geneva

12. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft TestBed 11 nodes –8 Tyan Motherboards onboard NIC (Intel/e1000) 1GByte Memory, IDE HDD Scientific Linux 3.0.3 Globus toolkit 2.4.3 –3 IBM Series x345 (Intel/e1000) 2 GByte Memory, SCSI HDD RedHat 8.0 Globus toolkit 2.4.3 Cisco catalyst 6509 –10 Gbit interface connected to DFN backbone –48 Port interface 10/100/1000 TP

13. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Quality Evaluation root@10gtk111 root]# iperf -c 192.108.46.113 -u -b 1000M -f m -i 1 -t 30 ------------------------------------------------------------ Client connecting to 192.108.46.113, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 0.06 MByte (default) [ 5] local 192.108.46.111 port 32793 connected with 192.108.46.113 port 5001 [ ID] Interval Transfer Bandwidth [ 5] -0.0- 1.0 sec 114 MBytes 957 Mbits/sec [ 5] 1.0- 2.0 sec 114 MBytes 957 Mbits/sec... [ 5] 0.0-30.0 sec 3422 MBytes 957 Mbits/sec [ 5] Server Report: [ 5] 0.0-30.0 sec 3422 MBytes 957 Mbits/sec 0.021 ms 0/2440780 (0%) [ 5] Sent 2440780 datagrams [root@10gtk113 root]# iperf -s -u -f m -i 1 ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 0.06 MByte (default) [ 3] local 192.108.46.113 port 5001 connected with 192.108.46.111 port 32793 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 114 MBytes 957 Mbits/sec 0.015 ms 0/81407 (0%) [ 3] 1.0- 2.0 sec 114 MBytes 957 Mbits/sec 0.016 ms 0/81382 (0%) [ 3] 2.0- 3.0 sec 114 MBytes 957 Mbits/sec 0.014 ms 0/81382 (0%)... [ 3] 29.0-30.0 sec 114 MBytes 957 Mbits/sec 0.017 ms 0/81382 (0%) [ 3] 0.0-30.0 sec 3422 MBytes 957 Mbits/sec 0.022 ms 0/2440780 (0%) 10gkt113 10gkt111 oplapro73 957Mbit; jitter: 0,021; ooo: 0 LAN W A N [root@10gtk111 root]# iperf -s -u -f m -i 1 ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 0.06 MByte (default) ------------------------------------------------------------ [ 3] local 192.108.46.111 port 5001 connected with 192.108.46.113 port 32769 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 114 MBytes 957 Mbits/sec 0.019 ms 0/81409 (0%) [ 3] 1.0- 2.0 sec 114 MBytes 957 Mbits/sec 0.015 ms 0/81383 (0%)... [ 3] 0.0-10.0 sec 1141 MBytes 957 Mbits/sec 0.028 ms 0/813885 (0%) [root@10gtk113 root]# iperf -c 192.108.46.111 -u -b 1000M -f m -i 1 -t 10 ------------------------------------------------------------ Client connecting to 192.108.46.111, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 0.06 MByte (default) ------------------------------------------------------------ [ 5] local 192.108.46.113 port 32769 connected with 192.108.46.111 port 5001 [ ID] Interval Transfer Bandwidth [ 5] -0.0- 1.0 sec 114 MBytes 957 Mbits/sec [ 5] 1.0- 2.0 sec 114 MBytes 957 Mbits/sec... [ 5] 0.0-10.0 sec 1141 MBytes 957 Mbits/sec [ 5] Server Report: [ 5] 0.0-10.0 sec 1141 MBytes 957 Mbits/sec 0.027 ms 0/813885 (0%) [ 5] Sent 813885 datagrams ooo - out of order

14. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft 10gkt111 957Mbit; jitter: 0,021; ooo: 0 ooo - out of order LAN Quality Evaluation 10gkt113 oplapro73 W A N 953Mbit; jitter: 0,018 ms; ooo: 215 – 0,008% 884Mbit; jitter: 0,015 ms; ooo: 4239 – 0,018% W A N [oplapro73] ~ > iperf -s -u -f m -i 1 ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 2.00 MByte (default) [ 3] local 192.16.160.13 port 5001 connected with 192.108.46.113 port 32769 [ 3] 0.0- 1.0 sec 114 MBytes 955 Mbits/sec 0.015 ms 191/81365 (0.23%) [ 3] 0.0- 1.0 sec 5 datagrams received out-of-order... [ 3] 29.0-30.0 sec 114 MBytes 956 Mbits/sec 0.014 ms 69/81389 (0.085%) [ 3] 29.0-30.0 sec 8 datagrams received out-of-order [ 3] 0.0-30.0 sec 3408 MBytes 953 Mbits/sec 0.019 ms 7299/2438606 (0.3%) [ 3] 0.0-30.0 sec 215 datagrams received out-of-order [root@10gtk113 root]# iperf -c 192.16.160.13 -u -b 1000M -f m -i 1 -t 30 ------------------------------------------------------------ Client connecting to 192.16.160.13, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 0.06 MByte (default) [ 5] local 192.108.46.113 port 32769 connected with 192.16.160.13 port 5001 [ ID] Interval Transfer Bandwidth [ 5] -0.0- 1.0 sec 114 MBytes 957 Mbits/sec... [ 5] 29.0-30.0 sec 114 MBytes 957 Mbits/sec [ 5] 0.0-30.0 sec 3419 MBytes 956 Mbits/sec [ 5] Server Report: [ 5] 0.0-30.0 sec 3408 MBytes 953 Mbits/sec 0.018 ms 7084/2438606 (0.29%) [ 5] 0.0-30.0 sec 215 datagrams received out-of-order [ 5] Sent 2438606 datagrams Reno scalable [oplapro73] ~ > iperf -c 192.108.46.113 -u -b 1000M -f m -i 1 -t 30 ------------------------------------------------------------ Client connecting to 192.108.46.113, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 2.00 MByte (default) [ 3] local 192.16.160.13 port 38143 connected with 192.108.46.113 port 5001 [ 3] 0.0- 1.0 sec 112 MBytes 937 Mbits/sec [ 3] 1.0- 2.0 sec 113 MBytes 946 Mbits/sec... [ 3] 28.0-29.0 sec 110 MBytes 925 Mbits/sec [ 3] 0.0-30.0 sec 3346 MBytes 936 Mbits/sec [ 3] Sent 2386505 datagrams [ 3] Server Report: [ 3] 0.0-30.0 sec 3162 MBytes 884 Mbits/sec 0.015 ms 135094/2386375 (5.7%) [ 3] 0.0-30.0 sec 4239 datagrams received out-of-order [root@10gtk113 root]# iperf -s -u -f m -i 1 ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 0.06 MByte (default) [ 3] local 192.108.46.113 port 5001 connected with 192.16.160.13 port 38143 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 106 MBytes 892 Mbits/sec 0.012 ms 2895/78757 (3.7%) [ 3] 0.0- 1.0 sec 180 datagrams received out-of-order... [ 3] 29.0-30.0 sec 104 MBytes 874 Mbits/sec 0.019 ms 4729/79007 (6%) [ 3] 29.0-30.0 sec 81 datagrams received out-of-order [ 3] 0.0-30.0 sec 3162 MBytes 884 Mbits/sec 0.015 ms 130855/2386375 (5.5%) [ 3] 0.0-30.0 sec 4239 datagrams received out-of-order

15. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Bandwidth evaluation 10gkt101 – 10gtk105 oplapro73 W A N 348Mbit/sec – single stream W A N Reno

16. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Bandwidth evaluation 10gkt101 oplapro73 W A N 112MByte/sec – 24 parallel streams W A N Reno Iperf

17. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Bandwidth evaluation 10gkt101 oplapro73 W A N 5 nodes a 112MByte/sec – 24 parrallel streams Reno Iperf

18. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft gridftp performance

19. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft Gpfs for data source and dest. Overcome the 20(25) MByte limit of o singel HD Ajust HW installation to achive the requirements Diging into HW details for discovering bottelnecks (packet drop due to bad PCI timing) Stabilis the transport Future Work

20. Service Challenge, 2. Dec. 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Bruno Hoeft