Presentation is loading. Please wait.

Presentation is loading. Please wait.

Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft 10Gbit between GridKa and openlab (and their obstacles) Forschungszentrum Karlsruhe GmbH.

Published byDavid Rogers Modified over 9 years ago

Similar presentations

Presentation on theme: "Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft 10Gbit between GridKa and openlab (and their obstacles) Forschungszentrum Karlsruhe GmbH."— Presentation transcript:

1 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft 10Gbit between GridKa and openlab (and their obstacles) Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany http://www.gridka.de Bruno Hoeft

2 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Outline LAN of GridKa –Strucure –Prevision of installation in 2008 WAN –History (1G/2003) –Current 10G –Testbed –Challenges crossing multi NREN (National Research and Education Network) –Quality and quantity evaluation of GridKa and openlap network connection –File transfer –Caching effect

3 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft CMS ATLAS LHCb CERN Tier 0 Centre at CERN working groups virtual organizations Germany (FZK) Tier 1 USA (Fermi, BNL) UK (RAL) France (IN2P3) Italy (CNAF) ………. CERN Tier 1 ………. Tier 0 The LHC multi-Tier Computing Model  Tier 3 (Institute computer) Tier 4 ( Desktop)      Grid Computing Centre Karlsruhe Tier 2 (Uni-CCs, Lab-CCs) Lab y Uni a Lab i Uni b Lab z Lab x Uni c Uni d Uni e LHC Computing Grid Project - LCG

4 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Projects at GridKa Atlas (SLAC, USA) (CERN) (FermiLab,USA) calculating jobs with “real” data LHC Experimentenon-LHC Experimente 1,5 Mio. Jobs and 4,2 Mio. hours calculation in 2004

5 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft gradual extention of GridKa resources Apr 2004Okt 2004Apr 2005 % of 2008 Processors68010701.28030 % Computing power / kSI2k580920129012 % Disk [TB]16022027018 % Tape [TB]28037547512 % Internet [Gb/s]210* 10 50 % April 2005: biggest Linux-Cluster within the German Science Society largest Online-Storage at a single installation in Germany strongest Internet connection in Germany available at the Grid with over 100 installations in Europe * Internet connetion for sc

6 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft router private network … … C ompute N ode … Ethernet 320 Mbit Ethernet 100 Mbit Ethernet 1 Gbit FiberChannel 2Gbit Ethernet 10 Gbit FS … … FS NAS FS SAN … … switch Network installation Storage direct internet access 1 Gbit/s GridKa PIX L ogin S erver … DFN L ogin S erver router sc nodes … SAN 10Gbit/s s ervice c hallenge only

7 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft router 1Gbit/s GridKa PIX L ogin S erver … FS … … private network Storage direct internet access 1 Gbit/s … FS NAS FS DFN … … C ompute N ode … … L ogin S erver switch router Network installation SAN Master Controller RM -Ganglia -Nagios -Cacti switch incl. Management Network Ethernet 320 Mbit Ethernet 100 Mbit Ethernet 1 Gbit FiberChannel 2Gbit Ethernet 10 Gbit 10Gbit/s s ervice c hallenge only sc nodes … SAN

8 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft CN - Rack … … … … FS … … Block Adminis- tration Block A Backbone Router completed end 2005 CN - Rack … … … … FS … … Block Adminis- tration Block C Backbone Router 2006 2007 ?? 10 Gbit 10 Gbit Internet 10 Gbit light path to CERN 10 Gbit CN - Rack … … … … FS … … Block Adminis- tration Block B Backbone Router CN - Rack … … … … FS … … Block Adminis- tration Block D Backbone Router Projection of installation in 2008

9 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft CERN GridFTP server GridFTP server WAN 2003/4 -- Gigabit GridKa – CERN (DataTag) Géant 10 Gbps DFN 2.4 Gbps GridFTP tested over 1 Gbps Karlsruhe Frankfurt 2x 1 Gbps 98% of 1Gbit Geneva 1000 0

10 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft CERN GridFTP server GridFTP server 10Gigabit WAN SC GridKa – CERN(openlab) Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt Geneva ar-karlsruhe1-ge5-2-700.g-win.dfn.de r-internet.fzk.de cr-karlsruhe1-po12-0.g-win.dfn.de dfn.de1.de.geant.net de.it1.it.geant.net it.ch1.ch.geant.net swiCE2-P6-1.switch.ch swiCE3-G4-3.switch.ch 10 Gbps

11 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft CERN GridFTP server GridFTP server Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt ar-karlsruhe1-ge5-2-700.g-win.dfn.de r-internet.fzk.de cr-karlsruhe1-po12-0.g-win.dfn.de dfn.de1.de.geant.net swiCE3-G4-3.switch.ch de.fr1.fr.geant.net fr.ch1.ch.geant.net LSP Routing Geneva 10Gigabit WAN SC GridKa – CERN(openlab)

12 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft 10Gigabit WAN SC GridKa – CERN(openlab) CERN GridFTP server GridFTP server Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt -Bandwidth evaluation (tcp/udp) - MPLS via France (MPLS - MultiProtocol Label Switching) - LBE (Least Best Effort) - GridFTP server pool HD to HD Storage to Storage - SRM Geneva

13 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Hardware Various Xeon dual 2.8 and 3.0 GHz IBM x-series (Intel and Broadcom NIC) Recently added 3.0 GHz EM64T (800 FSB) Cisco 6509 with 4 10 Gb ports and lots of 1 Gb Storage:Datadirect 2A8500 with 16 TB Linux RH ES 3.0 (U2 and U3), GPFS 10 GE Link to GEANT via DFN (least best effort  ) TCP/IP stack 4 MB buffer 2 MB window size

14 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Quality Evaluation (UDP stream) 10gkt113 10gkt111 oplapro73 957 Mbit jitter: 0,028; ooo: 0 jitter: 0,021; ooo: 0 957 Mbit LAN ooo - out of order W A N [10gtk111] iperf -s –u [10gtk113] iperf -c 192.108.46.111 -u -b 1000M [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0 -10 sec 1141 MBytes 957 Mbits/sec 0.028 ms 0/813885 (0%) [10gtk113] iperf -s –u [10gtk111] iperf -c 192.108.46.113 -u -b 1000M [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0 -30 sec 3422 MBytes 957 Mbits/sec 0.022 ms 0/2440780 (0%) LAN WAN 953 Mbit; jitter: 0,018 ms; ooo: 215 – 0,008% 884 Mbit; jitter: 0,015 ms; ooo: 4239 – 0,018% [oplapro73]> iperf -s -u [10gtk113] iperf -c 192.16.160.13 -u -b 1000M [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0 -30 sec 3408 MBytes 953 Mbits/sec 0.019 ms 7299/2438606 (0.3%) [ 3] 0 -30 sec 215 datagrams received out-of-order [oplapro73]> iperf -c 192.108.46.113 -u -b 1000M [10gtk113] iperf -s -u [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0 -30 sec 3162 MBytes 884 Mbits/sec 0.015 ms 130855/2386375 (5.5%) [ 3] 0 -30 sec 4239 datagrams received out-of-order Reno scalable symmetric = 957 Mbit/sec = 953 Mbit/sec 884 Mbit/sec asymmetric

15 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Bandwidth evaluation (TCP stream) W A N 348Mbit/sec – single stream W A N Reno 10gkt101 – 10gtk105 Reno oplapro73 iperf

16 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft 10gkt10[1-5] W A N 5 nodes a 112MByte/sec – 24 parrallel streams Reno iperf Oplapro7[1-5] Reno Bandwidth evaluation

17 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Evaluation of max throughput 700 1400 5600 4900 4200 3500 2800 2100 7000 6300 Mbit/s 18:00 20:00 9 nodes each site -8 * 845 Mbit -1 * 540 Mbit higher speed at one stream is resulting in a packet loss

18 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Gridftp sc1 throughput Sc1 – 500MByte sustained 4 th Feb. 056 th Feb. 058 th Feb. 057 th Feb. 05 5 th Feb. 05 4900 4200 3500 2800 2100 1400 700 Mbit/s to /dev/null to hd/SAN 19 Nodes - 15 WorkerNodes* 20MByte IDE/ATA HD - 1 FileServer * 50MByte SCSI HD - 3 FileServer* 50MByte SAN

19 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft SC2 approx 1/5 of the load

20 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft SC2 five nodes at GridKa gridftp to gpfs

21 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft SC2-part 2 - Trouble shooting with radiant - Shaping diffrent host performances (load bolancing) - Parallel threads did not perfom better - Best performance 20 parallel file copies Equal Nodes (no performance diffrenz)

22 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft cache Gridftp 2 /dev/null 80 70 60 50 40 30 MByte/sec MByte - TCP windos 1,0 1,5

23 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft cache Gridftp 2 gpfs MByte - TCP windos 80 60 40 20 0 MByte/sec 0,5 1,0 1,5

24 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft cache 80 70 60 50 40 30 MByte/sec 80 60 40 20 0 MByte/sec MByte - TCP windos 0,5 1,0 1,5 1,0 1,5

25 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft cache Gridftp 8Gbyte File 80 70 60 50 40 30 20 MByte/sec MByte - TCP windos 0,5 1,0 1,5 80 70 60 50 40 MByte/sec 0 1,0 1,5 2,0 3,0 * * current throughput average throughput 2Gbyte 16% of time 2,5 messurement points

26 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Conclusion multi NREN 10Gbit link up to 7.0Mbit usable SC2 part 1 : approx. 1/5 of the CERN aggregated load SC2 part 2 : 250 MByte stable, peak over 400MByte TCP for WAN (un-)modified?

27 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Gpfs for data dest. Digging into HW details to discover bottlenecks (packet drop due to bad PCI timing) Stabilise the transport Installation of SRM Installation of dcache Planned 2006 –Lightpath via X-Win (DFN) / Geant2 to CERN Future Work some solved, but still ongoing since sc2 is not aproaching the edge, the impreession is far more stable for SC, migrate to production since sc2 end of March single stream up to 115MByte sustained multistream 60 to 70MByte

28 Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft

Download ppt "Service Challenge Meeting - ISGC_2005, 26. Apr. 2005 Bruno Hoeft 10Gbit between GridKa and openlab (and their obstacles) Forschungszentrum Karlsruhe GmbH."

Similar presentations

TAB, 03. March 2006 Bruno Hoeft German LHC – WAN present state – future concept Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O.

TAB, 03. March 2006 Bruno Hoeft German LHC – WAN present state – future concept Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O.
ESLEA and HEPs Work on UKLight Network. ESLEA Exploitation of Switched Lightpaths in E- sciences Applications Exploitation of Switched Lightpaths in E-

ESLEA and HEPs Work on UKLight Network. ESLEA Exploitation of Switched Lightpaths in E- sciences Applications Exploitation of Switched Lightpaths in E-
27.05.2004Bernd Panzer-Steindel, CERN/IT WAN RAW/ESD Data Distribution for LHC.

Bernd Panzer-Steindel, CERN/IT WAN RAW/ESD Data Distribution for LHC.
GridKa May 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Installing dCache into an existing Storage environment at GridKa Forschungszentrum.

GridKa May 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Installing dCache into an existing Storage environment at GridKa Forschungszentrum.
Service Data Challenge Meeting, Karlsruhe, Dec 2, 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Plans and outlook at GridKa Forschungszentrum.

Service Data Challenge Meeting, Karlsruhe, Dec 2, 2004 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Plans and outlook at GridKa Forschungszentrum.
T0/T1 – Network Meeting 19.07.2005 Bruno Hoeft T0/T1 – GridKa/CERN Network Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box.

T0/T1 – Network Meeting Bruno Hoeft T0/T1 – GridKa/CERN Network Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box.
LCG Grid Deployment Board, March 2003 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Status of GridKa for LCG-1 Forschungszentrum Karlsruhe.

LCG Grid Deployment Board, March 2003 Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Status of GridKa for LCG-1 Forschungszentrum Karlsruhe.
Current Testbed : 100 GE 2 sites (NERSC, ANL) with 3 nodes each. Each node with 4 x 10 GE NICs Measure various overheads from protocols and file sizes.

Current Testbed : 100 GE 2 sites (NERSC, ANL) with 3 nodes each. Each node with 4 x 10 GE NICs Measure various overheads from protocols and file sizes.
Service Challenge Meeting “Service Challenge 2 update” James Casey, IT-GD, CERN IN2P3, Lyon, 15 March 2005.

Service Challenge Meeting “Service Challenge 2 update” James Casey, IT-GD, CERN IN2P3, Lyon, 15 March 2005.
Tiziana FerrariWP2.3 Advance Reservation Demonstration: Description and set-up 1 WP2.3 Advance Reservation Demonstration: Description and set-up DRAFT,

Tiziana FerrariWP2.3 Advance Reservation Demonstration: Description and set-up 1 WP2.3 Advance Reservation Demonstration: Description and set-up DRAFT,
Tiziana FerrariWP2.3 Advance Reservation Demonstration: Description and set-up 1 WP2.3 Advance Reservation Demonstration: Description and set-up DRAFT,

Tiziana FerrariWP2.3 Advance Reservation Demonstration: Description and set-up 1 WP2.3 Advance Reservation Demonstration: Description and set-up DRAFT,
KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association STEINBUCH CENTRE FOR COMPUTING - SCC www.kit.edu.

KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association STEINBUCH CENTRE FOR COMPUTING - SCC
ATLAS computing in Geneva Szymon Gadomski, NDGF meeting, September 2009 S. Gadomski, ”ATLAS computing in Geneva, NDGF, Sept 091 the Geneva ATLAS Tier-3.

ATLAS computing in Geneva Szymon Gadomski, NDGF meeting, September 2009 S. Gadomski, ”ATLAS computing in Geneva", NDGF, Sept 091 the Geneva ATLAS Tier-3.
GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.

GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.
Mapping a Network by Latency (and other things). ------------------------------------------------------------ Client connecting to 10.1.1.1, UDP port.

Mapping a Network by Latency (and other things) Client connecting to , UDP port.
02 nd April 03Networkshop 20031 Managed Bandwidth Next Generation F. Saka UCL NETSYS (NETwork SYStems centre of excellence)

02 nd April 03Networkshop Managed Bandwidth Next Generation F. Saka UCL NETSYS (NETwork SYStems centre of excellence)
March 27,28 2009IndiaCMS Meeting, Delhi1 T2_IN_TIFR of all-of-us, for all-of-us, by some-of-us Tier-2 Status Report.

March 27, IndiaCMS Meeting, Delhi1 T2_IN_TIFR of all-of-us, for all-of-us, by some-of-us Tier-2 Status Report.
CMS Data Transfer Challenges LHCOPN-LHCONE meeting Michigan, Sept 15/16th, 2014 Azher Mughal Caltech.

CMS Data Transfer Challenges LHCOPN-LHCONE meeting Michigan, Sept 15/16th, 2014 Azher Mughal Caltech.
1 A Basic R&D for an Analysis Framework Distributed on Wide Area Network Hiroshi Sakamoto International Center for Elementary Particle Physics (ICEPP),

1 A Basic R&D for an Analysis Framework Distributed on Wide Area Network Hiroshi Sakamoto International Center for Elementary Particle Physics (ICEPP),
Status Report on Tier-1 in Korea Gungwon Kang, Sang-Un Ahn and Hangjin Jang (KISTI GSDC) April 28, 2014 at 15th CERN-Korea Committee, Geneva Korea Institute.

Status Report on Tier-1 in Korea Gungwon Kang, Sang-Un Ahn and Hangjin Jang (KISTI GSDC) April 28, 2014 at 15th CERN-Korea Committee, Geneva Korea Institute.

Similar presentations

Ads by Google