Atlas Canada Lightpath Data Transfer Trial Corrie Kost, Steve McDonald (TRIUMF) Bryan Caron (Alberta), Wade Hong (Carleton)

Slides:

Advertisements

Similar presentations

TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot

Advertisements

DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)

Premio Predator G2 Workstation Training

The International Grid Testbed: a 10 Gigabit Ethernet success story in memoriam Bob Dobinson GNEW 2004, Geneva Catalin Meirosu on behalf of the IGT collaboration.

UCLPv1 demo at JT 2005 Table of Contents >The Users (UCLP) >Users’ needs – 9k APP Project >How do users achieve their needs? UCLP (Control LightPaths)

Linux Clustering A way to supercomputing. What is Cluster? A group of individual computers bundled together using hardware and software in order to make.

End to end lightpaths for large file transfer over fast long-distance networks Jan 29, 2003 Bill St. Arnaud, Wade Hong, Geoff Hayward, Corrie Cost, Bryan.

Jumbo Packets for Application Performance Joint Techs Workshop Vancouver, BC 2005.

CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.

Teraserver Darrel Sharpe Matt Todd Rob Neff Mentor: Dr. Palaniappan.

DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)

CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.

PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.

ESLEA Bedfont Lakes Dec 04 Richard Hughes-Jones Network Measurement & Characterisation and the Challenge of SuperComputing SC200x.

CdL was here DataTAG CERN Sep 2002 R. Hughes-Jones Manchester 1 European Topology: NRNs & Geant SuperJANET4 CERN UvA Manc SURFnet RAL.

TRIUMF Site Report for HEPiX, SLAC, October 10-14,2005 TRIUMF SITE REPORT Corrie Kost Update since Hepix Spring 2005.

Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck Pavan Balaji  Hemal V. Shah ¥ D. K. Panda 

TRIUMF SITE REPORT – Corrie Kost April Catania (Italy) Update since last HEPiX/HEPNT meeting.

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02)

Sven Ubik, Petr Žejdl CESNET TNC2008, Brugges, 19 May 2008 Passive monitoring of 10 Gb/s lines with PC hardware.

Tools for TEIN2 operation Yoshitaka Hattori Jin Tanaka APAN-JP.

Atlas Canada Lightpath Data Transfer Trial

Terabyte IDE RAID-5 Disk Arrays David A. Sanders, Lucien M. Cremaldi, Vance Eschenburg, Romulus Godang, Christopher N. Lawrence, Chris Riley, and Donald.

© Copyright IBM Corporation 2006 Course materials may not be reproduced in whole or in part without the prior written permission of IBM IBM BladeCenter.

1 A Basic R&D for an Analysis Framework Distributed on Wide Area Network Hiroshi Sakamoto International Center for Elementary Particle Physics (ICEPP),

DELL PowerEdge 6800 performance for MR study Alexander Molodozhentsev KEK for RCS-MR group meeting November 29, 2005.

TNC 2007 Bandwidth-on-demand to reach the optimal throughput of media Brecht Vermeulen Stijn Eeckhaut, Stijn De Smet, Bruno Volckaert, Joachim Vermeir,

Online Systems Status Review of requirements System configuration Current acquisitions Next steps... Upgrade Meeting 4-Sep-1997 Stu Fuess.

Large File Transfer on 20,000 km - Between Korea and Switzerland Yusung Kim, Daewon Kim, Joonbok Lee, Kilnam Chon

Evaluation of the LDC Computing Platform for Point 2 SuperMicro X6DHE-XB, X7DB8+ Andrey Shevel CERN PH-AID ALICE DAQ CERN 10 October 2006.

“ PC  PC Latency measurements” G.Lamanna, R.Fantechi & J.Kroon (CERN) TDAQ WG –

HEPiX, CASPUR, April 3-7, 2006 – Steve McDonald Steven McDonald TRIUMF Network & Computing Services Canada’s National Laboratory.

1 Selecting LAN server (Week 3, Monday 9/8/2003) © Abdou Illia, Fall 2003.

Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.

EVN-NREN meeting, Schiphol, , A. Szomoru, JIVE Recent eVLBI developments at JIVE Arpad Szomoru Joint Institute for VLBI in Europe.

TRIUMF Site Report for HEPiX/HEPNT, Vancouver, Oct20-24/2003 – Corrie Kost TRIUMF SITE REPORT Corrie Kost Head Scientific Computing.

Wright Technology Corp. Minh Duong Tina Mendoza Tina Mendoza Mark Rivera.

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 1 Evaluation of the LDC Computing Platform for Point 2 SuperMicro X7DBE-X Andrey Shevel CERN PH-AID.

High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.

TRIUMF a TIER 1 Center for ATLAS Canada Steven McDonald TRIUMF Network & Computing Services iGrid 2005 – San Diego Sept 26 th.

NL Service Challenge Plans Kors Bos, Sander Klous, Davide Salomoni (NIKHEF) Pieter de Boer, Mark van de Sanden, Huub Stoffers, Ron Trompert, Jules Wolfrat.

Performance tests of storage arrays Irina Makhlyueva ALICE DAQ group 20 September 2004.

Internet data transfer record between CERN and California Sylvain Ravot (Caltech) Paolo Moroni (CERN)

Masaki Hirabaru NICT Koganei 3rd e-VLBI Workshop October 6, 2004 Makuhari, Japan Performance Measurement on Large Bandwidth-Delay Product.

TRIUMF Site Report – HEPiX/HEPNT – NIKHEF, Amsterdam, May 19-23/2003 TRIUMF Site Report HEPiX/HEPNT NIKHEF, Amsterdam May 19-23/2003 Corrie Kost.

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard.

PFLDNet Workshop February 2003 R. Hughes-Jones Manchester Some Performance Measurements Gigabit Ethernet NICs & Server Quality Motherboards Richard Hughes-Jones.

IDE disk servers at CERN Helge Meinhard / CERN-IT CERN OpenLab workshop 17 March 2003.

TERENA Networking Conference, Zagreb, Croatia, 21 May 2003 High-Performance Data Transport for Grid Applications T. Kelly, University of Cambridge, UK.

Weekly Report By: Devin Trejo Week of June 21, 2015-> June 28, 2015.

Sep. 17, 2002BESIII Review Meeting BESIII DAQ System BESIII Review Meeting IHEP · Beijing · China Sep , 2002.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.

Linux IDE Disk Servers Andrew Sansum 8 March 2000.

COMPASS Computerized Analysis and Storage Server Iain Last.

SA1 operational policy training, Athens 20-21/01/05 Presentation of the HG Node “Isabella” and operational experience Antonis Zissimos Member of ICCS administration.

$ 1000 COMPUTER. EVGA 132-CK-NF79-A1 NVIDIA nForce 790i Ultra SLI Socket 775 ATX MB w/RAID, 3-Way SLI, DDR3 & Core 2 Extreme Support Supports up to 8.

Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.

L1/HLT trigger farm Bologna setup 0 By Gianluca Peco INFN Bologna Genève,

HP Proliant Server  Intel Xeon E3-1220v3 (3.1GHz / 4-core / 8MB / 80W).  HP 4GB Dual Rank x8 PC E (DDR3-1600) Unbuffered Memory Kit.  HP Ethernet.

Intel Corporation LGA775 Socket Options Jim Rimpleman.

Recent experience with PCI-X 2.0 and PCI-E network interfaces and emerging server systems Yang Xia Caltech US LHC Network Working Group October 23, 2006.

DataGrid WP7 Meeting Jan 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress)

CAnet 4 LHC network resources US LHC network working group meeting Oct. 23, 2006 Thomas Tam CANARIE Inc.

MB MPLS MPLS Technical Meeting Sep 2001 R. Hughes-Jones Manchester SuperJANET Development Network Testbed – Cisco GSR SuperJANET4 C-PoP – Cisco GSR.

NL Service Challenge Plans

Cluster Active Archive

R. Hughes-Jones Manchester

DataTAG Project update

MB-NG Review High Performance Network Demonstration 21 April 2004

High-Performance Data Transport for Grid Applications

Presentation transcript:

Atlas Canada Lightpath Data Transfer Trial Corrie Kost, Steve McDonald (TRIUMF) Bryan Caron (Alberta), Wade Hong (Carleton)

The iGrid2002 Network

TRIUMF Local Loop Only one SM fiber pair available between TRIUMF and the CA*net 4 ONS at the BCnet POP CWDM gear arrived at TRIUMF on Sept 4th Extreme 10GbE LR blades arrived on the same day

CERN Local Loop Extreme BD with LR Blade arrives on Sept 17 within 2 days of shipping ONS finally installed and operational on Sept 20th

We are live continent to continent! e2e lightpath up and running 21:45 CET traceroute to cern-10g ( ), 30 hops max, 38 byte packets 1 cern-10g ( ) ms ms ms

The Network Hardware At CERN and TRIUMF –Intel 10GbE Server NICs (alpha) –Extreme Networks Black Diamond 6808 At TRIUMF –MRV Nbase Xyplex Fiber Driver WDM44s to multiplex 2GbEs to CA*net 4 ONS at BCnet POP

Intel 10GbE Cards Intel kindly loaned us 2 of their Pro/10GbE LR server adapters cards despite the end of their Alpha program –based on Intel® 82597EX 10 Gigabit Ethernet Controller

Extreme Networks Extreme Networks generously loaned and shipped to us 2 Black Diamond 6808s with their new10GbE LRi blades

Extreme Networks TRIUMF CERN

The Endpoint Hosts Linux-based PC Servers –SuperMicro P4DL6 (Dual Xeon 2GHz) –ServerWorks GC-LE Chipset –400 MHz front side bus –1 GB DDR2100 RAM –Dual Channel Ultra 160 onboard SCSI –SysKonnect 9843 SX GbE –3 independent PCI buses –PCI-X 64 bit/133 Mhz capable

IDE Disk Arrays CERN Receive HostTRIUMF Send Host

Operating System Redhat 7.3 based Linux kernel –Needed to support filesystems > 1TB Upgrades and patches –Patched to –Intel Pro 10GbE Linux driver (early stable) –SysKonnect 9843 SX Linux driver (latest) –Ported Sylvain Ravot’s tcp tune patches

BBFTP Transfer Vancouver ONS ons-van01(enet_15/1) Vancouver ONS ons-van01(enet_15/2)

BBFTP Transfer Chicago ONS GigE Port 1 Chicago ONS GigE Port 2

Tsunami Transfer Vancouver ONS ons-van01(enet_15/1) Vancouver ONS ons-van01(enet_15/2)

Tsunami Transfer Chicago ONS GigE Port 1 Chicago ONS GigE Port 2

Sunday Nite Summaries

Tuesday Nite Summaries Vancouver ONS ons-van01(enet_15/1) Vancouver ONS ons-van01(enet_15/2)

Comparative Results Tool Transferred Average Max Avg wuftp 100 MbE600 MB 3.4 Mbps wuftp 10 GbE6442 MB 71 Mbps iperf275 MB940 Mbps1136 Mbps pftp600MB532 Mbps bbftp (10 streams)1.4 TB666 Mbps710 Mbps Tsunami - disk to disk0.5 TB700 Mbps825 Mbps Tusnami - disk to memory12 GB> 1GBps

Lessons Learned Linux software RAID faster than most conventional SCSI based systems More disk spindles the better Channel bonding of two GbEs seems to work very well (on an unshared link) The larger files the better for thruput

Acknowledgements Canarie –Bill St. Arnaud, Rene Hatem, Damir Pobric, Thomas Tam, Jun Jian Atlas Canada –Mike Vetterli, Randall Sobie, Jim Pinfold, Pekka Sinervo, Gerald Oakham, Bob Orr, Michel Lefebrve, Richard Keeler HEPnet Canada –Dean Karlen TRIUMF –Peter Gumplinger, Fred Jones, Mike Losty, Jack Chakhalian, Renee Poutissou BCNET –Mike Hrybyk, Marilyn Hay, Dennis O’Reilly, Don McWilliams

Acknowledgements Extreme Networks –Amyn Pirmohamed, Steven Flowers, John Casselman, Darrell Clarke, Rob Bazinet, Damaris Soellner Intel Corporation –Hugues Morin, Caroline Larson, Peter Molnar, Harrison Li, Layne Flake, Jesse Brandeburg

Acknowledgements Indiana University –Mark Meiss, Stephen Wallace Caltech –Sylvain Ravot, Harvey Neuman CERN –Olivier Martin, Paolo Moroni, Martin Fluckiger, Stanley Cannon, J.P Martin-Flatin SURFnet/Universiteit van Amsterdam –Pieter de Boer, Dennis Paus, Erik.Radius, Erik-Jan.Bos, Leon Gommans, Bert Andree, Cees de Laat

Acknowledgements Yotta –Geoff Hayward, Reg Joseph, Ying Xie, E. Siu BCIT –Bill Rutherford Jalaam –Loki Jorgensen Netera –Gary Finley