Experiences from SLAC SC2004 Bandwidth Challenge

Slides:

Advertisements

Similar presentations

Current Testbed : 100 GE 2 sites (NERSC, ANL) with 3 nodes each. Each node with 4 x 10 GE NICs Measure various overheads from protocols and file sizes.

Advertisements

Shawn P. McKee University of Michigan University of Michigan UltraLight Meeting, NSF January 26, 2005 Network Working Group Report.

Presented by: Yash Gurung, ICFAI UNIVERSITY.Sikkim BUILDING of 3 R'sCLUSTER PARALLEL COMPUTER.

1 SLAC Site Report By Les Cottrell for UltraLight meeting, Caltech October 2005.

1 Testbeds Les Cottrell Site visit to SLAC by DoE program managers Thomas Ndousse & Mary Anne Scott April 27,

High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.

1 Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,

SC|05 Bandwidth Challenge ESCC Meeting 9th February ‘06 Yee-Ting Li Stanford Linear Accelerator Center ESCC Meeting 9th February ‘06 Yee-Ting Li Stanford.

ESLEA Bedfont Lakes Dec 04 Richard Hughes-Jones Network Measurement & Characterisation and the Challenge of SuperComputing SC200x.

1 High Performance WAN Testbed Experiences & Results Les Cottrell – SLAC Prepared for the CHEP03, San Diego, March 2003

1 Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell Sun SuperG Spring 2005, April,

Long Distance experiments of Data Reservoir system

10GbE WAN Data Transfers for Science High Energy/Nuclear Physics (HENP) SIG Fall 2004 Internet2 Member Meeting Yang Xia, HEP, Caltech

1 ESnet Network Measurements ESCC Feb Joe Metzger

TNC 2007 Bandwidth-on-demand to reach the optimal throughput of media Brecht Vermeulen Stijn Eeckhaut, Stijn De Smet, Bruno Volckaert, Joachim Vermeir,

Large File Transfer on 20,000 km - Between Korea and Switzerland Yusung Kim, Daewon Kim, Joonbok Lee, Kilnam Chon

J. Bunn, D. Nae, H. Newman, S. Ravot, X. Su, Y. Xia California Institute of Technology High speed WAN data transfers for science Session Recent Results.

J. Bunn, D. Nae, H. Newman, S. Ravot, X. Su, Y. Xia California Institute of Technology State of the art in the use of long distance network International.

UTA Site Report Jae Yu UTA Site Report 4 th DOSAR Workshop Iowa State University Apr. 5 – 6, 2007 Jae Yu Univ. of Texas, Arlington.

Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.

Summer School, Brasov, Romania, July 2005, R. Hughes-Jones Manchester 1 TCP/IP and Other Transports for High Bandwidth Applications TCP/IP on High Performance.

LambdaStation Monalisa DoE PI meeting September 30, 2005 Sylvain Ravot.

APAN 10Gbps End-to-End Performance Measurement Masaki Hirabaru (NICT), Takatoshi Ikeda (KDDI/NICT), and Yasuichi Kitamura (NICT) July 19, 2006 Network.

Shawn McKee University of Michigan University of Michigan UltraLight: A Managed Network Infrastructure for HEP CHEP06, Mumbai, India February 14, 2006.

SC04 Network Security Wrap-Up Version 3. Role of Network Security in SCinet ISP role/rule in protecting network (1) Protect network infrastructure (2)

End-to-End performance tuning Brian Davies Gridpp28 Manchester 2012.

First of ALL Big appologize for Kei’s absence Hero of this year’s LSR achievement Takeshi in his experiment.

Performance and Scalability of xrootd Andrew Hanushevsky (SLAC), Wilko Kroeger (SLAC), Bill Weeks (SLAC), Fabrizio Furano (INFN/Padova), Gerardo Ganis.

Monte Carlo Data Production and Analysis at Bologna LHCb Bologna.

Ultimate Integration Joseph Lappa Pittsburgh Supercomputing Center ESCC/Internet2 Joint Techs Workshop.

1 Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,

The Design and Demonstration of the UltraLight Network Testbed Presented by Xun Su GridNets 2006, Oct.

Networkshop March 2005 Richard Hughes-Jones Manchester Bandwidth Challenge, Land Speed Record, TCP/IP and You.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.

TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot

1 Experiences and results from implementing the QBone Scavenger Les Cottrell – SLAC Presented at the CENIC meeting, San Diego, May

Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.

S. Ravot, J. Bunn, H. Newman, Y. Xia, D. Nae California Institute of Technology CHEP 2004 Network Session September 1, 2004 Breaking the 1 GByte/sec Barrier?

-1- ESnet On-Demand Secure Circuits and Advance Reservation System (OSCARS) David Robertson Internet2 Joint Techs Workshop July 18,

Western Tier 2 Site at SLAC Wei Yang US ATLAS Tier 2 Workshop Harvard University August 17-18, 2006.

1 FAST TCP for Multi-Gbps WAN: Experiments and Applications Les Cottrell & Fabrizio Coccetti– SLAC Prepared for the Internet2, Washington, April 2003

PetaCache: Data Access Unleashed Tofigh Azemoon, Jacek Becla, Chuck Boeheim, Andy Hanushevsky, David Leith, Randy Melen, Richard P. Mount, Teela Pulliam,

Recent experience with PCI-X 2.0 and PCI-E network interfaces and emerging server systems Yang Xia Caltech US LHC Network Working Group October 23, 2006.

INFN Site Report R.Gomezel October 9-13, 2006 Jefferson Lab, Newport News.

PSC. BigBen Features Compute Nodes 2068 nodes running Catamount (QK) microkernel Seastar interconnect in a 3-D torus configuration No external.

UNM SCIENCE DMZ Sean Taylor Senior Network Engineer.

Joint Genome Institute

GLAST SLAC-NRL network, June 28 ‘06

Paola Grosso SLAC October

R. Hughes-Jones Manchester

5th DOSAR Workshop Louisiana Tech University Sept. 27 – 28, 2007

Networking between China and Europe

Link from SLAC to SC2003.

CENIC Road to Ten Gigabit: Biggest Fastest in the West

TransPAC HPCC Engineer

Procurements at CERN: Status and Plans

High Speed File Replication

ESnet Network Measurements ESCC Feb Joe Metzger

Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the

Wide Area Networking at SLAC, Feb ‘03

High Performance Active End-to-end Network Monitoring

Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,

LHC Tier 2 Networking BOF

Breaking the Internet2 Land Speed Record: Twice

Advanced Networking Collaborations at SLAC

Wide-Area Networking at SLAC

Link from SLAC to SC2003 Les Cottrell, SLAC.

GridTorrent Framework: A High-performance Data Transfer and Data Sharing Framework for Scientific Computing.

Jan. 24th, 2003 Kento Aida (TITECH) Sissades Tongsima (NECTEC)

Presentation transcript:

Experiences from SLAC SC2004 Bandwidth Challenge Les Cottrell, SLAC www.slac.stanford.edu/grp/scs/net/talk03/bwc-04.ppt

Sponsors/partners

SLAC/SC04 Bandwidth Challenge (plan C) 2 Sun Opteron/Chelsio-10GE SLAC/FNAL booth (2418) 6 Boston file servers 1 GE 2 Sun file server 1 GE 2 Sun Opteron/S2io-10GE Loaned Cisco Rtr 6 Sun Opteron/Chelsio-10GE 10Gbps from NLR (via SEA, DEN, CHI) SLAC Cisco Rtr NLR-PITT-SUNN-10GE-17 NLR-PITT-SUNN-10GE-17 Juniper T320 NLR demarc PSC SciNet NLR demarc 1 Sun file server 1 GE 15808 15808 15540 15454 ESnet/QWest OC192/SONET Sunnyvale/Level(3)/NLR 1 Sun Opteron/Chelsio-10GE 1380 Kifer Sunnyvale/Qwest/ESnet 1400 Kifer

SC2004: Tenth of a Terabit/s Challenge Joint Caltech, SLAC, FNAL, CERN, UF, SDSC, BR, KR, …. 10 10 Gbps waves to HEP on show floor Bandwidth challenge: aggregate throughput of 101.13 Gbps FAST TCP

Components 10 Gbps NICs v20z Chelsio SR XENPAK S2io 1982 10Mbps 3COM SVL/NLR 3510 disk array

Challenge aggregates from SciNet Aggregate Caltech & SLAC booth, in & out 7 lambdas to Caltech, 3 ro SLAC

Challenge aggregates from MonALISA Sustained ~ 10Gbps for extended periods

Weathermap showing 8.7Gbps on ESnet

To/From SLAC booth NLR: 9.43Gbps (9.07 goodput) + 5.65Gbps (5.44Gbps goodput) in reverse Two hosts to two hosts ESnet: 7.72Gbps (7.43Gbps goodput) Only one 10Gbps host at SVL Single V40Z host with 2*10GE NICs to 2*V20Z across country got 11.4Gbps S2io and Chelsio (& Cisco & Juniper) all interwork Chelsio worked stably on uncongested paths

TOE Chelsio had TCP Offload Engine Utilization factor of throughput & parallel streams Reduced cpu c.f. S2io non0TOE by factor ~ 3

Challenges Could not get 10Gbps waves to SLAC only SVL Equipment in 3 locations Keeping configs in lock-step (no NFS, no name service) Security concerns, used iptables Machines only available 2 weeks before, some not until we got to SC04 Jumbo frames not configured correctly at SLAC booth, used 1500B frames mainly Mix of hdw/swr: Opterons with various GHz & disks, Xeons; Solaris 10, Linux 2.4, 2.6 Coordination between booths (sep by 100 yds) Everything state of art (Linux 2.6.6, SR XENPAKs, NICs

Award Three to four times bandwidth of next challenger