High Speed File Replication

Slides:



Advertisements
Similar presentations
Web100 at SLAC Presented at the Web100 Workshop, Boulder, CO, August 2002.
Advertisements

1 QoS on Best-effort IP Networks Les Cottrell – SLAC Presented at the Joint SG13/SG16 Workshop Panel.
TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot
Iperf Tutorial Jon Dugan Summer JointTechs 2010, Columbus, OH.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 5, 2001.
High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.
CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.
Katz, Stoica F04 EECS 122: Introduction to Computer Networks Performance Modeling Computer Science Division Department of Electrical Engineering and Computer.
Internet and Intranet Protocols and Applications Section V: Network Application Performance Lecture 11: Why the World Wide Wait? 4/11/2000 Arthur P. Goldberg.
Reduced TCP Window Size for VoIP in Legacy LAN Environments Nikolaus Färber, Bernd Girod, Balaji Prabhakar.
KEK Network Qi Fazhi KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS.
1 Monitoring Internet connectivity of Research and Educational Institutions Les Cottrell – SLAC/Stanford University Prepared for the workshop on “Developing.
1 Report from NGI Testbed meeting at Berkeley, Jul 21-22, 1999 Les Cottrell – SLAC,
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 26, 2001.
1 Network performance measurements Les Cottrell – SLAC Prepared for the ICFA-SCIC, CERN December 8, 2001 Partially funded by DOE/MICS Field Work Proposal.
Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.
1 Using Netflow data for forecasting Les Cottrell SLAC and Fawad Nazir NIIT, Presented at the CHEP06 Meeting, Mumbai India, February
1 Internet Performance Monitoring Update Les Cottrell & Warren Matthews – SLAC Presented at the.
Data transfer over the wide area network with a large round trip time H. Matsunaga, T. Isobe, T. Mashimo, H. Sakamoto, I. Ueda International Center for.
1 Grid Related Activities at Caltech Koen Holtman Caltech/CMS PPDG meeting, Argonne July 13-14, 2000.
1 ESnet/HENP Active Internet End-to-end Performance & ESnet/University performance Les Cottrell – SLAC Presented at the ESSC meeting Albuquerque, August.
1 Overview of IEPM-BW - Bandwidth Testing of Bulk Data Transfer Tools Connie Logg & Les Cottrell – SLAC/Stanford University Presented at the Internet 2.
1 High performance Throughput Les Cottrell – SLAC Lecture # 5a presented at the 26 th International Nathiagali Summer College on Physics and Contemporary.
HighSpeed TCP for High Bandwidth-Delay Product Networks Raj Kettimuthu.
Scavenger performance Cern External Network Division - Caltech Datagrid WP January, 2002.
1 Internet End-to-end Monitoring Project - Overview Les Cottrell – SLAC/Stanford University Partially funded by DOE/MICS Field Work Proposal on Internet.
IEPM. Warren Matthews (SLAC) Presented at the ESCC Meeting Miami, FL, February 2003.
DoE NGI Program PI Meeting, October 1999Particle Physics Data Grid Richard P. Mount, SLAC Particle Physics Data Grid Richard P. Mount SLAC Grid Workshop.
PPDGLHC Computing ReviewNovember 15, 2000 PPDG The Particle Physics Data Grid Making today’s Grid software work for HENP experiments, Driving GRID science.
1 Passive and Active Monitoring on a High-performance Network Les Cottrell, Warren Matthews, Davide Salomoni, Connie Logg – SLAC
SLAC Status, Les CottrellESnet International Meeting, Kyoto July 24-25, 2000 SLAC Update Les Cottrell & Richard Mount July 24, 2000.
BNL Service Challenge 3 Status Report Xin Zhao, Zhenping Liu, Wensheng Deng, Razvan Popescu, Dantong Yu and Bruce Gibbard USATLAS Computing Facility Brookhaven.
1 Internet Performance Monitoring for the HENP Community Les Cottrell & Warren Matthews – SLAC Presented.
GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.
Internet Connectivity and Performance for the HEP Community. Presented at HEPNT-HEPiX, October 6, 1999 by Warren Matthews Funded by DOE/MICS Internet End-to-end.
1 Experiences and results from implementing the QBone Scavenger Les Cottrell – SLAC Presented at the CENIC meeting, San Diego, May
Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.
INDIANAUNIVERSITYINDIANAUNIVERSITY Status of FAST TCP and other TCP alternatives John Hicks TransPAC HPCC Engineer Indiana University APAN Meeting – Hawaii.
1 PingER performance to Bangladesh Prepared by Les Cottrell, SLAC for Prof. Hilda Cerdeira May 27, 2004 Partially funded by DOE/MICS Field Work Proposal.
The EU DataTAG Project Richard Hughes-Jones Based on Olivier H. Martin GGF3 Frascati, Italy Oct 2001.
1 IEPM / PingER project & PPDG Les Cottrell – SLAC Presented at the NGI workshop, Berkeley, 7/21/99 Partially funded by DOE/MICS Field Work Proposal on.
1 FAST TCP for Multi-Gbps WAN: Experiments and Applications Les Cottrell & Fabrizio Coccetti– SLAC Prepared for the Internet2, Washington, April 2003
1 PingER6 Preliminary PingER Monitoring Results from the 6Bone/6REN. Warren Matthews Les Cottrell.
Toward a Measurement Infrastructure. Warren Matthews (SLAC) Presented at the e2e Workshop Miami, FL, February 2003.
iperf a gnu tool for IP networks
Accelerating Peer-to-Peer Networks for Video Streaming
Achieving high performance throughput in production networks
Fast Pattern-Based Throughput Prediction for TCP Bulk Transfers
Realization of a stable network flow with high performance communication in high bandwidth-delay product network Y. Kodama, T. Kudoh, O. Tatebe, S. Sekiguchi.
R. Hughes-Jones Manchester
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
Khiem Lam Jimmy Vuong Andrew Yang
Milestones/Dates/Status Impact and Connections
Grid Canada Testbed using HEP applications
Using Netflow data for forecasting
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
Wide Area Networking at SLAC, Feb ‘03
Experiences from SLAC SC2004 Bandwidth Challenge
My Experiences, results and remarks to TCP BW and CT Measurements Tools Jiří Navrátil SLAC.
CS Lecture 2 Network Performance
File Transfer Issues with TCP Acceleration with FileCatalyst
Cisco Prime NAM for WAN Optimization Deployment
Breaking the Internet2 Land Speed Record: Twice
Advanced Networking Collaborations at SLAC
IEPM. Warren Matthews (SLAC)
Beyond FTP & hard drives: Accelerating LAN file transfers
Wide-Area Networking at SLAC
PIPE Dreams Trouble Shooting Network Performance for Production Science Data Grids Presented by Warren Matthews at CHEP’03, San Diego March 24-28, 2003.
Summer 2002 at SLAC Ajay Tirumala.
Evaluation of Objectivity/AMS on the Wide Area Network
Presentation transcript:

High Speed File Replication Les Cottrell & Davide Salomoni – SLAC www.slac.stanford.edu/grp/scs/net/talk/thru-escc-nov00/ Presented at ESCC meeting, San Ramon, CA Nov. 2000 Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

Overview PPDG & replication High thruput measurements Methodology LAN WAN SC2000 Packet reordering Impact on others How do we measure the QoS Introduction to PingER and active end-to-end measurement methodology Problem areas illustrated by results from PingER: Generally, e.g. S. America, Spain, China, Germany to .edu & .ca How do E. Europe & Russia look? How does performance affect applications Validating ping measurements and impact on FTP & Web performance Overview of impact of performance on applications including email, web, FTP, interactive apps Detailed look at bulk data transfer expectations for HENP sites Detailed look at critical performance metrics (RTT, loss, jitter, availability) and impact on VoIP What can be done to improve QoS: More bandwidth Reserved bandwidth Differentiated services

Participants in PPDG & GriPhyN Projects Nets Caltech SLAC CalREN NTON ESNet Abilene SDSC MREN Fermilab Wisconsin Indiana Boston BNL JLAB ANL Florida LBNL/UCB Sites

Site to site replication service PRIMARY SITE Data Acquisition, CPU, Disk, Tape Robot SECONDARY SITE CPU, Disk, Tape Robot Network Protocols Tuned for High Throughput Use of DiffServ for (1) Predictable high priority delivery of high - bandwidth data streams (2) Reliable background transfers Use of integrated instrumentation to detect/diagnose/correct problems in long-lived high speed transfers [NetLogger + DoE/NGI developments] Coordinated reservaton/allocation techniques for storage-to-storage performance

Measurement methodology Iperf with multiple windows & streams Selected 10 sites Of critical interest to have high performance with SLAC Can get iperf servers installed Production links, do not control utilization

What does thruput depend on ? Bandwidth end to end, i.e. min(BWlinks) AKA bottleneck bandwidth Round Trip Time (RTT) For TCP keep pipe full window ~ RTT*BW Thruput ~ 1/(sqrt(loss)) Competing utilization Src Rcv RTT ACK Time Pipe=RTT*BW

LAN thruput vs windows & streams Hi-perf = big windows & multiple streams Default window size

LAN throughput measurements Sun/Solaris vs PIII/Linux

Progress towards goal: 100 Mbytes/s Site-to-Site Focus on SLAC – Caltech over NTON; Using NTON wavelength division fibers up & down W. Coast US; Replaced Exemplar with 8*OC3 & Suns with Pentium IIIs & OC12 (622Mbps) SLAC Cisco 12000 with OC48 (2.4Gbps) and 2 × OC12; Caltech Juniper M160 & OC48 ~500 Mbits/s single stream achieved recently over OC12.

Intercontinental high performance thruput on production networks SLAC (California US) to CERN (Switzerland)

SLAC to CERN thruput vs windows & streams Hi-perf = big windows & multiple streams 1MB Improves ~ linearly with streams for small windows 100kB 64kB 16kB 32kB 8kB Default window size

SLAC to CERN thruput vs windows & streams >8Mbps >10Mbps Window size (kB) <2Mbps Parallel streams Animate1 Animate2

E.g. thruput vs windows & streams ANL Caltech Colorado Window IN2P3, FR CERN, CH Mbits/s Daresbury, UK Streams I NFN, IT Mbits/s Mbits/s

Measured WAN thruput Poor agreement improvement w move to I2 Improve with time (upgrades)

Iperf throughput conclusions Pathchar does a poor job of predicting thruput at these rates Need > 1 stream Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams Improvements of 5 to 60 in thruput by using multiple streams & larger windows Increasing streams often more effective than increasing windows See www-iepm.slac.stanford.edu/monitoring/bulk/

File transfer + Compression Bbftp tool from Gilles Farrache, IN2P3 10 streams & 256 kByte window SLAC > Lyon, FR got 25Mbps with NO compression With compression cpu power is important Sun E4500 4 cpu @ 336Mhz best could do was 13.6Mbps with 5 streams, more streams go slower (e.g. 10 streams =>7.4Mbps) Sun E450 4 cpu @ 450MHz 26MHz with 10 streams Compression factor 2-3 times So data rate boosted to 27 – 41 Mbps (E4500) or 52 - 78 Mbps (E450)

SC2000 WAN Challenge SC2000, Dallas to SLAC RTT ~ 48msec SLAC/FNAL booth: Dell PowerEdge PIII 2 * 550MHz with 64bit PCI + Dell 850MHz both running Linux, each with GigE, connected to Cat 6009 with 2GigE bonded to SCInet Extreme switch NTON/SLAC: OC48 to GSR to Cat 5500 Gig E to Sun E4500 4*460MHz and Sun E4500 6*336MHz Internet 2: 300 Mbits/s NTON: measured 990Mbits/s ==200TBytes in 20 days (BaBar yearly production ==250MBytes in 2 seconds (12K BaBar events or 1% error stats) ==copy 10 minute QuickTime movie in ~ 1 second == 50K simultaneous VoIP calls (enough for 500 sites like SLAC)

Packet reordering Impacts TCP congestion avoidance algorithms Is more common than had been thought Took 256 PingER sites Measured with pings 5 * 1 sec separations 50 back-to-back Look for out of sequence

Impact on Others Make ping measurements with & without iperf loading Loss loaded(unloaded) RTT

Effect of load on other traffic Measured ping RTT for 60 secs on normally unloaded link with & without iperf load Difference ~30-50ms

Possible alleviation Less than Best Effort QoS Choose streams to optimize thruput/impact Measure RTT and use to control number of streams

WAN thruput conclusions High FTP performance across WAN links is possible Even with 20-30Mbps bottleneck can do > 100Gbytes/day OS must support big windows selectable by application Need multiple parallel streams Loss is important in particular interval between losses Compression looks promising, but needs cpu power Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams Improvements of 5 to 60 in thruput by using multiple streams & larger windows Impacts others users, need Less than Best Effort QoS service or friendlier applications

More Information This talk: IEPM/PingER home site Packet reordering www.slac.stanford.edu/grp/scs/net/talk/thru-escc-nov00/ IEPM/PingER home site www-iepm.slac.stanford.edu/ Bulk throughput measurements: www-iepm.slac.stanford.edu/monitoring/bulk/ Effect of load on thruput & loss www-iepm.slac.stanford.edu/monitoring/load/ Windows vs. streams www-iepm.slac.stanford.edu/monitoring/bulk/window-vs-streams.html SC2000 thruput to SLAC www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html Packet reordering www-iepm.slac.stanford.edu/monitoring/reorder/