1 High performance Throughput Les Cottrell – SLAC Lecture # 5a presented at the 26 th International Nathiagali Summer College on Physics and Contemporary.

Slides:



Advertisements
Similar presentations
Martin Suchara, Ryan Witt, Bartek Wydrowski California Institute of Technology Pasadena, U.S.A. TCP MaxNet Implementation and Experiments on the WAN in.
Advertisements

TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot
Congestion Control An Overview -Jyothi Guntaka. Congestion  What is congestion ?  The aggregate demand for network resources exceeds the available capacity.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 5, 2001.
Congestion Control Tanenbaum 5.3, /12/2015Congestion Control (A Loss Based Technique: TCP)2 What? Why? Congestion occurs when –there is no reservation.
Explicit Congestion Notification ECN Tilo Hamann Technical University Hamburg-Harburg, Germany.
High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.
Katz, Stoica F04 EECS 122: Introduction to Computer Networks Performance Modeling Computer Science Division Department of Electrical Engineering and Computer.
UCB Implementing QoS Jean Walrand EECS. UCB Outline What? Bandwidth, Delay Where? End-to-End, Edge-to-Edge, Edge-to-End, Overlay Mechanisms Access Control.
1 Achieving high performance throughput in production networks Les Cottrell – SLAC Presented at the Internet 2 HENP Networking Working Group kickoff meeting.
1 Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,
Internet and Intranet Protocols and Applications Section V: Network Application Performance Lecture 11: Why the World Wide Wait? 4/11/2000 Arthur P. Goldberg.
Internet Bandwidth Measurement Techniques Muhammad Ali Dec 17 th 2005.
Transport Level Protocol Performance Evaluation for Bulk Data Transfers Matei Ripeanu The University of Chicago Abstract:
Ns Simulation Final presentation Stella Pantofel Igor Berman Michael Halperin
MB - NG MB-NG Meeting UCL 17 Jan 02 R. Hughes-Jones Manchester 1 Discussion of Methodology for MPLS QoS & High Performance High throughput Investigations.
Introduction 1 Lecture 14 Transport Layer (Congestion Control) slides are modified from J. Kurose & K. Ross University of Nevada – Reno Computer Science.
The Effects of Systemic Packets Loss on Aggregate TCP Flows Thomas J. Hacker May 8, 2002 Internet 2 Member Meeting.
Lect3..ppt - 09/12/04 CIS 4100 Systems Performance and Evaluation Lecture 3 by Zornitza Genova Prodanoff.
KEK Network Qi Fazhi KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS.
Experiences in Design and Implementation of a High Performance Transport Protocol Yunhong Gu, Xinwei Hong, and Robert L. Grossman National Center for Data.
Raj Jain The Ohio State University R1: Performance Analysis of TCP Enhancements for WWW Traffic using UBR+ with Limited Buffers over Satellite.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 26, 2001.
Sharing Information across Congestion Windows CSE222A Project Presentation March 15, 2005 Apurva Sharma.
Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.
1 Using Netflow data for forecasting Les Cottrell SLAC and Fawad Nazir NIIT, Presented at the CHEP06 Meeting, Mumbai India, February
Congestion Control - Supplementary Slides are adapted on Jean Walrand’s Slides.
1 Overview of IEPM-BW - Bandwidth Testing of Bulk Data Transfer Tools Connie Logg & Les Cottrell – SLAC/Stanford University Presented at the Internet 2.
HighSpeed TCP for High Bandwidth-Delay Product Networks Raj Kettimuthu.
1 BWdetail: A bandwidth tester with detailed reporting Masters of Engineering Project Presentation Mark McGinley April 19, 2007 Advisor: Malathi Veeraraghavan.
Scavenger performance Cern External Network Division - Caltech Datagrid WP January, 2002.
Iperf Quick Mode Ajay Tirumala & Les Cottrell. Sep 12, 2002 Iperf Quick Mode at LBL – Les Cottrell & Ajay Tirumala Iperf QUICK Mode Problem – Current.
NET100 Development of network-aware operating systems Tom Dunigan
National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Web100 Basil Irwin & George Brett.
Transport Layer 3-1 Chapter 3 Transport Layer Computer Networking: A Top Down Approach 6 th edition Jim Kurose, Keith Ross Addison-Wesley March
Ethernet. Ethernet standards milestones 1973: Ethernet Invented 1983: 10Mbps Ethernet 1985: 10Mbps Repeater 1990: 10BASE-T 1995: 100Mbps Ethernet 1998:
1 Passive and Active Monitoring on a High-performance Network Les Cottrell, Warren Matthews, Davide Salomoni, Connie Logg – SLAC
TCP transfers over high latency/bandwidth networks Internet2 Member Meeting HENP working group session April 9-11, 2003, Arlington T. Kelly, University.
Jennifer Rexford Fall 2014 (TTh 3:00-4:20 in CS 105) COS 561: Advanced Computer Networks TCP.
CSE Computer Networks Prof. Aaron Striegel Department of Computer Science & Engineering University of Notre Dame Lecture 19 – March 23, 2010.
SLAC Status, Les CottrellESnet International Meeting, Kyoto July 24-25, 2000 SLAC Update Les Cottrell & Richard Mount July 24, 2000.
Performance Engineering E2EpiPEs and FastTCP Internet2 member meeting - Indianapolis World Telecom Geneva October 15, 2003
30 June Wide Area Networking Performance Challenges Olivier Martin, CERN UK DTI visit.
1 Achieving high performance throughput in production networks Les Cottrell – SLAC Presented at the Internet 2 HENP Networking Working Group kickoff meeting.
Chapter 11.4 END-TO-END ISSUES. Optical Internet Optical technology Protocol translates availability of gigabit bandwidth in user-perceived QoS.
Internet Connectivity and Performance for the HEP Community. Presented at HEPNT-HEPiX, October 6, 1999 by Warren Matthews Funded by DOE/MICS Internet End-to-end.
NET100 Development of network-aware operating systems Tom Dunigan
TCP continued. Discussion – TCP Throughput TCP will most likely generate the saw tooth type of traffic. – A rough estimate is that the congestion window.
TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot
1 Experiences and results from implementing the QBone Scavenger Les Cottrell – SLAC Presented at the CENIC meeting, San Diego, May
TCP Traffic Characteristics—Deep buffer Switch
Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.
INDIANAUNIVERSITYINDIANAUNIVERSITY Status of FAST TCP and other TCP alternatives John Hicks TransPAC HPCC Engineer Indiana University APAN Meeting – Hawaii.
Peer-to-Peer Networks 13 Internet – The Underlay Network
Increasing TCP's CWND based on Throughput draft-you-iccrg-throughput-based-cwnd-increasing-00 Jianjie You IETF92 Dallas.
1 IEPM / PingER project & PPDG Les Cottrell – SLAC Presented at the NGI workshop, Berkeley, 7/21/99 Partially funded by DOE/MICS Field Work Proposal on.
1 FAST TCP for Multi-Gbps WAN: Experiments and Applications Les Cottrell & Fabrizio Coccetti– SLAC Prepared for the Internet2, Washington, April 2003
Network-aware OS DOE/MICS ORNL site visit January 8, 2004 ORNL team: Tom Dunigan, Nagi Rao, Florence Fowler, Steven Carter Matt Mathis Brian.
Samuel Wood Manikandan Punniyakotti Supervisors: Brad Smith, Katia Obraczka, JJ Garcia-Luna-Aceves
CS450 – Introduction to Networking Lecture 19 – Congestion Control (2)
Fast Pattern-Based Throughput Prediction for TCP Bulk Transfers
Transport Protocols over Circuits/VCs
High Speed File Replication
Using Netflow data for forecasting
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
Wide Area Networking at SLAC, Feb ‘03
CS Lecture 2 Network Performance
Wide-Area Networking at SLAC
TCP flow and congestion control
Presentation transcript:

1 High performance Throughput Les Cottrell – SLAC Lecture # 5a presented at the 26 th International Nathiagali Summer College on Physics and Contemporary Needs, 25 th June – 14 th July, Nathiagali, Pakistan Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2 How to measure Selected about a dozen major collaborator sites in California, Colorado, Illinois, FR, CH, UK over last 9 months –Of interest to SLAC –Can get logon accounts Use iperf –Choose window size and # parallel streams –Run for 10 seconds together with ping (loaded) –Stop iperf, run ping (unloaded) for 10 seconds –Change window or number of streams & repeat Record streams, window, throughput (Mbits/s), loaded & unloaded ping responses

3 Default window size SLAC to CERN thruput vs windows & streams Hi-perf = big windows & multiple streams Improves ~ linearly with streams for small windows 8kB 16kB 32kB 100kB 1MB 64kB

4 E.g. thruput vs windows & streams ANL Colorado IN2P3, FR CERN, CH Caltech Window Mbits/s Streams I NFN, IT Mbits/s Daresbury, UK Mbits/s

5 Progress towards goal: 100 Mbytes/s Site-to-Site Focus on SLAC – Caltech over NTON; Using NTON wavelength division fibers up & down W. Coast US; Replaced Exemplar with 8*OC3 & Suns with Pentium IIIs & OC12 (622Mbps) SLAC Cisco with OC48 (2.4Gbps) and 2 × OC12; Caltech Juniper M160 & OC48 ~500 Mbits/s single stream achieved recently over OC12.

6 SC2000 WAN Challenge SC2000, Dallas to SLAC RTT ~ 48msec –SLAC/FNAL booth: Dell PowerEdge PIII 2 * 550MHz with 64bit PCI + Dell 850MHz both running Linux, each with GigE, connected to Cat 6009 with 2GigE bonded to Extreme SC2000 floor switch –NTON: OC48 to GSR to Cat 5500 Gig E to Sun E4500 4*460MHz and Sun E4500 6*336MHz Internet 2: 300 Mbits/s NTON 960Mbits/s Details: –www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.htmlwww-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html

7 Iperf throughput conclusions 1/2 Can saturate bottleneck links For a given iperf measurement, streams share throughput equally. For small window sizes throughput increases linearly with number of streams Predicted optimum window sizes can be large (> Mbyte) Need > 1 stream to get optimum performance Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams Improvements of 5 to 60 in thruput by using multiple streams & larger windows Loss not sensitive to throughput

8 Iperf thruput conclusions 2/2 For fixed streams*window product, streams are more effective than window size: There is an optimum number of streams above which performance flattens out See www-iepm.slac.stanford.edu/monitoring/bulk/www-iepm.slac.stanford.edu/monitoring/bulk/ 4.6Mbits/s864kBCaltech 1.7Mbits/s2256kBCaltech 26.8Mbits/s864kBCERN 9.45Mbits/2256kBCERN ThroughputStreamsWindowSite

9 Network Simulator (ns-2) From UCB, simulates network –Choice of stack (Reno, Tahoe, Vegas, SACK…) –RTT, bandwidth, flows, windows, queue lengths … Compare with measured results –Agrees well –Confirms observations (e.g. linear growth in throughput for small window sizes as increase number of flows)

10 Agreement of ns2 with observed

11 Ns-2 thruput & loss predict Indicates on unloaded link can get 70% of available bandwidth without causing noticeable packet loss Can get over 80-90% of available bandwidth Can overdrive: no extra throughput BUT extra loss 90%

12 Simulator benefits No traffic on network (nb throughput can use 90%) Can do what if experiments No need to install iperf servers or have accounts No need to configure host to allow large windows BUT –Need to estimate simulator parameters, e.g. RTT use ping or synack Bandwidth, use pchar, pipechar etc., moderately accurate AND its not the real thing –Need to validate vs. observed data –Need to simulate cross-traffic etc

13 Impact of cross-traffic on Iperf between SLAC & GSFC/ Maryland SCP HTTP bbftp iperf All TCP traffic Iperf port traffic To SLAC From SLAC

14 Impact on Others Make ping measurements with & without iperf loading –Loss loaded(unloaded) –RTT

15 Impact of applying QoS Defined 3 classes of service, application marked packets: –Scavenger service (1%), Best effort, & Priority service (30%) –Used DiffServ features in Cisco 7507 with DS3 link Appears to work as expected Measurements made by Dave Hartzell, of GreatPlains net, May 01

16 Improvements for major International BaBar sites Throughput improvements of 1 to 16 times in a year Links are being improved: ESnet, PHYnet, GARR, Janet, TEN-155 Improvements to come: IN2P3 => 155Mbps RAL => 622Mbps

17 Gigabit/second networking The start of a new era: –Very rapid progress towards 10Gbps networking in both the Local (LAN) and Wide area (WAN) networking environments are being made. –40Gbps is in sight on WANs, but what after? –The success of the LHC computing Grid critically depends on the availability of Gbps links between CERN and LHC regional centers. What does it mean? –In theory: 1GB file transferred in 11 seconds over a 1Gbps circuit (*) 1TB file transfer would still require 3 hours and 1PB file transfer would require 4 months –In practice: major transmission protocol issues will need to be addressed (*) according to the 75% empirical rule CERN

18 Very high speed file transfer (1) –High performance switched LAN assumed: requires time & money. –High performance WAN also assumed: also requires money but is becoming possible. very careful engineering mandatory. –Will remain very problematic especially over high bandwidth*delay paths: Might force the use Jumbo Frames because of interactions between TCP/IP and link error rates. –Could possibly conflict with strong security requirements CERN

19 CERN Very high speed file transfer (2) Following formula proposed by Matt Mathis/PSC (“The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm”) to approximate the maximum TCP throughput under periodic packet loss: (MSS/RTT)*(1/sqrt(p)) where MSS is the maximum segment size, 1460 bytes, in practice,and “p” is the packet loss rate. Are TCP's "congestion avoidance" algorithms compatible with high speed, long distance networks. –The "cut transmit rate in half on single packet loss and then increase the rate additively (1 MSS by RTT)" algorithm may simply not work. –New TCP/IP adaptations may be needed in order to better cope with “lfn”, e.g. TCP Vegas

20 Acceptable link error rates CERN

21 Very high speed file transfer (tentative conclusions) Tcp/ip fairness only exist between similar flows, i.e. similar duration, similar RTTs. Tcp/ip congestion avoidance algorithms need to be revisited (e.g. Vegas rather than Reno/NewReno) –faster recovery after loss, selective acknowledgment. Current ways of circumventing the problem, e.g. –Multi-stream & parallel socket just bandages or the practical solution to the problem? Web100, a 3MUSD NSF project, might help enormously! better TCP/IP instrumentation (MIB), will allow read/write to internal TCP parameters self-tuning tools for measuring performance improved FTP implementation applications can tune stack Non-Tcp/ip based transport solution, use of Forward Error Corrections (FEC), Early Congestion Notifications (ECN) rather than active queue management techniques (RED/WRED)? CERN

22 Optimizing streams Choose # streams to optimize throughput/impact –Measure RTT from Web100 –App controls # streams

23 WAN thruput conclusions High FTP performance across WAN links is possible –Even with 20-30Mbps bottleneck can do > 100Gbytes/day OS must support big windows selectable by application Need multiple parallel streams Loss is important in particular interval between losses Compression looks promising, but needs cpu power Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams Improvements of 5 to 60 in thruput by using multiple streams & larger windows Impacts others users, need Less than Best Effort QoS service

24 More Information This talk: – IEPM/PingER home site –www-iepm.slac.stanford.edu/www-iepm.slac.stanford.edu/ Transfer tools: – TCP Tuning: –

25 High Speed Bulk Throughput Driven by: –Data intensive science, e.g. data grids –HENP data rates, e.g. BaBar 300TB/year, collection doubling yearly, i.e. PBytes in couple of years –Data rate from experiment ~ 20MBytes/s ~ 200GBytes/d –Multiple regional computer centers (e.g. Lyon-FR, RAL-UK, INFN-IT, LBNL-CA, LLNL-CA, Caltech-CA) need copies of data –Boeing 747 high throughput, BUT poor latency (~ 2 weeks) & very people intensive So need high-speed networks and ability to utilize –High speed today = few hundred GBytes/day (100GB/d ~ 10Mbits/s) Data vol Moore’s law