National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Web100 Basil Irwin & George Brett.

Slides:



Advertisements
Similar presentations
Wendy Huntoon - PSC Jim Ferguson - NCSA I2 Members Meeting May 2002
Advertisements

Shouldnt High-Performance Networks Be As Easy To Use As the Web? Basil Irwin Senior Network Engineer NETS July 13, 1999 National Center for Atmospheric.
Autotuning in Web100 John W. Heffner August 1, 2002 Boulder, CO.
Pushing Up Performance for Everyone Matt Mathis 7-Dec-99.
Kathy Benninger, Pittsburgh Supercomputing Center Workshop on the Development of a Next-Generation Cyberinfrastructure 1-Oct-2014 NSF Collaborative Research:
Restricted Slow-Start for TCP William Allcock 1,2, Sanjay Hegde 3 and Rajkumar Kettimuthu 1,2 1 Argonne National Laboratory 2 The University of Chicago.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 5, 2001.
Summary Role of Software (1 slide) ARCS Software Architecture (4 slides) SNS -- Caltech Interactions (3 slides)
Congestion Control Tanenbaum 5.3, /12/2015Congestion Control (A Loss Based Technique: TCP)2 What? Why? Congestion occurs when –there is no reservation.
Explicit Congestion Notification ECN Tilo Hamann Technical University Hamburg-Harburg, Germany.
1 Chapter 3 Transport Layer. 2 Chapter 3 outline 3.1 Transport-layer services 3.2 Multiplexing and demultiplexing 3.3 Connectionless transport: UDP 3.4.
14 July 2000TWIST George Brett NLANR Distributed Applications Support Team (NCSA/UIUC)
1 Emulating AQM from End Hosts Presenters: Syed Zaidi Ivor Rodrigues.
Data Sources & Using VIVO Data Visualizing Scholarship VIVO provides network analysis and visualization tools to maximize the benefits afforded by the.
Performance Diagnostic Research at PSC Matt Mathis John Heffner Ragu Reddy 5/12/05 PathDiag ppt.
Lect3..ppt - 09/12/04 CIS 4100 Systems Performance and Evaluation Lecture 3 by Zornitza Genova Prodanoff.
IPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Collaborating with iPlant.
Development of network-aware operating systems Tom Dunigan
Advanced Network Architecture Research Group 2001/11/149 th International Conference on Network Protocols Scalable Socket Buffer Tuning for High-Performance.
NLANR, Internet2, and End-to-End performance Scot Colburn National Center for Atmospheric Research May Florianópolis, Brasil.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 26, 2001.
Relationships July 9, Producers and Consumers SERI - Relationships Session 1.
 End to End Performance Initiative Russ Hobby HENP Networking Working Group Meeting, Ann Arbor, Michigan 26 October 2001.
1 27-Apr-2004AstroGrid Consortium Meeting - PL talk Agenda Tuesday status review / goals / science Lunch + AGLI Itn-05 review Dinner at Coton (share.
Advanced Network Architecture Research Group 2001/11/74 th Asia-Pacific Symposium on Information and Telecommunication Technologies Design and Implementation.
1 BWdetail: A bandwidth tester with detailed reporting Masters of Engineering Project Presentation Mark McGinley April 19, 2007 Advisor: Malathi Veeraraghavan.
NET100 Development of network-aware operating systems Tom Dunigan
UT-BATTELLE U.S. Department of Energy Oak Ridge National Laboratory Net100 PIs: Wendy Huntoon/PSC, Tom Dunigan/ORNL, Brian Tierney/LBNL Impact and Connections.
TCP performance Sven Ubik FTP throughput capacity load ftp.uninett.no 12.3 Mb/s 1.2 Gb/s 80 Mb/s (6.6%) ftp.stanford.edu 1.3 Mb/s 600.
Commodity Grid Kits Gregor von Laszewski (ANL), Keith Jackson (LBL) Many state-of-the-art scientific applications, such as climate modeling, astrophysics,
The HMO Research Network (HMORN) is a well established alliance of 18 research departments in the United States and Israel. Since 1994, the HMORN has conducted.
NET100 … as seen from ORNL Tom Dunigan November 8, 2001.
NET100 Development of network-aware operating systems Tom Dunigan
NEES Cyberinfrastructure Center at the San Diego Supercomputer Center, UCSD George E. Brown, Jr. Network for Earthquake Engineering Simulation Analyzing.
Transport Layer 3-1 Chapter 3 Transport Layer Computer Networking: A Top Down Approach 6 th edition Jim Kurose, Keith Ross Addison-Wesley March
1. Introduction REU 2006-Packet Loss Distributions of TCP using Web100 Zoriel M. Salado, Mentors: Dr. Miguel A. Labrador and Cesar D. Guerrero 2. Methodology.
The TCP-ESTATS-MIB Matt Mathis John Heffner Raghu Reddy Pittsburgh Supercomputing Center Rajiv Raghunarayan Cisco Systems J. Saperia JDS Consulting, Inc.
1 Evaluating NGI performance Matt Mathis
Jennifer Rexford Fall 2014 (TTh 3:00-4:20 in CS 105) COS 561: Advanced Computer Networks TCP.
Web100 Basil Irwin National Center for Atmospheric Research Matt Mathis Pittsburgh Supercomputing Center Halloween, 2000.
End-to-End Performance Initiative March 9, 2001 Internet2 Spring Member Meeting.
National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Web100 Roll Out I2 Members Meeting.
Performance Engineering E2EpiPEs and FastTCP Internet2 member meeting - Indianapolis World Telecom Geneva October 15, 2003
Cyberinfrastructure: Many Things to Many People Russ Hobby Program Manager Internet2.
End-to-End Performance Initiative Russ Hobby May 30, 2001 Internet2 Campus Workshop Atlanta, Georgia.
NET100 Development of network-aware operating systems Tom Dunigan
NSF Middleware Initiative Purpose To design, develop, deploy and support a set of reusable, expandable set of middleware functions and services that benefit.
TCP continued. Discussion – TCP Throughput TCP will most likely generate the saw tooth type of traffic. – A rough estimate is that the congestion window.
TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot
National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Web100 and Logistical Networking.
UT-BATTELLE U.S. Department of Energy Oak Ridge National Laboratory Net100: developing network-aware operating systems New (9/01) DOE-funded (Office of.
© 2015 Pittsburgh Supercomputing Center Opening the Black Box Using Web10G to Uncover the Hidden Side of TCP CC PI Meeting Austin, TX September 29, 2015.
UT-BATTELLE U.S. Department of Energy Oak Ridge National Laboratory Net100 year 1 leftovers (proposal): PSC –none ORNL –router access to SNMP data (besides.
Network-aware OS DOE/MICS ORNL site visit January 8, 2004 ORNL team: Tom Dunigan, Nagi Rao, Florence Fowler, Steven Carter Matt Mathis Brian.
Samuel Wood Manikandan Punniyakotti Supervisors: Brad Smith, Katia Obraczka, JJ Garcia-Luna-Aceves
Advanced Network Diagnostic Tools Richard Carlson EVN-NREN workshop.
Internet2 Spring 2001 Washington, DC March 9, 2001 George Brett NLANR Distributed Applications Support Team (NCSA/UIUC) End-to-End Applications.
Data Sources & Using VIVO Data Visualizing Science VIVO provides network analysis and visualization tools to maximize the benefits afforded by the data.
Chapter 3 outline 3.1 transport-layer services
Transport Protocols over Circuits/VCs
Introducing Novell IPv6 Stack
Network Technology Update
Understanding Throughput & TCP Windows
Lecture 19 – TCP Performance
Automatic TCP Buffer Tuning
Sven Ubik TCP performance Sven Ubik
A tool for locating QoS failures on an Internet path
Transport Layer: Congestion Control
Anant Mudambi, U. Virginia
Presentation transcript:

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Web100 Basil Irwin & George Brett Web100 Project

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications What Is Web100? A Software Suite which would: –Enable ordinary network users to attain full network data rates across all networks without requiring help from networking experts. –Automatically tune the network stack for underlying network environment. –Provide network diagnostic and measurement information.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Motivation From the July 1999 report from the Advanced Networking Infrastructure Needs in the Atmospheric and Related Sciences (ANINARS) Workshop “ FTP (or FTP-like) bulk data-transfer is the most important networking function used to construct applications in this scientific discipline, yet failure to achieve effective bandwidths equal to apparently available bandwidths is most evident with bulk data- transfer applications. A variety of host-software problems contributes to this failure, and [NSF] programs should be developed to help solve these problems.”

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Motivation Applications, researchers, end-hosts are still unable to fully utilize high performance network infrastructure. –Still common for end-user to be unable to exceed 3 Mbps without help from a networking expert.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Why Poor Performance ? –Network software (mostly TCP) optimized for low bandwidth -- not high bandwidth --environments. –Lack of effective instrumentation and tools to diagnose end-host performance issues. Web100 is designed to address both these issues

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Who Any Application that transfers large data sets. –Application Independent –Example Areas Atmospheric Sciences Astronomy High Energy Physics Biomedical Geo Sciences

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Technical Description Three critical areas: –Core Technologies –User Support –Vendor Liaison In order for the core technologies to be successful, they must be usable by the end-user and adopted by the vendor community.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Core Technologies OS Instrumentation –Instrumentation of TCP using a MIB (Management Information Base) is the foundation of Web100 Real-time per-TCP-session metrics Allow Autotuning within a host Identify bottlenecks between sender and receiver. Support measurement for end-user performance. Diagnostics for network engineers and system administrators.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Initial (Network) Applications –Autotuning automatically adjust TCP to achieve maximum throughput across all connections within the host. Will use TCP-MIB to get information and then set parameters for automatic, transparent and dynamic tuning of TCP stack. –Performance Measurement Tools Suite of host based measurement tools using TCP- MIB. –Real-time graphical display of application performance, diagnosis of network path, etc.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Interim Solution –FTP Optimized FTP by leveraging ongoing application tuning work at NCSA. Will not initially use Web100 TCP-MIB –uses existing socket API to adjust TCP tuning parameters to support the estimated delay bandwidth product. Provides necessary stop gap measure –Short-term: while TCP-MIB is being developed. –Long-term: for non-Linux users until Web100 methods are adopted by proprietary vendors

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Progress To Date –Prototype TCP-MIB Has been designed. –80+ Variables have been defined »Includes some (~6) Read-write Variables. –Web100 Software TCP-MIB implementation –Linux –Implements 60 TCP-MIB Variables – including read- write. Test Applications –GUI interface to read and display variables –Prototype autotuning application

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Sample Output ESTABLISHED PktsIn 1974 | PktsOut 1951 | Enabled: DataPktsIn 972 | DataPktsOut 1002 | SACK N AckPktsIn 1975 | AckPktsOut 949 | ECN N DataBytesIn | DataBytesOut | Timestamps N DupAcksIn 0 | PktsRetran 0 | | BytesRetran 0 | loss episodes 0 | cwnd | winscale rcvd 0 timeouts 0 | max cwnd | rwin rcvd TO after FR 0 | ssthresh 0 | max rwin rcvd | min ssthresh 0 | winscale sent 0 | max ssthresh 0 | rwin sent | | max rwin sent rto (ms) 20 | rtt (ms) 1 | mss 1448 | Rate min rto (ms) 20 | min rtt (ms) 0 | min mss 1448 | Out (kbps) 0.1 max rto (ms) 20 | max rtt (ms) 1 | max mss 1448 | In (kbps)

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Progress To Date Pre-alpha release –Halloween, 2000 –Only to a few selected developers to Test the implementation Test main concepts and TCP-MIB variables –See if others can develop tools that use information from the prototype.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications User Support Provided by NCSA –Liaison between the user community and the developers: Make sure Web100 software continues to address the needs of the user community. –In-depth support for Beta users. Developing long-term support for larger groups of users. –Technical documentation Aimed at user community.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Vender Relationships Acceptance of Web100 work by OS vendors is critical to success: –Cisco is already helping with funding. –IBM, Sun, and Microsoft are aware of the work –Plan to work with others Web100 conference planned: –Within the next 6 months. –Once demonstrable results are seen. –Vendors and Web100 collaborators.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Funding Start up funds: –$100K from Cisco as part of their University Research program. Long Term Funding: –NSF Grant, start date of 9/15/00 $2.9 M for 3 years to develop the core Web100 technologies, support the early users and work with vendors.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Collaboration Core Partners: –Technical design and software development Pittsburgh Supercomputing Center National Center for Atmospheric Research –User Support and GUI development National Center for Supercomputing Applications Early Users NGI developers and researchers to help test, debug and augment the core Web100 software suite.

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications Acknowledgements Administrative: –Wendy Huntoon (PAC) –Marla Meehl (NCAR) Technical development Team: –Matt Mathis, Jeff Semke, John Heffner (PSC) –Basil Irwin (NCAR) GUI development –John Estabrook (NCSA) User Support Services –Tanya Brethour, George Brett (NCSA)

National Center for Atmospheric Research Pittsburgh Supercomputing Center National Center for Supercomputing Applications For More Information