Experiences Tuning Cluster Hosts 1GigE and 10GbE Paul Hyder Cooperative Institute for Research in Environmental Sciences, CU Boulder Cooperative Institute.

Slides:



Advertisements
Similar presentations
High Speed Total Order for SAN infrastructure Tal Anker, Danny Dolev, Gregory Greenman, Ilya Shnaiderman School of Engineering and Computer Science The.
Advertisements

August 10, Circuit TCP (CTCP) Helali Bhuiyan
CISCO NETWORKING ACADEMY Chabot College ELEC Address Resolution Protocol.
Implementing Inter-VLAN Routing
Precept 3 Host Configuration 1 Peng Sun. What TCP conn. running? Commands netstat [-n] [-p] [-c] (Linux) lsof -i -P (Mac) ss (newer version of netstat)
LCG TCP performance optimization for 10 Gb/s LHCOPN connections 1 on behalf of M. Bencivenni, T.Ferrari, D. De Girolamo, Stefano.
Transparent Caching The art of caching network traffic without requiring user / browser side configuration.
How’s My Network (HMN)? A Java approach to Home Network Measurement Alan Ritacco, Craig Wills, and Mark Claypool Computer Science Department Worcester.
DataTAG Meeting CERN 7-8 May 03 R. Hughes-Jones Manchester 1 High Throughput: Progress and Current Results Lots of people helped: MB-NG team at UCL MB-NG.
IP Routing: an Introduction. Quiz
Installing and Maintaining ISA Server. Planning an ISA Server Deployment Understand the current network infrastructure Review company security policies.
TCP/IP Tools Lesson 5. Objectives Skills/ConceptsObjective Domain Description Objective Domain Number Using basic TCP/IP commands Understanding TCP/IP3.6.
Click to edit Master subtitle style Chapter 17: Troubleshooting Tools Instructor:
CIS 193A – Lesson10 Protecting Your Network. CIS 193A – Lesson10 Focus Question What information contained in packets can be used as matching criteria.
© 2008 Cisco Systems, Inc. All rights reserved.Cisco ConfidentialPresentation_ID 1 Chapter 5: Inter-VLAN Routing Routing And Switching.
CN2668 Routers and Switches Kemtis Kunanuraksapong MSIS with Distinction MCTS, MCDST, MCP, A+
CCNA Guide to Cisco Networking Fundamentals Fourth Edition
KEK Network Qi Fazhi KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS.
CRON: Cyber-infrastructure for Reconfigurable Optical Networks PI: Seung-Jong Park, co-PI: Rajgopal Kannan GRA: Cheng Cui, Lin Xue, Praveenkumar Kondikoppa,
1 A Basic R&D for an Analysis Framework Distributed on Wide Area Network Hiroshi Sakamoto International Center for Elementary Particle Physics (ICEPP),
Hands-On Microsoft Windows Server 2003 Networking Chapter Three TCP/IP Architecture.
TNC 2007 Bandwidth-on-demand to reach the optimal throughput of media Brecht Vermeulen Stijn Eeckhaut, Stijn De Smet, Bruno Volckaert, Joachim Vermeir,
Cisco PIX firewall Set up 3 security zones ***CS580*** John Trafecanty Jules R. Nya Baweu August 23, 2005.
1 CSTS WG CSTS WG Prototyping for Forward CSTS Performance Boulder November 2011 Martin Karch.
1 Chapter Overview Using the New Connection Wizard to configure network and Internet connections Using the New Connection Wizard to configure outbound.
TCP/IP Essentials A Lab-Based Approach Shivendra Panwar, Shiwen Mao Jeong-dong Ryoo, and Yihan Li Chapter 5 UDP and Its Applications.
1 Chapter Overview Routing Principles. Understanding Routing--important Routing refers to the process of taking a packet from one device and sending it.
Internet and Intranet Fundamentals Class 9 Session A.
Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.
Hour 7 The Application Layer 1. What Is the Application Layer? The Application layer is the top layer in TCP/IP's protocol suite Some of the components.
11 NETWORK CONNECTION HARDWARE Chapter 3. Chapter 3: NETWORK CONNECTION HARDWARE2 NETWORK INTERFACE ADAPTER  Provides the link between a computer and.
High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.
Virtualization for the LHCb Online system CHEP Taipei Dedicato a Zio Renato Enrico Bonaccorsi, (CERN)
BNL’s Network diagnostic tool IPERF was used and combined with different strategies to analyze network bandwidth performance such as: -Test with iperf.
Agilent Technologies Copyright 1999 H7211A+221 v Capture Filters, Logging, and Subnets: Module Objectives Create capture filters that control whether.
Masaki Hirabaru NICT Koganei 3rd e-VLBI Workshop October 6, 2004 Makuhari, Japan Performance Measurement on Large Bandwidth-Delay Product.
Homework 02 NAT 、 DHCP 、 Firewall 、 Proxy. Computer Center, CS, NCTU 2 Basic Knowledge  DHCP Dynamically assigning IPs to clients  NAT Translating addresses.
Networking in Linux. ♦ Introduction A computer network is defined as a number of systems that are connected to each other and exchange information across.
Jose Luis Flores / Amel Walkinshaw
D-Link TSD 2009 workshop D-Link Net-Defends Firewall Training ©Copyright By D-Link HQ TSD Benson Wu.
WEEK 11 – TOPOLOGIES, TCP/IP, SHARING & SECURITY IT1001- Personal Computer Hardware System & Operations.
1 Interview Questions - What is the difference between TCP and UDP? - What is Nagle's Algorithm? - Describe the TCP handshaking process. - What is Slow.
ERICSON BRANDON M. BASCUG Alternate - REGIONAL NETWORK ADMINISTRATOR HOW TO TROUBLESHOOT TCP/IP CONNECTIVITY.
GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.
TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot
Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.
Connect communicate collaborate Performance Metrics & Basic Tools Robert Stoy, DFN EGI TF, Madrid September 2013.
SMOOTHWALL FIREWALL By Nitheish Kumarr. INTRODUCTION  Smooth wall Express is a Linux based firewall produced by the Smooth wall Open Source Project Team.
Instructor & Todd Lammle
NAT、DHCP、Firewall、FTP、Proxy
Click to edit Master subtitle style
Troubleshooting ip Chapter 5e.
CompTIA Network+ N Authorized Cert Guide
Networking & Wireless Routers II Driver Installation
Network Tools and Utilities
Selecting Unicast or Multicast Mode
Experiences Tuning Cluster Hosts
Enrico Bonaccorsi, (CERN) Loic Brarda, (CERN) Gary Moine, (CERN)
R. Hughes-Jones Manchester
Transport Protocols over Circuits/VCs
Introduction to Networking
Troubleshooting IP Communications
OpenFlow Switch as a low-impact Firewall
Unit 27: Network Operating Systems
Using the WUGS-20 GigE Line Card
TCP/IP Networking An Example
Troubleshooting ip Chapter 5e.
Achieving reliable high performance in LFNs (long-fat networks)
Evaluation of Objectivity/AMS on the Wide Area Network
Presentation transcript:

Experiences Tuning Cluster Hosts 1GigE and 10GbE Paul Hyder Cooperative Institute for Research in Environmental Sciences, CU Boulder Cooperative Institute for Research in Environmental Sciences, CU Boulder (CIRES at NOAA/ERSL/GSD High Performance Computing) (CIRES at NOAA/ERSL/GSD High Performance Computing) Paul.Hyder at noaa.gov Paul.Hyder at noaa.gov

Tuning Focus n Cluster Front Ends and Cron Server Hosts n File transfer servers (scponly) n BWCTL host n Remote client hosts n 10GbE Testbed (7.2 Gb/sec uses ~49% of one 3G CPU) (7.2 Gb/sec uses ~49% of one 3G CPU)

How We Apply the Well Known Rules n Jumbo Frames –8K on hosts –9K on network n Tune TCP to match BDP n Encourage application writers to use large read and write buffers n Install tuned Applications –PSC.edu patch to ssh OpenSSH:channels.h #define CHAN_TCP_PACKET_DEFAULT (32*1024) #define CHAN_TCP_WINDOW_DEFAULT (4*CHAN_TCP_PACKET_DEFAULT)

Throughput Testing n Iperf (2.0.2) from shell scripts –Vary buffer (-l) and window (-w) –Modify ifconfig and PCI configuration –Loop takes 3 days n Bwctl with remote hosts –?Anyone on NLR? n Use scp/sftp/rsync as final test

I’m Curious n How much TCP tuning information do you provide users and admins? n Are hosts being tuned? n Does your internal LAN support jumbo frames?

GSD Cluster GigE Defaults n [wr]mem_default 2MB n [wr]mem_max 16MB n ipv4/tcp_[wr]mem 64KB 2MB 16MB n optmem_max 512K n txqueuelen n netdev_max_backlog 3000 n ipv4/tcp_sack and ipv4/tcp_timestamps on n Don’t touch ipv4/tcp_mem

Jumbo Frame Plot

What doesn’t work n Jumbo Frames –Switch Fabrics n High density cards n Complex vLAN configurations n Stand alone GigE switches –Firewalls –ICMP for path mtu discovery n Disabled completely n Network devices don’t respond

Linux 2.6 and Jumbos IP hostA > hostB.22: S 544:544(0) win IP hostB.22 > hostA.52434: S 207:207(0) ack 545 win IP hostA > hostB.22:. 2255:6599(4344) ack 2293 win IP hostA > hostB.22: P 6599:10943(4344) ack 2293 win IP router > hostA: icmp 36: hostB unreachable - need to frag (mtu 1500) IP hostA > hostB.22:. 2255:3703(1448) ack 2293 win 16304

Host Side Checks n Interrupt Aggregation (Linux NAPI) n Memory to match buffer tuning n More than one CPU n Static ARP entries

Network Device Settings n Static ARP entries or increase timeout n Increase FDB timeouts n Verify jumbo frame configuration

10GbE Quick Notes n Know your PCI hardware (MMRBC, Latency timer, and Splits) n TCP stack is ~0.200ms n Increase netdev_max_backlog to (throughput = backlog * 100MHz * ave_bytes_pkt) n Set *_cong to CERN values n Write buffers in code ~128KB

10G buffer plot

Questions?

Reference URLs n n n n – n n n n n n