The Internet2 HENP Working Group Internet2 Spring Meeting May 8, 2002 Shawn McKee University of Michigan HENP Co-chair

March 19, 2002 April 2002 Slide 2 Overview Mission, Membership and Motivation HENP WG Goals Recent activities Where to from here?

March 19, 2002 April 2002 Slide 3 Internet2 HENP WG* Mission network infrastructuresmonitoring toolsfacilitiescollaborative systemsTo help ensure that the required national and international network infrastructures, monitoring tools and facilities, and support for collaborative systems, are deployed and developed on an ongoing basis, in time to meet the requirements of the major HENP experimental programs, as well as the HENP community at-large. To encourage that the Group's targeted developments are applied broadly, in other fields, within and beyond the bounds of scientific research. The goals of this Working Group are synergistic with the Internet2 End-to-End Initiative, which has HENP as one of its focal disciplines. Formed as an Internet2 WG on Oct * Co-Chairs: S. McKee (Michigan), H. Newman (Caltech); Secretary J. Williams (Indiana); With thanks to R. Gardner (Indiana)

March 19, 2002 April 2002 Slide 4 Current Membership We have 88 members signed up on our mailing list at Our membership is a mix of scientists, engineers, local and backbone network experts and funding agency members We are looking to expand our membership, especially internationally…

March 19, 2002 April 2002 Slide 5 Why HENP Networking? Since the early 1980’s physicists have depended upon leading-edge networks to enable ever larger international collaborations. Major HENP collaborations require rapid access to event samples from massive data stores, not all of which can be locally stored at each computational site. Evolving integrated applications, i.e. Data Grids, rely on seamless, transparent operation of the underlying LANs and WANs. Networks are among the most basic Grid building blocks.

March 19, 2002 April 2002 Slide 6 Tier 1 Tier2 Center Online System Offline Farm, CERN Computer Ctr ~25 TIPS BNL Center France Italy UK Institute Institute ~0.25TIPS Workstations ~100 MBytes/sec Mbits/sec Physicists work on analysis “channels” Each institute has ~10 physicists working on one or more channels Yearly data volume is ~10 Petabytes Physics data cache ~PByte/sec ~2.5 Gbits/sec Tier2 Center ~2.5 Gbps Tier 0 +1 Tier 3 Tier 4 Tier2 Center Tier 2 CERN/Outside Resource Ratio ~1:2 Tier0/(  Tier1)/(  Tier2) ~1:1:1 Hierarchical Computing Model

March 19, 2002 April 2002 Slide 7 HENP WG Goals One of our first task upon forming our working group was to define a set of goals These goals needed to address the range of interests and concerns of our community We arrived at 9 goals, which are now part of our charter. These are are the best definition of what our working group is about…These are are the best definition of what our working group is about…

March 19, 2002 April 2002 Slide 8 HENP WG Goal #1 1.Support the development and deployment of toolkits, documentation and guidelines for best practices so that existing expert knowledge and tools for high throughput data transfers, packet loss and throughput-limit monitoring, and collaborative systems are widely known. Contact: Tom Hacker/Shawn McKee (U Mich)

March 19, 2002 April 2002 Slide 9 HENP WG Goal #2 2.Support the deployment of testing and monitoring tools and applications, link and site instrumentation, and a standard methodology, in association with the Internet2 End-to-End Initiative, so that all of HENP's major network paths can be adequately monitored, and used at full capability. Contact: Iosif Legrand()/Les Cottrell (SLAC)*

March 19, 2002 April 2002 Slide 10 HENP WG Goal #3 3.Share information and provide advice on the configuration of routers, switches, PCs and network interfaces, and network testing and problem resolution, to achieve high performance over local and wide area networks in production. Contact: Sylvain Ravot(CERN)/

March 19, 2002 April 2002 Slide 11 HENP WG Goal #4 4.Work with the staffs of the international (Tier0), national (Tier1), regional (Tier2) and local (Tier3) Grid facilities as needed, to verify high throughput and network responsiveness, in association with the Grid projects, and for new collaborative interactive systems. Contact: Les Cottrell* (SLAC)/

March 19, 2002 April 2002 Slide 12 HENP WG Goal #5 5.Work with the sites, expert networking teams at national labs, universities, and computational science centers, to develop and expand on a knowledge base for high performance networks. Develop a support base of knowledgeable engineers, technicians and scientists to assist with the above goals. Contact: Jim Williams (IndianaU)/

March 19, 2002 April 2002 Slide 13 HENP WG Goal #6 6.Work to identify and track the programmatic network needs of the current and future generation HENP program: quantitatively in terms of bandwidths, throughputs and computer system capabilities required to provide the needed overall performance; and qualitatively in terms of the features and characteristics of existing and new systems needed for efficient distributed data access, processing and analysis as well as remote collaboration. Contact: Harvey Newman(CalTech)/

March 19, 2002 April 2002 Slide 14 HENP WG Goal #7 7.Work with GriPhyN, PPDG, EU DataGrid, iVDGL and other Grid projects to ensure that the US and global network infrastructures are satisfying the programmatic needs of scientists using grids. Contact: Dantong Yu(BNL)/Iosif Legrand()*

March 19, 2002 April 2002 Slide 15 HENP WG Goal #8 8.Work with the network engineering staffs of the US and global research networks to help define the requirements and operational procedures for a Global Grid Operations Center, and mission-specific Grid Operations Centers as needed, so that the ensemble of research networks is able to work efficiently and provide the high level of capability required. Contact: Jim Williams (IndianaU)/

March 19, 2002 April 2002 Slide 16 HENP WG Goal #9 9.Investigate emerging new network technologies, such as optical switching and lambda-based network infrastructure, for applicability and potential use in support of the HENP program. If appropriate, develop a strategic plan for coordinated deployment of these technologies among HENP sites. Contact: Phil Demar(FNAL)/Bill St. Arnaud(CANARIE)

March 19, 2002 April 2002 Slide 17 About These Goals… As you can see from our goals we have many network related issues to pursue. HENPThese goals are generally not specific to HENP. HENP-WGICFA-SCICI2 Engineering GGF Networking WGWe are striving to work with many related groups on these issues: HENP-WG; ICFA-SCIC; I2 Engineering in the US; the GGF Networking WG, etc. NOTE: Grids implicitly assume high, repeatable network performance… => Both grid and networking groups MUST work together to insure an optimal outcome!

March 19, 2002 April 2002 Slide 18 Recent HENP Talks Maximizing End-to-end Network Performance High Performance Throughput Present and Future Networks: an HENP Perspective Performance of Parallel TCP Streams in a Production Network Grid Monitoring QoS Implementation and Testing Report from the PPDG Monitoring Group Network-related Issues in a Grid Environment Remarks on HENP and the I2 End-to-end Performance Initiative Numerous other discussions, perspectives, reports and overviews

March 19, 2002 April 2002 Slide 19 Some HENP Issues… The HENP WG has focused on high performance because of the unprecedented data volume we are anticipating. Some examples of recent interest: Local infrastructure Protocol issues for high performance Issues in achieving high performance networking

March 19, 2002 April 2002 Slide 20 Local Networking Infrastructure LANs used to lead WANs in performance, capabilities and stability, but this is no longer true. WANs are deploying 10 Gigabit technology compared with 1 Gigabit on leading edge LANs. ESNet, I2New protocols and services are appearing on backbones (Diffserv, IPV6, multicast) (ESNet, I2). Insuring our HENP institutions have the required LOCAL level of networking infrastructure to effectively participate in the evolving hierarchical grid computing model is a major challenge.

March 19, 2002 April 2002 Slide 21 TCP WAN Performance Mathis, et. al., Computer Communications Review v27, 3, July 1997, demonstrated the dependence of bandwidth on network parameters: BW - Bandwidth MSS – Max. Segment Size RTT – Round Trip Time PkLoss – Packet loss rate If you want to get 90 Mbps via TCP/IP on a WAN link from LBL to UM you need a packet loss < 1.8e-6 !! (~70 ms RTT). This gets MUCH (~10 3 ) worse if you try to achieve ~1 gigabit across the Atlantic Ocean!

March 19, 2002 April 2002 Slide 22 Achieving High Performance Networking Server and Client CPU, I/O and NIC throughput sufficient Must consider firmware, hard disk interfaces, bus type/capacity Knowledge base of hardware: performance, tuning issues, examples Absolutely RequiredTCP/IP stack configuration and tuning is Absolutely Required Large windows, multiple streams? No Local infrastructure bottlenecks Gigabit Ethernet “clear path” between selected host pairs To 10 Gbps Ethernet by ~2003 Careful Router/Switch configuration and monitoring Enough router “Horsepower” (CPUs, Buffer Size, Backplane BW) Packet Loss must be ~Zero (well below 0.1%) i.e. No “Commodity” networks (need I2, ESNet type networks) End-to-end monitoring and tracking of performance

March 19, 2002 April 2002 Slide 23 Where Are We Headed? We have had two official HENP WG meetings and our third is co-scheduled with the Internet2 Spring Meeting, May 6-8 (1-3 PM Today!) Our goals have been populated with one or two contacts each. During the next few weeks we hope to have compiled goal related information on our WWW page: And longer term…(next slide)

March 19, 2002 April 2002 Slide 24 Some Desired HENP WG Outcomes… localDefine local hardware, software and configuration details necessary to achieve ~gigabit TCP connections to the desktop Enable high-quality, interactive video-conferencing on demand over the WAN multi-gigabitEnable multi-gigabit site-to-site interconnects with related network services to facilitate the tiered grid computing model networksProvide a set of worldwide, transparent, high- performing networks, which allow interaction with users and applications that have special requirements

March 19, 2002 April 2002 Slide 25 Thank-you Thank-you for your attention! Questions? See our WWW page for more information: