INDIANAUNIVERSITYINDIANAUNIVERSITY High Performance Networking for Colleges and Universities: From the Last Kilometer to a Global Terabit Research Network TERENA Networking Conference 2001 Antalya, Turkey Michael A. McRobbie PhD Vice President for Information Technology and Chief Information Officer Steven Wallace Chief Technologist and Director Advanced Network Management Laboratory Indiana Pervasive Computing Research Initiative

INDIANAUNIVERSITYINDIANAUNIVERSITY Network-Enabled Science and Research in the 21 st Century Science and research is becoming progressively more global with network-enabled world wide collaborative communities rapidly forming in a broad range of areas Many are based around a few expensive – sometimes unique – instruments or distributed complexes of sensors that produce vast amounts of data These global communities will carry out research based on this data

INDIANAUNIVERSITYINDIANAUNIVERSITY Network-Enabled Science and Research in the 21 st Century This data will be: –collected via geographically distributed instruments –analyzed by supercomputers and large computer clusters –visualized with advanced 3-D display technology and –stored in massive or large data storage systems All of this will be distributed globally

INDIANAUNIVERSITYINDIANAUNIVERSITY Examples of Network-Enabled Science NSF funded Grid Physics Network’s (GriPhyN) need for petascale virtual data grids (i.e. capable of analyzing petabyte datasets) –Compact Muon Selenoid (CMS) and A Toroidal LHC Apparatus (ATLAS) experiments using the Large Hadron Collider (LHC) located at (CERN) [> 2.5 Gb/s] –Laser Interferometer Gravitational Wave Observatory (LIGO) [200GB-5TB data sets needing 2.5 Gb/s or greater for reasonable transfer times] –Atacama Large Millimeter Array (ALMA) Collaborative video (e.g. HDTV) [20Mb/s] Sloan Digital Sky Survey (SDSS) [> 1Gb/s]

INDIANAUNIVERSITYINDIANAUNIVERSITY A Vision for 21 st Century Network- Enabled Science and Research The vision is for this global infrastructure and data to be integrated into “Grids” – seamless global collaborative environments tailored to the specific needs of individual scientific communities

INDIANAUNIVERSITYINDIANAUNIVERSITY Components of Global Grids High performance networks are fundamental to integrating Global Grids together There are, very broadly speaking, three components to Global Grids: –Campus Networks (the last kilometer) –National and Regional Research and Education Networks (NRRENs) –Global connections between NRRENs

INDIANAUNIVERSITYINDIANAUNIVERSITY Impediments to Global Grids Of these three components, on a world-wide scale, investment and engineering is only adequate for NRRENs –Campus networks rarely provide scalable bandwidth to the desktop commensurate with speeds of campus connections to NRRENs –Global connections between NRRENs are major bottlenecks – they are very slow compared to NRREN backbone speeds

INDIANAUNIVERSITYINDIANAUNIVERSITY Building Global Grids To build a true Global Grid requires: Scalable campus networks providing ubiquitous high bandwidth connections to every desktop commensurate with campus connections to NRRENs Global connectivity between NRRENs of comparable speeds to the NRREN backbones, which is also stable, persistent and of production quality like the NRRENs themselves

INDIANAUNIVERSITYINDIANAUNIVERSITY Presentation Overview This talk describes: 1.Some NRRENs and their common characteristics 2.“Grid Ready” Campus Networks with Indiana University’s network architecture and management of IT as an example 3.A solution to the global connectivity problem that scales to a terabit global research network

INDIANAUNIVERSITYINDIANAUNIVERSITY 1. NRRENs Abilene –OC48 ->OC192 –OC48 connected GigaPoPs (moving to min. OC12) –ITN provider –1 Gb/s sustained data rates seen CAnet3 US Fed nets (e.g. ESnet) DANTE -> GEANT APAN CERNET

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY NRRENs OC48 (2.4 Gb/s) backbone implemented today –Moving to OC192 (9.6 Gb/s) as next evolution Institutions access backbone at OC12 or greater (a few connections at OC48) Native high-speed IPv4 Support for IPv6 (but at much lower performance due to router constraints)

INDIANAUNIVERSITYINDIANAUNIVERSITY NRRENs Advanced Services –Typically run as open (visible) networks, not a commercial service –IP multicast deployed in most backbones, but still not as production as unicast; but not reliable internationally –QoS – mixed results, still in it’s infancy. Very little going across more than one network Intra-regional interconnect speeds range from OC3 to OC12. Soon to be OC48 in some cases

INDIANAUNIVERSITYINDIANAUNIVERSITY 2.1 Campus Networks: the Critical Last Kilometer Grid applications will require “guaranteed” multi-Mbps bandwidth “per” application – “end to end" That is, these speeds must be sustained from the desktop, through the various levels of the campus network to the NRREN and then to the application target Thus campus networks must be architected to provide scalable levels of connectivity to NRRENs as their speeds increase It makes no sense to have a shared 10Mbps desktop connection (delivering at most about 1 Mbps) into a 10,000 Mbps (10Gbps) NRREN!! Providing appropriate levels of connectivity to the desktop and scalable campus network architectures to support them is a top priority in IT strategic planning at US Universities A sizable portion of telecommunications budgets can go to this (e.g. $25M at Indiana University in 00/01)

INDIANAUNIVERSITYINDIANAUNIVERSITY Grid-Ready Campus Network Architectures The three main levels of campus network architectures are: –Desktop/intrabuilding (the last 100 meters) –Interbuilding (connecting groups of buildings) –Backbone (connecting those groups) Each level must provide progressively more capacity and the whole architecture must be scalable Campus networks commonly have two external connections –Commercial Internet –Regional gigaPoPs (to NRRENs)

INDIANAUNIVERSITYINDIANAUNIVERSITY Campus Networking (last 100 meters) 10Mb/s switched to the desktop –Adequate for VHS-quality video distribution Video conferencing Digital library such as CD quality music –Gigabyte datasets transferred in 15 minutes 100Mb/s switched to the desktop –Cost of 10/100Mb switch ports less than $50USD –Gigabyte datasets transferred in 2 minutes –Suitable for HDTV distribution 1000Mb/s switched to the desktop now practical over Cat5 copper –Cost of 100/1000Mb switch ports less than $500USD –Terabyte datasets transferred in 2.5 hours Fiber to the desktop probably reserved for 10Gb/s and beyond but necessary between buildings

INDIANAUNIVERSITYINDIANAUNIVERSITY Grid Ready Campus Network Enablers Gigabit wire-speed ASIC-based routers –currently providing 1 Gb/s uplinks, next generation will support 10 Gb/s uplinks –QoS support in hardware Commodity priced Ethernet switches that support 10/100 Mb/s and 1 Gb/s connections

INDIANAUNIVERSITYINDIANAUNIVERSITY 2.2 Indiana University’s Grid Ready Campus Network Switched 10Mbps standard for all 55,000+ desktops Switched 100Mbps available on request 1Gbps available in selected cases OC12 (650 Mb/s) Internet2 connectivity Native support for IP multicast in both the layer 2 Ethernet switches and the layer 3 routers Support for DiffServ based quality of service

INDIANAUNIVERSITYINDIANAUNIVERSITY 2.3 Managing IT in US Higher Education In the US IT is recognized as being: –of central importance in higher education –fundamental to teaching, learning & research –essential to responsible and accountable institutional management –a source of institutional competitive advantage It is also a major source of expenditure in US universities: (fully costed) between 5 & 10% of an institution’s total budget

INDIANAUNIVERSITYINDIANAUNIVERSITY Responsibility for Managing IT US Universities have elevated IT to a portfolio of central importance reporting directly to the president or chief academic officer This portfolio tends to be the responsibility of the chief information officer (CIO) University CIOs are typically responsible for central IT and support of distributed IT (e.g. in departments, schools, faculties) This parallels earlier developments in US business

INDIANAUNIVERSITYINDIANAUNIVERSITY Strategic Planning for IT Given the vital importance of IT, US universities have developed IT strategic plans These plans guide the institution’s future development and investment in IT They are also used to leverage considerable additional public and private funding for university IT infrastructure

INDIANAUNIVERSITYINDIANAUNIVERSITY 2.4 An Example: Indiana University Founded in 1820 State public university with: -$1.9B budget (99-00) with 27% from the State of Indiana -7 campuses State-wide (two largest and research intensive campuses in Bloomington and Indianapolis) -97,150 students -4,276 faculty -9,844 appointed staff -42,000+ course sections -$1B endowment

INDIANAUNIVERSITYINDIANAUNIVERSITY IT at Indiana University CIO position created in 96; reports directly to IU President Responsible for central IT on all campuses -Central IT budget from all sources $100M -1,200 staff -Departments, schools & faculties expend about a further $50M Central IT comprises -telecommunications (e.g. data, voice, video on campus, intra & interstate, & internationally) represents 35% of the budget -research & academic computing (e.g. supercomputing, massive data storage, large-scale VR) represents 13% of the budget -teaching & learning technologies (e.g. user support/education, desktop life-cycle funding, classroom IT, enterprise software licensing, student labs, Web support) represents 28% of the budget -administrative computing (enterprise information systems – e.g. student, financial, HR & library systems, enterprise databases & storage) represents 24% of the budget

INDIANAUNIVERSITYINDIANAUNIVERSITY Strategic Planning for IT at Indiana University Goal of Indiana University to be a leader in the “…use and application of information technology”. CIO responsible for developing IT Strategic Plan to achieve this goal: -first University-wide IT Strategic Plan -used IT Committee system; 200 people involved in preparation -prepared December 97 to May 98, then discussed University-wide & approved by President and Board of Trustees, December 98 -CIO responsible for implementation -5 year plan consisting of 10 major recommendations; 68 actions ( -Implementation Plan and full costings developed in parallel -Full cost $210M over 5 years; $120M in new funding from the State, $90M in re-programmed University funding

INDIANAUNIVERSITYINDIANAUNIVERSITY 3.1 Towards a Global Terabit Research Network (GTRN) Global network-enabled collaborative Grids require true high-speed global research and education network that: –is of production quality (managed as production service, redundant, stable, secure) –is persistent (is based on a long-term agreement with carrier(s) and others) –provides a uniform form of connection globally through global network access points (GNAPs) –provides interconnect speeds comparable to NRRENS backbone speeds (presently OC48 going to OC192) –scales to a terabit per second data rate during this decade

INDIANAUNIVERSITYINDIANAUNIVERSITY Impediments to Building Global Grids International connections very slow compared with NRREN backbone speeds Long term funding uncertain (e.g. NSF HPIIS program) Global connection effort not well-coordinated Overly reliant on transit through US infrastructure Frequently connections are via ATM or IP clouds, making management of advanced services difficult Poor coordination of advanced service deployment Extreme difficulty ensuring reasonable end-to-end performance

INDIANAUNIVERSITYINDIANAUNIVERSITY Connectivity to US Transit Infrastructure Asia Pacific Americas Europe

INDIANAUNIVERSITYINDIANAUNIVERSITY STAR TAP and International Transit Service (ITN) STAR TAP, CAnet3 and Abilene provide some level of International transit across North America –Abilene offers convenient international transit at multiple landing sites, however transit not offered to other NRRENs (e.g. ESnet) STAR TAP requires a connection to AADS best effort ATM service (reducing the ability to deploy QoS)

INDIANAUNIVERSITYINDIANAUNIVERSITY Indiana University Firsthand experience with these difficulties through its Global NOC

INDIANAUNIVERSITYINDIANAUNIVERSITY

INDIANAUNIVERSITYINDIANAUNIVERSITY 3.2 Towards a GTRN A single global backbone interconnecting global network access points (GNAPs) that provide peering within a country or region Global backbone speeds comparable to those at NRRENS, i.e. OC192 in 2002 Based on stable carrier infrastructure Persistent based on long-term (5-10 year) agreements with carriers, router vendors and optical transmission equipment vendors

INDIANAUNIVERSITYINDIANAUNIVERSITY Towards a GTRN Scalable – e.g. OC768 by 2004, multiple wavelengths running striped OC768s by 2005, terabit/sec transmission by 2006 GNAPs connect at OC48 and above. To scale up as backbone speeds scale up Production service with 24x7x365 management through a global NOC Coordinated global advanced service deployment (e.g. QoS, IPv6, multicast)