OmniPoP: GigaPoP-of-GigaPoPs Design and Future Patrick Christian, OmniPoP TAC (chair) University of Wisconsin - Madison April 25, 2007

Outline Formation drivers Administrative organization Design goals ArchitectureQuestions

Formation Drivers OmniPoP: GigaPoP-of-GigaPoPs Design and Future

Formation Drivers Omni- –A prefix meaning “all” PoP –Three letter acronym popular in network engineering meaning “Point of Presence” –A place where an organization’s (or several) networking equipment is present OmniPoP –A place where all types of networking equipment is present –A place where all members are welcome to place equipment –A place to develop a rich mesh of all types of services between members –A facility to provide common and small group services

Formation Drivers Formation Drivers CIC Fiber In Chicago Internet2 Connection For CIC Members National Lambda Rail Colocation Space For Network Equipment Direct Connection Between CIC Members Commodity Internet Service 24/7/365 Production Support Network Operations Center Regional Optical Networks

Formation Drivers OmniPoP Colocation Space Multiple fiber-rich, friendly locations Network Hardware Optical switches Ethernet switches Routers, and more NOC 7/24/365 Hands and Eyes Production operations Technical Support and Engineering Services Governance

Administrative Organization OmniPoP: GigaPoP-of-GigaPoPs Design and Future

Administrative Organization CIC OmniPoP participants (4/07 connection status) –University of Chicago (soon) –University of Illinois (UIUC & UIC) (soon) –Indiana University (active) –University of Iowa (active) –University of Michigan (active) –Michigan State University (active) –University of Minnesota (active) –Northwestern University (soon) –Ohio State University (soon) –Purdue University (active) –University of Wisconsin – Madison (active)

Administrative Organization CIC CIOs (MSU CIO Dave Gift TAC liaison) Technical Advisory Council (2 reps per member) U-WI: Patrick Christian (chair) All Techs (unlimited, interested parties) Subcommittees Technical Tactical Team (5 reps, subset of TAC) Karen Partlow CIC Assistant Director Tech Collaborations

Administrative Governance Governing Board –One representative from each Partner (initially the participating CIC schools) –Responsible for Policies Finances, including major expenditures Approving new Partners (not limited to CIC schools) Selecting the OmniPoP operator Requires two-thirds vote to approve all actions –Meets at least annually

Administrative Governance Technical Advisory Council (TAC) –Up to two representatives from each Partner plus representation from the OmniPoP operator & CIC –Responsible for Preparing an annual strategic technical plan Recommending an annual budget Recommending changes to services offered Approving routine purchases and service contracts Tactical Technical Team Not more than six members To work with the OmniPoP operator and CIC on day- to-day technical coordination of operational issues

Design Goals OmniPoP: GigaPoP-of-GigaPoPs Design and Future

Design Goals Platform capabilities –Robust, flexible and highly available infrastructure –OmniPoP facility capable/supports leading edge interfaces –High-speed interface density (needed 20x10GEs day 1) –Line rate AND/OR oversubscription architecture capabilities –Provision high-performance, full-featured switching & routing capability to reduce CIC duplication of infrastructure in Chicago –Transmit data at low latency and low jitter rates as well as minimize bottlenecks between devices –Separation of control/data plane

Design Goals (select) Technical features Nx10GE (802.3ad) link aggregation support (e.g. 2x10GE to I2) Significant pluggable optics support VLAN tagging (802.1q) & tag swapping Queue in Queue capabilities RADIUS support Strong QoS capabilities Multicast (PIM, MLD, and IGMP snooping) Broadcast/multicast suppression (rapid) Per VLAN spanning tree Spanning tree controls (root guard, etc.) BFD (bidirectional forwarding detection) L2 filtering Broadcast/multicast suppression

Design Goals Misc. non-technical –Give up a little control to enable member- institutions to save $$$ & time –# & location(s) of facilities –How to buy issues (RFP v. existing contract) –Reduced operating system software complexity (single train to reduce “feature fights” within CIC) –Operator has experience managing platform (familiar CLI & tool extensions) – reduces OpEx

Architecture OmniPoP: GigaPoP-of-GigaPoPs Design and Future

Architecture Selected Force10 E1200 switch Equipment collocated at “StarLight” (710 NLSD) Each institution receives a 1 & 10GE “line rate port” –Required 20x10GE ports –Purchased 3 line rate & 2 (4:1) 16 port cards Shared links use line rate ports Best effort across shared links (for now) with no shaping/policing Aggregation of CIC NLR access Aggregation of connectivity to national/international R&E networks Interconnection of CIC institutions to each other (pending) Aggregation of Internet2 access

Architecture (future) Commercial peering (non-transit/transit) Collocation space in other Chicago locations Layer 3 infrastructure Layer 1 infrastructure??? DRAGON/OSCARs RD&D???

Architecture – Phase I

OmniPoP: Phase II

Architecture - Current

Questions

OmniPoP Collaboration Lessons learned Need collaboration idea & drivers to maintain group cohesion Recognize and balance constant tension between collaboration and self-control exists & requires management CIC an important group to facilitate “people networking” to identify other collaborators and common problems Good ideas grow – Chicago fiber and OmniPoP collaboration groups started small (2 members) and grew quickly Needs assessment to determine sharing opportunities difficult and time consuming but vital to success Successful collaborations yield substantial rewards after accounting for additional time, communication and effort required to sustain collaboration