A Proposed Architecture for the GENI Backbone Platform Jon Turner jon.turner@wustl.edu http://www.arl.wustl.edu/~jst/

GENI Backbone Platform Flexible infrastructure for experimental networks Implements two primary abstractions metalinks – abstraction of physical links metarouters – abstraction of physical network devices Metalinks point-to-point or multipoint point-to-point links may have provisioned bandwidth built on top of substrate links Metarouters substrate platform provides generic resources variety of resource types with minimal limitations on use functionality defined by researchers may forward packets, switch TDM circuits or implement multimedia processing functions

GENI Backbone Overview substrate link metalink substrate platform metarouter substrate links may run over Ethernet, IP, MPLS, . . . metanet protocol stack

High Level Objectives Enable experimental nets and minimize obstacles focus on providing resources – architectural neutrality enable use by real end users Stability and reliability reliable core platform effective isolation of experimental networks Ease of use enable researchers to be productive without heroic efforts toolkits that facilitate use of high performance elements Scalable performance enable >100K users, wide range of metarouter capacities high ratio of processing to IO Technology diversity and adaptability variety of processing resources – add more types later

Advanced Telecom Computing Architecture New industry standard defines standard packaging enables assembly of multi-supplier systems Standard 14 slot chassis high bandwidth serial links variety of processing blades redundant switch blades integrated management Relevance to GENI flexible, open subsystems compelling research platform faster transition of research ideas into practice carrier card optional mezzanine cards power connector fabric connector optional Rear Transition Module So, let me say a little more about this, because I think it’s a really important development for the networking research community, and it’s one that many of us have not been aware of. So what is ATCA? Well, it’s a packaging standard that defines standard printed circuit board formats, common connector definitions and standard backplanes and chasses. So why, you ask, should networking researchers care about a packaging standard? Well, we should care, because it has led to the creation of a new market for intermediate board-level subsystems that can be purchased from different suppliers and assembled into systems with tremendous flexibility. The key thing to understand here is that because these subsystems are produced and sold by subsystem vendors to multiple systems companies, they must be flexible and open. This means that networking researchers can buy these components and assemble them into novel systems that they can configure and program. That is, for the first time, we have access to experimental platforms that are no less powerful than the hardware platforms produced by major systems companies, but these are under our control.

Virtualized Line Card Architecture Similar to conventional router architecture line cards connected by a switch fabric traffic makes a single pass through the switch fabric Requires fine-grained virtualization line cards must support multiple meta line cards requires intra-component resource sharing and traffic isolation Mismatch for current device technologies multi-core NPs lack memory protection mechanisms lack of tools and protection mechanisms for independent, partial FPGA designs Hard to vary ratio of processing to IO ILC2 Switch Fabric ILC1 . . . ILCn OLC1 OLC2 OLCn Input Line Cards Output Line Cards substrate . . . Processing Resources So, how can we best use technology components like these to construct a diversified router. There are several approaches one can take. The first one shown here is similar to a conventional router architecture, in the sense that it consists of line cards connected by a switch fabric, with packets passing from input line cards through the switch fabric to output line cards. To enable multiple metarouters to co-exist on such a system, we need to diversify the line cards, by equipping them with generic processing resources that can be divided up among the different meta line cards. One way to do this is to allocate the different processor cores of a network processor to different meta line cards. Unfortunately, this kind of fine-grained diversification is difficult to achieve with current network processors, which lack the protection mechanisms needed to isolate different meta line cards from one another.

Processing Pool Architecture Processing Engines (PEs) implement metarouters variety of types Line Cards terminate ext. links, mux/dmx metalinks Shared PEs include substrate component Dedicated PEs need not include substrate use switch and Line Cards for protection and isolation PEs in larger metarouters linked by metaswitch Larger metarouters may own Line Cards allows metanet to define transmission format/framing configured by lower-level transport network Line Cards PEs Switch

Ensuring Metarouter Isolation PE . . . metarouter 1 metarouter 2 LC constrained routing nonblocking switch fabric no interference Constrain routing on switch port basis. use switch with VLAN support for constrained routing substrate controls VLAN configuration Nonblocking switch fabric ensures traffic isolation. congestion at one port does not affect traffic to another traffic within clusters cannot interfere Here’s an example that illustrates the issue. Here, we have two metarouters, each of which involves several processing engines. The PEs within each metarouter communicate through the switch fabric. To isolate these PE-to-PE traffic flows from one another, we need to do two things. First, we need to constrain the routing. The 10 GE switch blades that are becoming available over the next year can route packets based on VLAN tags and these VLAN tags can be configured through an administrative interface available only to the substrates, not the metarouters. This makes it possible to ensure that a PE in one router cannot send traffic to the PE in another. The second thing we need to do is to isolate the traffic flows, but because the different metarouters occupy distinct physical ports on the router, we get this property for free, so long as the switch fabric is nonblocking. There is one last thing I’ve glossed over until now, and that has to do with the outgoing traffic streams sent by different metarouters to shared line cards. With current technology components, the router substrate cannot rate limit the PEs at the switch fabric inputs. However, we can require, as a matter of policy that metarouters rate limit these flows, and the outgoing line cards can monitor the metarouters’ traffic flows in order to ensure that the rate limits are being observed. Metarouters that violate the limits can then be disabled by the substrate, in order to protect other metarouters.

Current Development System Network Processor blades dual IXP 2850 NPs 3xRDRAM, 3xSRAM, TCAM dual 10GE interfaces 10x1GE IO interfaces General purpose blades dual Xeons, 4xGigE, disk 10 Gb/s Ethernet switch VLANs for traffic isolation

Prototype Operation One NP blade (with RTM) implements Line Card GPE NPE LC RTM Switch Lookup (2 ME) ExtRx IntTx Queue Manager Key Extract Hdr Format (1 ME) IntRx ExtTx Rate Monitor TCAM DRAM SRAM external interface switch interface ingress side egress side Lookup (1 ME) Rx Tx Queue Manager (2 ME) Key Extract Hdr Format TCAM DRAM SRAM One NP blade (with RTM) implements Line Card separate ingress/egress pipelines Second NP hosts multiple metarouter fast-paths multiple static code options for diverse metarouters configurable filters and queues GPEs host conventional OS with virtual machines

Line Card Ingress side demuxes with TCAM filters (port #s) Lookup (2 ME) ExtRx IntTx Queue Manager Key Extract Hdr Format (1 ME) IntRx ExtTx Rate Monitor TCAM DRAM SRAM external interface switch interface ingress side egress side Ingress side demuxes with TCAM filters (port #s) Egress side provides traffic isolation per interface Target 10 Gb/s line rate for 80 byte packets

NPE Hosting Multiple Metarouters Lookup (1 ME) Rx (2 ME) Tx Queue Manager Parse Hdr Format TCAM DRAM SRAM Substr. Decap Parse and Header Format include MR-specific code parse extracts header fields to form lookup key Hdr Format makes required changes to header fields Lookup block uses opaque key for TCAM lookup and returns opaque result for use by Hdr Format Multiple static code options can be supported multiple metarouters per code option each has own filters, queues and block of private memory

Possible Additional PE Types ATCA carrier card 10 GE switch with connections to each switch blade 4 mezzanine card slots with 10 GE ext. IO interface to RTM connector FPGA mezzanine card Xilinx Virtex-5 LX330 over 200K flip flops and LUT6s over 1 MB of on-chip SRAM on-board SDRAM and SRAM chips Cavium NP card up to 16 MIPs processor cores 600 MHz, dual issue per core L1 cache, shared L2 (2 MB) more conventional SMP prog. style SDRAM FPE SRAM 10 GE Switch Power Flash SDRAM FPE SRAM GLU SPI-4 Cavium NP DRAM Cavium NP

Scaling Up Baseline config (1+1) Multi-chassis direct 14 slot ATCA chassis separate blade server for GPEs 14 GPEs + 9 NPEs+ 3 LCs 10 GE inter-chassis connection Multi-chassis direct up to seven chassis pairs 98 GPEs + 63 NPEs+ 21 LCs 2-hop forwarding as needed Multi-chassis indirect up to 24 chassis pairs 336 GPEs + 216 NPEs + 72 LCs ATCA 2 Blade Server ATCA 2 Blade Server 6 24 port 10GE Switches ATCA 2 24 12 Blade Server

Summary GENI requires capable backbone platform to enable wide range of experimental research to support production use of experimental networks by large numbers of non-research users Required hardware building blocks are at hand ATCA provides useful framework powerful server blades and NP blades high performance switching components What’s still to do? software to manage/configure resources for users sample metarouter code for NPs FPGA-based processing engines tools to speed-up metarouter development demonstration of multi-PE metarouters