The ShadowNet ProtoGENI Measurement Infrastructure Jim Griffioen Lab for Advanced Networking University of Kentucky Lexington, KY Kobus Van der Merwe AT&T Labs - Research Florham Park, NJ Other Project Members Zongming Fei (Kentucky) Eric Boyd (Internet 2)

Outline ProtoGENI ShadowNet Leveraging AT&T ShadowNet March 17, 2010GEC7

ProtoGENI ShadowNet March 17, 2010GEC7

Project Overview Problem: ProtoGENI backbone router resources are limited and can be challenging to use. Idea: Leverage the logical router features of Juniper routers to dynamically create virtual routers (slivers) in the backbone that provide carrier-grade performance and services. Challenge 1: Creating the control software needed to virtualize the Juniper M7i and integrate with the ProtoGENI network Challenge 2: Make it easy for users to see what is happening on their backbone router slivers. March 17, 2010GEC7

Project Goals 1. Deploy virtualizable commercial routers (Juniper m7i) in the ProtoGENI backbone that support commercial OS/software. 2. Add software support to these virtual routers that will enable per-slice monitoring and measurement. 3. Develop tools and interfaces that will allow slice users to use the measurement infrastructure in simple and easy ways. March 17, 2010GEC7

March 17, 2010GEC7 Source: ProtoGENI Network

March 17, 2010GEC7 Source: ProtoGENI Shadownet Sites Year 1 Year 2

ProtoGENI Backbone Node Architecture March 17, 2010GEC7 Sliver 1 NetFPGA Sliver n General Purpose Slivers Gigabit Ethernet Switch Internet 2 Non-sliced PC Sliced PC

ProtoGENI Backbone Node Architecture March 17, 2010GEC7 Logical Router 1 Logical Router 2 Logical Router n Juniper M7i Router Sliver 1 NetFPGA Sliver n ShadowBox Controller Juniper Component Manager Virtual Server General Purpose Slivers Gigabit Ethernet Switch Internet 2 Non-sliced PC Sliced PC ShadowBox Router

ProtoGENI Backbone Node Architecture March 17, 2010GEC7 Logical Router 1 Logical Router 2 Logical Router n Juniper M7i Router Sliver 1 NetFPGA Sliver n ShadowBox Controller Juniper Component Manager Virtual Server perfSONAR 1perfSONAR n General Purpose Slivers Measurement Slivers Gigabit Ethernet Switch Internet 2 Non-sliced PC Sliced PC ShadowBox Router

Leveraging AT&T ShadowNet March 17, 2010GEC7

Why ShadowNet? ShadowNet is roughly addressing same problem as GENI, however Less clean slate… Focus on services and network management… Need the ability to more rapidly evolve the way we run our network and the services we offer in our network (pull): Inherently difficult: – Potential impact to existing services Networks are shared, new service/feature might negatively interact with existing services Gets worse with time: networks are cumulative (hardly ever gets switched off) Very long test cycles – Need for support systems Configuration management, network management, service monitoring, provisioning, customer interfaces, billing, fault management Legacy lock in: Existing (complicated) systems need to be modified to support new services Extremely long development time New vendor technologies (push): Programmability and virtualization available from major vendors – Allow non-vendor code to execute on routers – Loosen the tight coupling between physical boxes and logical functions Rethink the way we deploy services and operate our network

ShadowNet as (part of) a solution National footprint network/platform/testbed for research and service trials – Connected to, but separate from production network Limit impact on operational network Look like a customer to AT&T network – In between lab and production Stable enough for service trials Open/flexible enough for research experiments – General purpose, shareable testbed facility Would like to make this a widely available/useful facility, akin to general purpose computing facilities The role of ShadowNet: Operational (but non-production) environment to enable: – Evaluation of new technologies/vendor capabilities No impact on existing network/services – Service testing/trials in a realistic environment (including customer trials) Utilize virtualization and partitioning capabilities to limit interaction and reduce risk – Evolution of network support systems Free from legacy lock – Research in operational setting Both networking and Internet services Safe playground for network evolution – This model might become the way we want to build our network

ShadowNet node architecture Operational nodes: Richardson, TX Pleasanton, CA Chicago, IL Waiting for network connectivity: Middletown, NJ Page 14 Juniper M7i Router Juniper M7i Juniper M7i Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Cisco Catalyst 3560G-48TS GigE ShadowNet rack Each node: – Gateway router, Juniper M7i – 2 X GigE connectivity to AT&T network – 7 X SunFire x4150 servers – 2 X multiservice routers, Juniper M7i – Cisco GigE switch (Catalyst 3560) – OOB access AS 5105: – Full BGP table – 4 /24 prefixes – Advertise up to /32 Set of building blocks that can be flexibly combined into an operational network (or networks)

Page 15 ShadowNet Sharable and composable infrastructure Strong separation between physical and logical devices: Physical machines -> virtual machines Physical routers -> logical routers Physical links -> logical gigE links: pseudowires, tunnels, VLANs etc ShadowNet slices consist of logical devices that have been plumbed together However, allow allocation of physical devices to a slice

March 17, 2010GEC7 Life cycle of ShadowNet devices

March 17, 2010GEC7 Using ShadowNet CloudNet experimentation Combining cloud computing with VPN Fairly elaborate setup involving many components Create VPLS VPN between three sites Prototype dynamic VPN connectivity Experiment with (live) virtual machine and storage migration Mechanisms for optimizing WAN migration In the works: Cloud control architecture Slice with bunch of VMs for architectural support for network debugging Declarative approach to network management Extend to provide mobility functionality "The interesting thing about cloud computing is that we've redefined cloud computing to include everything that we already do. I can't think of anything that isn't cloud computing with all of these announcements. Larry Ellison, CEO Oracle Wall Street Journal, September 26, 2008

Enterprise Cloud Challenges Existing cloud platforms do not meet the needs of enterprise customers Insufficient security controls Need isolation at server and network level Deployment is difficult - transparency Cloud resources are completely separate from local ones Cant make VMs look like part of existing enterprise network Limited control over network resources Cannot specify network topology or IP addresses Cannot reserve bandwidth or request QoS guarantees for network links Page 18

Page 19 CloudNet Enterprise-Ready Virtual Private Clouds Use VPNs to separate customer resources Customers cloud resources are only reachable from other VPN end points More flexible control of how IP addresses are assigned Physical network is transparent to customer Assume a virtual machine abstraction CloudNet writeup (intranet): VPNs provide both network resource isolation and strong security

Page 20 Virtual Private Clouds Virtual Private Cloud: Collection of cloud resources presented to customer as a private set of cloud resources, transparently and securely connected to customer VPN Manage network resources in the same dynamic manner as cloud resources Server PE VPN A VPN B Server Cloud Site X AT&T Backbone PE Server Cloud Site Y Server VPC A VPC B

Page 21 System/Architecture Components Server PE CE Router PE VPN A VPN B Network Manager High level abstraction: Create compute resources Map into VPN Cross domain interaction CloudNet Portal Cloud Manager Cloud Platform AT&T Backbone PE Network DomainCloud Domain Cloud Manager: Create compute resources Map into VPN (cloud side) Network Manager (IRSCP): VPN management (network side)

Page 22 Cloudnet in ShadowNet: Physical nodes involved CloudNet slice AT&T backbone (7132) AT&T backbone (7132) Juniper M7i Juniper M7i Juniper M7i Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Cisco Catalyst 3560G-48TS ShadowNet rack Juniper M7i Juniper M7i Juniper M7i Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Cisco Catalyst 3560G-48TS ShadowNet rack GRE tunnels Juniper M7i Juniper M7i Juniper M7i Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Sun Fire X4150 Server Cisco Catalyst 3560G-48TS ShadowNet rack PLTN CHCG RCSN

Page 23 Cloudnet in ShadowNet: VPLS MPLS VPN in a slice P1 P3 Cisco Switch P1 Juniper Router Cisco Switch P3 Juniper Router VLAN-CCC VLAN Logical link: VLAN cross connect example PLTN5 RCSN6 CHCG6 PE1 P1 RR/IRSCP P3PE3 P2PE2 Logical tunnel VLAN circuit cross connect Physical ethernet PLTN RCSN CHCG

Page 24 VM migration across WAN Ipsec client on laptop provides remote access to VPN Run game server on VM Run game client on laptop Game server move with VM Application very sensitive to network impairments Client monitor typically shows game detects minor changes VM migration across WAN just works using VPLS VPNs Optimize for WAN conditions: Storage: moving between asynchronous and synchronous replication VM: optimizing migration logic + redundancy elimination PLTN5 RCSN6 CHCG6 PE1 P1 RR/IRSCP P3PE3 P2PE2 VpnRemap PLTN CHCG RCSN r0 drbd VM0 Game Server VM0 Game Client Laptop ipsec

Thank You! Questions? This material is based upon work supported in part by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of GPO Technologies, Corp, the GENI Project Office, or the National Science Foundation.