OpenFlow for Campuses A Tutorial at GEC10 March 15, 2011 Srini Seetharaman, DT R&D Lab, USA Masa Kobayashi, NEC Labs/Stanford

Introductions

Agenda Time Description 13:00-14:00 Introduction: How OpenFlow works, Potential, Limitations, Current vendors, SDN/OPEN 14:00-14:30 Show tools, demo mininet and NOX intro for researcher 14:30-15:00 Configuring NEC, HP and Pronto switches Break 15:30-16:15 Controller demonstrations (SNAC, BigSwitch’s controller and NEC’s PFC) 16:15-16:45 Slice using FlowVisor, Expedient, and opt-in manager 16:45-17:00 Q&A, Discussions, Community building, Conclusions

Goals of this Tutorial By the end, everyone should know: what OpenFlow is how it’s used and how you can use it where it’s going how OpenFlow fits in the Software-Defined Networking (SDN) spectrum how to slice your network Present a useful mix of lecture-based content and actual operational info Have fun What: protocol,

Why OpenFlow?

Specialized Packet Forwarding Hardware The Ossified Network Routing, management, mobility management, access control, VPNs, … Feature Feature Million of lines of source code 5400 RFCs Barrier to entry Operating System Specialized Packet Forwarding Hardware Billions of gates Bloated Power Hungry Many complex functions baked into the infrastructure OSPF, BGP, multicast, differentiated services, Traffic Engineering, NAT, firewalls, MPLS, redundant layers, … An industry with a “mainframe-mentality”, reluctant to change 6 6

Research Stagnation Lots of deployed innovation in other areas OS: filesystems, schedulers, virtualization DS: DHTs, CDNs, MapReduce Compilers: JITs, vectorization Networks are largely the same as years ago Ethernet, IP, WiFi Rate of change of the network seems slower in comparison Need better tools and abstractions to demonstrate and deploy In these other areas, you can real systems on top of high-quality, open platforms, and you don’t need to reinvent the wheel

Closed Systems (Vendor Hardware) Stuck with interfaces (CLI, SNMP, etc) Hard to meaningfully collaborate Vendors starting to open up, but not usefully Need a fully open system – a Linux equivalent

none have all the desired attributes! Open Systems Performance Fidelity Scale Real User Traffic? Complexity Open Simulation medium no yes Emulation low Software Switches poor NetFPGA high Network Processors Vendor Switches gap in the tool space none have all the desired attributes!

See Ethane SIGCOMM 2007 paper for details Ethane, a precursor to OpenFlow Centralized, reactive, per-flow control Controller Flow Switch Flow Switch Flow Switch Host B Host A Flow Switch See Ethane SIGCOMM 2007 paper for details

OpenFlow: a pragmatic compromise + Speed, scale, fidelity of vendor hardware + Flexibility and control of software and simulation Vendors don’t need to expose implementation Leverages hardware inside most switches today (ACL tables)

How does OpenFlow work? 12

Ethernet Switch

Control Path (Software) Data Path (Hardware)

OpenFlow Controller Control Path OpenFlow Data Path (Hardware) OpenFlow Protocol (SSL/TCP) Control Path OpenFlow Data Path (Hardware)

OpenFlow Client Controller PC OpenFlow Example Software Layer MAC src Flow Table MAC src dst IP Src Dst TCP sport dport Action Hardware Layer * 5.6.7.8 port 1 port 1 port 2 port 3 port 4 5.6.7.8 1.2.3.4

OpenFlow Basics Flow Table Entries Rule Action Stats Packet + byte counters Forward packet to zero or more ports Encapsulate and forward to controller Send to normal processing pipeline Modify Fields Any extensions you add! Now I’ll describe the API that tries to meet these goals. Switch Port VLAN ID VLAN pcp MAC src MAC dst Eth type IP Src IP Dst IP ToS IP Prot L4 sport L4 dport + mask what fields to match

Examples Switching Flow Switching Firewall Switch Port MAC src dst Eth type VLAN ID IP Src Dst Prot TCP sport dport Action * * 00:1f:.. * * * * * * * port6 Flow Switching Switch Port MAC src dst Eth type VLAN ID IP Src Dst Prot TCP sport dport Action port3 00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6 Firewall Switch Port MAC src dst Eth type VLAN ID IP Src Dst Prot TCP sport dport Action * * * * * * * * * 22 drop

Examples Routing VLAN Switching Switch Port MAC src dst Eth type VLAN ID IP Src Dst Prot TCP sport dport Action * * * * * * 5.6.7.8 * * * port6 VLAN Switching Switch Port MAC src dst Eth type VLAN ID IP Src Dst Prot TCP sport dport Action port6, port7, port9 * * 00:1f.. * vlan1 * * * * *

Centralized vs Distributed Control Both models are possible with OpenFlow Centralized Control Distributed Control Controller Controller OpenFlow Switch OpenFlow Switch Controller OpenFlow Switch OpenFlow Switch Controller OpenFlow Switch OpenFlow Switch

Flow Routing vs. Aggregation Both models are possible with OpenFlow Flow-Based Every flow is individually set up by controller Exact-match flow entries Flow table contains one entry per flow Good for fine grain control, e.g. campus networks Aggregated One flow entry covers large groups of flows Wildcard flow entries Flow table contains one entry per category of flows Good for large number of flows, e.g. backbone

Reactive vs. Proactive (pre-populated) Both models are possible with OpenFlow First packet of flow triggers controller to insert flow entries Efficient use of flow table Every flow incurs small additional flow setup time If control connection lost, switch has limited utility Proactive Controller pre-populates flow table in switch Zero additional flow setup time Loss of control connection does not disrupt traffic Essentially requires aggregated (wildcard) rules

Usage examples Alice’s code: Simple learning switch Per Flow switching Network access control/firewall Static “VLANs” Her own new routing protocol: unicast, multicast, multipath Home network manager Packet processor (in controller) IPvAlice VM migration Server Load balancing Mobility manager Power management Network monitoring and visualization Network debugging Network slicing What is possible in the controller? Anything that needs intelligent routing of a flow At Stanford, we have even shown how OpenFlow may be used for: VM migration Power management Load balancing Network monitoring and debugging Easier network visualization … and much more you can create! 23 23

Quiz Time How do I provide control connectivity? Is it really clean slate? Why aren’t users complaining about time to setup flows over OpenFlow? (Hint: What is the predominant traffic today?) Considering switch CPU is the major limit, how can one take down an OpenFlow network? How to perform topology discovery over OpenFlow-enabled switches? What happens when you have a non-OpenFlow switch inbetween? What if there are two islands connected to same controller? How scalable is OpenFlow? How does one scale deployments?

What can you not do with OpenFlow ver1.0 Non-flow-based (per-packet) networking ex. Per-packet next-hop selection (in wireless mesh) yes, this is a fundamental limitation BUT OpenFlow can provide the plumbing to connect these systems Use all tables on switch chips yes, a major limitation (cross-product issue) BUT an upcoming OF version will expose these

What can you not do with OpenFlow ver1.0 New forwarding primitives BUT provides a nice way to integrate them through extensions New packet formats/field definitions BUT a generalized OpenFlow (2.0) is on the horizon Optical Circuits BUT efforts underway to apply OpenFlow model to circuits Low-setup-time individual flows BUT can push down flows proactively to avoid delays

Where it’s going OF v1.1: Extensions for WAN, spring 2011 OF v2+ multiple tables: leverage additional tables tags and tunnels multipath forwarding OF v2+ generalized matching and actions: an “instruction set” for networking

OpenFlow Implementations (Switch and Controller) 28

OpenFlow building blocks Monitoring/ debugging tools oftrace oflops openseer Stanford Provided ENVI (GUI) LAVI n-Casting Expedient Applications NOX Beacon Helios Maestro SNAC Controller Slicing Software FlowVisor Console FlowVisor There are components at different levels that work together in making it work The commercial switch details will follow in next slide There are a plethora of applications possible. I only list those available at Stanford Commercial Switches Stanford Provided Software Ref. Switch NetFPGA Broadcom Ref. Switch HP, NEC, Pronto, Juniper.. and many more OpenFlow Switches OpenWRT PCEngine WiFi AP OpenVSwitch 29

Current SDN hardware More coming soon... Juniper MX-series WiMax (NEC) NEC IP8800 UNIVERGE PF5240 WiMax (NEC) HP Procurve 5400 Netgear 7324 PC Engines Pronto 3240/3290 Ciena Coredirector More coming soon... 30

Commercial Switch Vendors Model Virtualize Notes HP Procurve 5400zl or 6600 1 OF instance per VLAN LACP, VLAN and STP processing before OpenFlow Wildcard rules or non-IP pkts processed in s/w Header rewriting in s/w CPU protects mgmt during loop NEC IP8800 Series and UNIVERGE PF5240 OpenFlow takes precedence Most actions processed in hardware MAC header rewriting in h/w More than 100K flows (PF5240) Pronto 3240 or 3290 with Pica8 or Indigo firmware 1 OF instance per switch No legacy protocols (like VLAN and STP) All support ver 1.0 All have approx 1500 flow table entry limit 31

Controller Vendors Vendor Notes Nicira’s NOX Nicira’s ONIX SNAC Vendor Open-source GPL C++ and Python Researcher friendly Nicira’s ONIX Closed-source Datacenter networks SNAC Code based on NOX0.4 Enterprise network C++, Python and Javascript Currently used by campuses Vendor Notes Stanford’s Beacon Open-source Researcher friendly Java-based BigSwitch controller Closed source Based on Beacon Enterprise network Maestro (from Rice Univ) Based on Java NEC’s Helios Written in C and Ruby NEC UNIVERGE PFC Based on Helios 32

Providers and business-unit Growing Community Vendors and start-ups Providers and business-unit More... More... Note: Level of interest varies 33

Software-Defined Networking (SDN) 34

Current Internet Closed to Innovations in the Infrastructure Closed App Operating System App Specialized Packet Forwarding Hardware Operating System App Specialized Packet Forwarding Hardware Operating System The next 3 slides are a set of animation to show how we enable innovation: - Infrastructure is closed to innovation and only driven by vendors. Consumers have little say - Business model makes it hard for new features to be added App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware App Operating System Specialized Packet Forwarding Hardware 35

“Software Defined Networking” approach to open it Network Operating System App Operating System App Specialized Packet Forwarding Hardware Operating System App Specialized Packet Forwarding Hardware Operating System How do we redefine the architecture to open up networking infrastructure and the industry! By bring to the networking industry what we did to the computing world App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware App Operating System Specialized Packet Forwarding Hardware 36

The “Software-defined Network” 2. At least one good operating system Extensible, possibly open-source 3. Well-defined open API App App App Network Operating System 1. Open interface to hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Switches, routers and other middleboxes are dumbed down The key is to have a standardized control interface that speaks directly to hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware 37

Mininet Setup h2 h3 h4 Controller port6633 c0 OpenFlow Tutorial loopback (127.0.0.1:6633) OpenFlow Tutorial 3hosts-1switch topology s1 OpenFlow Switch loopback (127.0.0.1:6634) dpctl (user space process) s1-eth0 s1-eth1 s1-eth2 h1-eth0 h3-eth0 h4-eth0 h2 10.0.0.2 h3 10.0.0.3 h4 10.0.0.4 virtual hosts

Switch Configuration Setup Of-demo-5 [eth1] 192.168.40.10 00:1b:21:5d:4c:3d Of-demo-6 [eth1] 192.168.40.11 00:1b:21:5d:4e:c5 GUI A13 13 A14 Disabled Backup 14 hpsw2 15 necsw5 A15 Controller 13 14 Pronto 3290 172.27.75.33 http://www.openflow.org/wk/index.php/OpenFlowTutorialGEC10

Virtualizing OpenFlow 40

Virtualization or “Slicing” Trend Controller 1 App Controller 2 Virtualization or “Slicing” OpenFlow NOX (Network OS) Network OS App App App Windows (OS) Linux Mac OS Windows (OS) Linux Mac OS Windows (OS) Linux Mac OS Virtualization layer x86 (Computer) Hidden slide (just for backup reasons) Shows how far along we can go in opening up the network Computer Industry Network Industry 41

Virtualization or “Slicing” Layer Isolated “slices” Many operating systems, or Many versions App Network Operating System 1 Network Operating System 2 Network Operating System 3 Network Operating System 4 Open interface to hardware Virtualization or “Slicing” Layer Open interface to hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware

Switch Based Virtualization Exists for NEC, HP switches but not flexible enough Controller Research VLAN 2 Flow Table Controller Research VLAN 1 Flow Table Production VLANs Normal L2/L3 Processing Experiments running on PRODUCTION infrastructure Key to get scale, key to get traffic on the network (e.g. can’t just do a reset...)

FlowVisor-based Virtualization Heidi’s Controller Craig’s Controller Aaron’s Controller Topology discovery is per slice OpenFlow Protocol OpenFlow FlowVisor & Policy Control OpenFlow Switch OpenFlow Protocol OpenFlow Switch OpenFlow Switch 44

FlowVisor-based Virtualization http Load-balancer Multicast Broadcast Separation not only by VLANs, but any L1-L4 pattern OpenFlow Protocol dl_dst=FFFFFFFFFFFF tp_src=80, or tp_dst=80 OpenFlow FlowVisor & Policy Control OpenFlow Switch OpenFlow Protocol OpenFlow Switch OpenFlow Switch

FlowSpace: Maps Packets to Slices

FlowVisor Message Handling OpenFlow Firmware Data Path Alice Controller Bob Cathy FlowVisor Rule Policy Check: Is this rule allowed? Policy Check: Who controls this packet? Full Line Rate Forwarding Exception Packet Packet

Use Case: New CDN - Turbo Coral ++ Basic Idea: Build a CDN where you control the entire network All traffic to or from Coral IP space controlled by Experimenter All other traffic controlled by default routing Topology is entire network End hosts are automatically added (no opt-in) Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport * 84.65.* *

Use Case: Aaron’s IP A new layer 3 protocol Replaces IP Defined by a new Ether Type Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport * AaIP * !AaIP

OpenFlow/GENI Deployments 50

OpenFlow Deployment at Stanford Switches (23) APs (50) WiMax (1)

Live Stanford Deployment Statistics http://yuba.stanford.edu/ofhallway/wide-ofv1.html

How did we get there? Staged Deployment of OpenFlow Add experimental VLAN Enable OpenFlow for Exptl VLAN Configure Controller for new network Add new Production subnetwork Gradually add/move users to new subnet Enable OpenFlow for new subnet Slice the network Verify correctness and performance Verify reachability

GENI OpenFlow deployment (2010) 10 institutions and 2 National Research Backbones Kansas State

GENI Network Evolution National Lambda Rail

GENI Integration FlowVisor Expedient Opt-in Manager Slicing control Experimenter’s portal for slice management Opt-in Manager Network admins’ portal to approve/ deny expt requests for traffic Expedient3 GENI API API X Expedient1 Expedient2 API X API X Opt-in Mgr1 Opt-in Mgr2 FlowVisor API FlowVisor API FlowVisor1 FlowVisor2 OpenFlow OpenFlow Substrate 1 Substrate 2

Mistakes we made OpenFlow over Q-in-Q OpenFlow routing is unaware and sends traffic with same MAC address in both direction, causing perpetual learning and CPU inflation Moving uplinks for 1 switch, while being pointed to the same controller (causing two islands) Causes controller learning to oscillate between the 2 uplinks Bad interaction with legacy protocols LLDP and STP are treated differently with different switches Loop in OpenFlow network being exposed to non-OF side Miscommunication between the aggregate operator and the experimenter during testing phase Loop across backbones Same campus connected over NLR and Internet2

Next steps for GENI Remove duct-tape Be better prepared Grow topology Fix any issues that we learned about during demos Be better prepared Higher stability and better isolation Test bandwidth slicing Grow topology Add more switches and hosts Wean off Each campus takes charge of control framework 33 bugs to be fixed in FlowVisor

Tutorial Setup so far h2 h3 h4 Controller port6633 c0 loopback (127.0.0.1:6633) OpenFlow Tutorial 3hosts-1switch topology s1 OpenFlow Switch loopback (127.0.0.1:6634) dpctl (user space process) s1-eth0 s1-eth1 s1-eth2 h1-eth0 h3-eth0 h4-eth0 h2 10.0.0.2 h3 10.0.0.3 h4 10.0.0.4 virtual hosts 59

Virtualized Network Setup Controller port 7000 loopback (127.0.0.1:7000) c0 FlowVisor port 6633 OpenFlow Tutorial 3hosts-1switch topology loopback (127.0.0.1:6634) s1 OpenFlow Switch loopback (127.0.0.1:6634) dpctl (user space process) s1-eth0 s1-eth1 s1-eth2 h1-eth0 h3-eth0 h4-eth0 h2 10.0.0.2 h3 10.0.0.3 h4 10.0.0.4 virtual hosts 60

FlowVisor Usage man ./doc/flowvisor.8 <flowvisor dir>/script/fvctl.sh listDevices list of all OpenFlow switches’ datapath ID getLinks list of all links (port # and datapath ID of both end switches) createSlice creating slice (specifying controller’s URL and slice name) listSlices addFlowSpace add flow space to slices listFlowSpace show current flow space 61

Concluding Remarks 62

Highlights of Deployments Stanford deployment McKeown group for 1.5 years: production and experiments To scale later this year to entire building (~500 users) Nation-wide trials and deployments 7 other universities and BBN deploying now GEC9 in Nov, 2010 showcased nation-wide OF Internet 2 and NLR starting to serve as the GENI Backbone Global trials Over 60 organizations experimenting 2011 likely to be a big year for OpenFlow

Current Trials 68 trials/deployments spanning 13 countries

Get involved! Ask and answer questions on mailing lists: openflow-discuss openflow-spec Share and update wiki content Submit bug-reports and/or patches to OF reference implementation Release open-source applications Write a controller!

Are you innovating in your network?

SDN Team at Stanford