1. Jeronimo Bezerra Florida International University TNC 2016 Networking Conference June 14th Coexisting Production and Experimental Testbeds: The AmLight experience

2. Outline Introduction to AmLight Network Testbeds Architecture Methodology Findings Future 2

3. Describing AmLight GLIF Open Lightpath Exchanges: – AMPATH – Miami/USA – SouthernLight – Sao Paulo/Brazil NAP: – AndesLight – Santiago/Chile – Fortaleza/Brazil 160 Gbps between Latin America and the U.S. – Second 100Gbps link being installed 120Gbps to Internet2 Partnership involving FIU, NSF, ANSP, RNP, RedClara and AURA AmLight is a Distributed Academic Exchange Point

4. AmLight SDN (1/2) Production SDN Infrastructure (since Aug 2014) Carries Academic and Non-Academic/Commercial traffic – L2VPN, IPv4, IPv6, Multicast Supports Network Programmability/Slicing – OpenFlow 1.0 – Flow Space Firewall for Network Programmability/Slicing – OESS for L2VPNs – NSI(OpenNSA+OESS) and OSCARS enabled – ONOS/SDN-IP for Academic IPv4 – Currently 5 slices for experimentation (including Global ONOS SDN-IP) 4

5. AmLight SDN (2/2) 5

6. Network Testbeds at AmLight SDN (1/2) Network Testbeds offered through Network Slices: Network Slices: Defined by a set of Interfaces and VLANs Each Slice has its own Openflow Controller Different Topologies Available How does AmLight support slices? Internet2 Flow Space Firewall (FSF) is being used to create slices FSF talks OpenFlow 1.0 to controllers and network devices Provides isolation between slices Filters OpenFlow messages based on Interfaces and VLANs Support filters: # of flows inserted and flows inserted per second. Supports a high # of parallel slices 6

7. Openflow (1.0, 1.3 in the future) – Through dedicated slices – Real devices (Brocade MLXe) – Own VLAN range – Different virtual topologies available – Layer 2 and Layer 3 matches – Low level configuration NSI v2 – Network Service Interface – High level abstraction for layer 2 multi- domain provisioning – No need to know the topology and physical devices/configurations Two possible interfaces to use AmLight: Network Testbeds at AmLight SDN (2/2)

8. Examples – ONOS/SDN-IP @ ONS 2016 8

9. Examples (3) – And more… 9 In partnership with RNP: – FIBRE (Future Internet testbeds / experimentation between BRazil and Europe): how to use an OpenFlow native backbone to interconnect FIBRE islands (or racks)? – FIBRE island installed at AMPATH/Miami and using AmLight In partnership with Internet2: – Internet2 Technology Exchange 2014 – Multi Domain controller managing slices from different SDN domains (Internet2, AmLight, Univ. of Utah and MAX) – Internet2 Global Summit – ONOS SDN-IP demonstration In partnership with University of Twente: – “Assessing the Quality of Flow Measurements from OpenFlow Devices” – Authors: Luuk Hendriks, Ricardo de O. Schmidt, Ramin Sadre, Jeronimo A. Bezerra, and Aiko Pras All of them running on the same production infrastructure

10. Eighteen Months Later: Lessons Learned Researchers expectations: 10 AmLight possibilities: Main Challenge Today is to Balance Expectations! We should avoid more obstacles to researchers! Each new Network Testbed is a new challenge: new apps, new methodology and always complex! “Need” full access to everything! Requires a lot of singularities: Untagged VLANs, Reactive OpenFlow Mode, All Actions, All Matches, Direct Access to the OpenFlow devices, … It’s a Shared Environment! Complexity involved for “major” changes: Proactive Mode, Untagged VLAN, etc.

11. Security vs Programmability FACT: OpenFlow agents deployed are still “experimental”: – Unsupported messages could crash and OpenFlow device How to guarantee experimental applications won’t affect my “production” slice? FlowSpace Firewall slices based on : – No extra filters are possible at this moment Multiple OF controllers could manage the same OpenFlow device: – Complicated to isolate who is sending specific OF messages Troubleshooting gets very still complicated: – Logs provided by the SDN stack is still poor 11

12. New Architecture – Proof of Concept Two Layers of Virtualization – Main/Production Layer – Experimentation Layer Experimentation Layer had a “Sanitizer” module added: – Controls what OpenFlow messages can be sent to the “Physical Layer” – Allows filters per OpenFlow Type, per-match and per- action – Off-loads switches from unsupported OpenFlow messages Sanitizer logs transactions and filters based on dictionaries: – XML files created as result of OF Tests – Detailed logs per slice or per type of message OpenFlow Sniffer keeps monitoring all communication – To help vendors in their troubleshooting activities 12

13. Testbed Sanitizer (1/3) Tests with OFTests: – Each device, software version and line card type is stressed in lab – Unsuccessful tests are collected and processed – When a specific match or action is not supported, it is added to the dictionary – Unsupported OpenFlow messages are added to the XML dictionary Filters are stateless: – Less powerful but easier to deploy and faster 13

14. Testbed Sanitizer (2/3) ONOS/SDN-IP vs Brocade CES: – ONOS/SDN-IP sends all flows in a single batch command – Brocade CES doesn’t support MAC rewrite – ONOS used to log “batch failed” – Tcpdump had to be used – Satinizer’s dictionary has a “CES and Mac-rewrite don’t mix” entry and log it Brocade CES NI 5.7 vs OpenFlow Vendor type: – Some OpenFlow messages type Vendor were forcing Brocade CES to restart the OpenFlow connection – Satinizer’s Dictionary has a “CES 5.7 doesn’t take unknown Vendor ID” filter and log it OESS Forwarding Verification vs Brocade MLX-4 4x10G line card: – Ethertype 0x88b6 not support, internal trace logs rotating too fast – Satinizer’s Dictionary has a “LP 4x10G and Etype “A” don’t mix” filter and log it 14

15. Testbed Sanitizer (3/3) Off-loading some filters help switches to focus on “supported” features – Also preserves switches internal trace logs queue New per-slice logging helps to identify which application sent a specific OpenFlow message – Helps researcher to improve his/her SDN application Troubleshooting logs helps vendors to reproduce the issue 15

16. Lessons Learned Testbed Sanitizer was a proof-of-concept to understand how complex and deep the problem was Future is unclear: should we develop a production sanitizer? Or should we “force” vendors to create a better code? Stateful filters are very important, but they are very complex to deploy OpenFlow deployments still have a lot to evolve Lack of troubleshooting tools Some tools/scripts developed to help the SDN operation: – OpenFlow Sniffer – Zabbix Plugins – SDNTrace – Testbed Sanitizer OF 1.3 will be even more complicated: meters, multi-tables, etc. 16

17. Future Challenges How to scale and support high # of parallel network testbeds? How to manage testbeds in a production network? While we learning, new testbeds need to be implemented: SDX How to migrate network devices between OF versions? – AmLight plans to migrate to OpenFlow 1.3 this year 17

18. Jeronimo Bezerra Florida International University TNC 2016 Networking Conference June 14th More info: www.sdn.amlight.net Questions?