1. TNC15 Networking Conference
Use of SDN in the AmLight intercontinental research and education network
The main idea of this presentation is to share our experience migrating a production layer 2 network to Software-Defined Network with Openflow. AmLight migrated its network links in the last half of I would like to share some results achieved.
TNC15 Networking Conference
15-18 June 2015
Porto, Portugal
Julio Ibarra, Principal Investigator
Heidi Morgan, Co-Principal Investigator
Jeronimo Bezerra, Chief Network Engineer

2. Who we are Partners: FIU, NSF, ANSP, RNP, RedCLARA, REUNA and AURA
AMPATH:
Academic International Exchange Point (IXP) in Miami, Florida
Interconnects Latin America and Caribbean RENs to other RENs in the world
SouthernLight:
Academic IXP in São Paulo, Brazil
Interconnects all Brazilian RENs and RedCLARA
AmLight:
International network links that connect the U.S. to Latin America
Partners: FIU, NSF, ANSP, RNP, RedCLARA, REUNA and AURA
First, some background on who we are, in the event you are not familiar with AmLight.
Resources:
AMPATH
SouthernLight
International Links
Partners:
Florida International University, the U.S. National Science Foundation, the Academic Network of Sao Paulo (ANSP), the Brazilian National Education and Research Network (RNP), the regional network of Latin America (RedCLARA) , Chile’s National Research and Educational Network (REUNA), and the Association of Universities for Research in Astronomy (AURA) established an international collaboration. The purpose of this collaboration is to facilitate and connect all academic networks in Latin America and the Caribbean to the U.S., and then, reach all Research and Educational Networks around the globe.
This presentation will focus on the network connections to South America.

3. AmLight Today Connections: 13 RENs
A set of 4 x 10G links with two topologies:
SDN (Layer 2) Ring:
Miami-São Paulo-Santiago-Miami (green)
MPLS (Layer 3) Ring:
Miami-Fortaleza-Rio-São Paulo-Miami (yellow)
Later this year: 100G link between São Paulo and Miami
Mutual Redundancy
This slide represents the AmLight network. Both, physical and logical representations are shown.
All of them have10Gbps of capacity, and they operate in different submarine cables to increase resilience.
Topology:
Using these links, two topologies were created:
One Academic Layer 2 Ring, connecting Miami to Sao Paulo, then Santiago to Miami and
One Academic Layer 3 Ring, connection Miami to Fortaleza, Rio de Janeiro, Sao Paulo then Miami (links in green and red)
Map represents the physical network topology, and colors represent carriers:
Red is Telefonica, Green is Lan Nautilus.
The Layer 2 ring was created using Brocade switches, with VLANs and per-Vlan Rapid Spanning Tree (per-VLAN RSTP).
The Layer 3 ring was created using Juniper routers with MPLS.
A 100G link between Miami and Sao Paulo will be added in August; first for experimentation, then for both production and experimentation.
Mutual Redundancy:
Some numbers about AmLight:
today we have 13 RENs connected to AmLight, directly or indirectly through another network operator. In total, more than one thousand universities and research centers in Latin America use these links.
The focus of this presentation will be the Academic Layer 2 ring, where SDN is operating.
Connections:
13 RENs
> 1000 Universities and Research Centers

4. AmLight Before SDN Configuration of circuits was based on static VLANs
High degree of coordination between multiple network teams
Multiple instances of per-VLAN RSTP were used
Interoperability issues
Constrained redundancy with network operators
Redundancy between rings was created with:
IEEE 802.1ad (QinQ) + L2VPNs
Additional ports to implement redundancy across rings
Prior to OpenFlow/SDN, AmLight was configured in the following way:
All VLANs were static and manually provisioned.
Every new Vlan required some coordination between multiple network teams.
Second, network resilience was based on per-VLAN RSTP. Per-Vlan RSTP works fine, but it’s a Brocade proprietary protocol. The standard RSTP does not support per-VLAN trees. This caused interoperability issues trying to build redundancy with other network operators.
A redundancy approach was created between both the Layer 2 and MPLS rings using QinQ.
With QinQ, all VLANs from the Layer 2 ring could be mapped inside L2VPNs in the MPLS ring, as if a single pseudowire.
Three 10Gig ports were used to make the QinQ and the MPLS pseudowire work in this scenario. There was a high CapX cost in the implementation of the redundancy design.
With this approach, AmLIght achieved 100% availability in 2013, because critical network components had multiple levels of redundancy.
Message: AmLight was a stable production environment supporting international R&E networking.

5. Why then move towards SDN?
Key motivations:
Improving operations efficiency
Introducing network programmability
Two main reasons to think about SDN:
Improving operations efficiency
And to
Introduce network programmability to AmLight
I will provide reasons in the next slides.

6. Motivation 01: Improving Operations Efficiency
Requests for Layer2 circuits was increasing
Provisioning process was complex
Some circuits involved up to seven different networks
Requiring a high level of coordination
Engaging diverse network teams
Multiple technologies were involved
From Layer 1 to MPLS
Some circuits took weeks or even months to be provisioned
In this context, operations efficiency refers to the process of provisioning layer 2 circuits.
Number of requests for circuits:
The number of layer2 circuit requests has been increasing. Some are temporary for use in demonstrations at conferences, such as Super computing, GLIF and others.
Provisioning some of these circuits involved up to seven different networks, which required a high level of coordination.
Multiple technologies:
Multiple technologies was a factor that added complexity to the provisioning process. Ethernet was the common end-end technology? Right?
Some of these networks have different technologies, for example VLAN, MPLS, SDN, Layer 1 circuits, etc. More different technologies involved, more complex it gets. Specially when we need to troubleshoot it.
Testing with multiple network technologies makes troubleshooting more complex. Before IP addresses could be added at multiple interfaces to run ping tests.
This complexity extended the provisioning activity at times to several weeks. As a result, it was necessary to find a way to improve the provisioning process.

7. Motivation 02: Introducing Network Programmability
Lack of support for network programmability limited applications
Little to no support for network-aware applications
Researchers could only view the network status (SNMP)
AmLight could only provide a view of the network to its researchers. Support for network-aware applications was lacking.
I want to create a path on demand. I want to do forwarding based on other fields besides MAC address.
Depending on its state, an application might want to request more bandwidth, on-demand.
It was an important objective of AmLight to support network-aware applications across international links.
SNMP was the only resource offered to researchers to view the status of the network.
Only one case by a researcher in Brazil. Rogerio?

8. Scenario Deployed (1/2) Activated OpenFlow 1.0 + Hybrid Ports
Improving operations efficiency:
Internet2’s OESS
OSCARS - IDCP
OpenNSA - NSI
B. Introducing network programmability
Internet2’s Flow Space Firewall
Hybrid ports is a feature of Brocade, supporting OpenFlow and non-OF traffic simultaneously on the same port. Works reliably.
OESS is a local orchestrator. It takes care of the local domain.
OSCARS AND OpenNSA provided inter-domain capabilities.
To deploy a production vlan, we have been using OESS, because OESS uses OSCARS for inter-domain provisioning. OSCARS uses IDCP protocol created by I2 and Esnet. It’s not a standard.
OpenNSA can also be used by applications for dynamic provisioning. OpenNSA uses the NSI protocol, which is a standard.
Operations efficiency is not only provisioning. The OESS controller is also used for troubleshooting, statistics, utilization,
Why OESS? Only one available for production. OESS can orchestrate transport for all the switches in the end-to-end path. Chosen by I2 as production software.
Programmability:
Chose Internet2’s Flow Space Firewall (FSF) to provide programmability.
FSF introduces a hypervisor that allows for the creation of virtual networks.
Using FSF, a slice can be created to represent a virtual network.
A slice may consist of a set of descriptions for switches, ports, vlans and other objects that can be virtualized.

9. Scenario Deployed (2/2)
This diagram represents the SDN architecture implemented on AmLight.
Purple boxes are testbeds.
Each testbed is isolated by using its own private slice.
Used by researchers.
Yellow boxes represent production applications:
Peach colored boxes represents the virtualization layer, providing interfaces between the physical devices and the network applications.
Describe from bottom to top.

10. Findings (1/2) A. Improving operations efficiency
Other networks (if IDCP or NSI supported) refers to more networks behind connected NRENS and other AmLight connectors.
Measurements were possible when other networks supported IDCP or NSI? Numbers in the before column apply?
Dashes mean same as before SDN

11. Findings (2/2) B. Introducing network programmability
We only offer OpenFlow to talk to our devices. Application must be able to talk OpenFlow.
Provisioning Defined by the User:
User controls the provisioning process.
Flow controlled hop-by-hop:
Eg., changing TTL on a per-hop basis. Changing the MAC address per hop. Changing the vlan ID. Having the capability to change an object on a per hop basis.
Network programmability is the main achievement of this project:
Network-aware applications will have AmLight as a real platform for innovation

12. Some Lessons Learned
Legacy protocols and old switching line cards have limitations
LACP, Counters, Ethertypes
Increased complexity of the deployment
Dedicated additional ports to work around these limitations
Out-of-band/Control Plane network could be challenging
Convergence methodology has to be improved
Specially in long-haul links
LACP is not supported by OF. Had to find a work around.
LACP is Link Aggregation Control Protocol.
A few older cards did not support multiple ether types for generic flows.
Generic flows span all ports for inbound traffic. A flow is not generic if an inbound port is specified.
Out-of-band/Control plane network …:
Control messages may not use data ports. RedCLARA’s network was used to estaberlish the connection between the Controller and the OF device.
Control messages must a non OF port.
Convergence methodology:
New circuit must be established when a port or switch status changes.
If port status change (port down) convergence becomes a factor for re-establishing a new circuit.
Longer delays of long haul links may impact the convergence

13. Out-of-band Control Plane Network
Out-of-band network built for transmission of OpenFlow control messages between Controller and OpenFlow devices
RedCLARA IP backbone was used for this solution
Relying on the Control Plane of another backbone. What if you do not have a partner
Relied on a Third party network to reach our devices.
Fortunately, RedCLARA is a partner of AmLight. We were able to partner with them on the solution.
What if a partner network were not avaialble? Would normally have to pay a service provider.

14. Future
Explore and add new features related to troubleshooting and security
Create a Software-Defined Internet Exchange (SDX)
involving AmLight, and
inter-connecting the U.S. and Brazil
Migrate to Openflow 1.3
Metering and improve the network convergence
Explore and add …:
- When things go wrong, we struggle with troubleshooting. We’re exploring new tools and new approaches to ease troubleshooting and to make the environment more secure for production and experimentation.
OF 1.3 addresses convergence issue and legacy protocols, such as LACP. Line cards must still be replaced.
SDX:
Today we have islands. We want them to be able to talk more than L2 and L3. SDX provides a direction.
OF 1.3:
Port groups to address the network convergence issue.

15. Thank You!
NSF AmLight-ExP, AtlanticWave-SDX, OpenWave, AmLight, OSDC-PIRE, CC-NIE, CC*IIE, AMPATH infrastructure, science application support, education, outreach and community building efforts are made possible by funding and support from:
National Science Foundation (NSF) awards ACI , ACI , ACI , ACI , ACI , ACI , ACI , OISE
FAPESP, ANSP – grant no. 2008/
Rede Nacional de Ensino e Pesquisa (RNP)
Association of Universities for Research in Astronomy (AURA)
Florida International University
Latin American Research and Education community
The many national and international collaborators who support our efforts

16. TNC15 Networking Conference
Use of SDN in the AmLight intercontinental research and education network
TNC15 Networking Conference
15-18 June 2015
Porto, Portugal
Julio Ibarra
Principal Investigator