AmLight-ExP: Collaboration at 100G LSST 2016 Project and Community Workshop Tucson, AZ – Aug 18th Long Haul Networks session AmLight-ExP: Collaboration at 100G Jeronimo Bezerra <jab@amlight.net>

Outline 100 G managed ring status June 23th Network Design Meeting Current Status Next Steps June 23th Network Design Meeting

100G managed ring status

AmLight-ExP’s Bandwidth Evolution (1/3) 2013 to December 2015: 6 x 10G 40 Gbps between South America and Miami 10 Gbps between Sao Paulo and Fortaleza/Brazil 10 Gbps between Brazil and Chile January 2016: 2 x 10G added 60 Gbps between South America and Miami

AmLight-ExP’s Bandwidth Evolution (2/3) April 2016: 1 x 100G “delivered” Miami to Sao Paulo via Atlantic First 100G link in the SACS cable Very unstable Problems with cross-connections, switches and optical devices June 2016: 1 x 100G added Miami to Sao Paulo via Pacific Second 100G link in the SACS cable Also unstable, same reasons 100G Atlantic link still being tested, still unstable

AmLight-ExP’s Bandwidth Evolution (3/3) End of June 2016: Finally, stability! Brocade 100G modules replaced due to interoperability issues with CIENA CIENA cards replaced and operational system replaced Terremark’s cross-connections and IFCs re-cleaned (n-times) July 2016: AMPATH switches upgraded with new modules and OS 100G links added to AmLight-ExP network 2 x 10G links between Sao Paulo and Miami decommissioned Total Capacity: 240 Gbps between South America and Miami 10 Gbps between Sao Paulo and Fortaleza/Brazil 10 Gbps between Brazil and Chile

AmLight-ExP’s Current Topology Aug 2016

Next Steps 100G Atlantic link will be “opened” in Fortaleza/Brazil 2 x 10G links to be decommissioned: Fortaleza - Sao Paulo Fortaleza – Miami Planned for October/16 100G Pacific link will be “opened” in Santiago/Chile: Santiago – Sao Paulo Santiago – Miami Planned for November/16 Total Capacity before Jan 2017: 220 Gbps between South America and Miami 100 Gbps between Sao Paulo and Fortaleza/Brazil 100 Gbps between Brazil and Chile

AmLight-ExP’s Topology Planned by mid 2017

Bandwidth Utilization Current Traffic: Testing Traffic:

AmLight-ExP – Plans for 2017 Activation of Spectrum between Sao Paulo and Miami Vendors and solutions are being evaluated Activation of Spectrum between Miami and Jacksonville Single solution planned for both activations

June 23th Network Design Meeting

June 23th Network Design Meeting Two presentations were provided: Jeronimo Bezerra presented some optical definitions, inter-domain connectivity options, redundancy approaches and interfaces to be evaluated by the LSST Engineering Team Sandra Jaque presented REUNA’s network expansion plan in course, including services to be offered to the LSST community

Connectivity Presentation (1/5) Main Connectivity Goal: End-to-End optical connectivity as the primary path Redundancy using Ethernet circuits

Connectivity Presentation (2/5) Provisioning Challenges: Is end to end optical across optical domains possible without “domain-based” regeneration? How can spectral width and frequency requirements be coordinated? How can channels be modeled and supported across domains with varying optical power and signal to noise requirements? Can super-channel capabilities be leveraged? Can the interconnected optical networks be protected from one another? There are obviously many others as well.

Connectivity Presentation (3/5) Provisioning Challenges Suggestion 01: End to end optical network Each domain interconnection would have back-to-back terminal transponders: Terminal Transponders could be from different vendors, each TT facing a different carrier (regenerators are a single device) Alien Wave is an option in case of different vendors Between Terminal Transponders a standard interface will be available (Ethernet, for instance) In case of packet loss, circuit could be manually opened to isolate spans Huge OPEX and network outages in order to isolate a problem For example, the first AmLight’s 100G activation took 45 days to isolate the problem with just one network in the path! Having optical devices per-site implies huge CAPEX Redundancy is completely isolated from the optical circuit (linear circuits) Technology-independent (RDMA, Ethernet, FibreChannel, etc.) Suggestion 02: Optical + Dedicated Ethernet network Each domain interconnection would have a dedicated 100G switch with tunable transceivers Tunable transceivers would be alien waves Dedicated switches mean dedicated services (no bandwidth will be shared – almost deterministic) In case of packet loss, perfSonar servers connected to these switches could be used Problem isolation is simplified Having switches per-site implies less CAPEX than optical devices Better integration with the protected environment Technology-dependent: Ethernet

Connectivity Presentation (4/5) Operational Challenges: End-to-End optical connectivity as the primary path Network troubleshooting will be a complex activity Redundancy won’t be optical (linear circuits) Redundancy using Ethernet circuits New services require coordination between multiple teams

Connectivity Presentation (5/5) Complex Troubleshooting:

Simplified View of the end to end path

Next Steps? LSST Eng Team will need to decide for the final provisioning approach LSST Eng Team will need to define the vendor(s) to work with Each Optical Domain will need to provide information about its fibers and optical measurements Each Optical Domain will need to assign one or more channels to be used by LSST Device’s specifications need to be finished by mid-2017 Field trials need to start in the end of 2017

AmLight-ExP: Collaboration at 100G Questions? LSST 2016 Project and Community Workshop Tucson, AZ – Aug 18th Long Haul Networks session AmLight-ExP: Collaboration at 100G Questions? Jeronimo Bezerra <jab@amlight.net>