1. AmLight-ExP: Collaboration at 100G
LSST 2016 Project and Community Workshop
Tucson, AZ – Aug 18th
Long Haul Networks session
AmLight-ExP: Collaboration at 100G
Jeronimo Bezerra

2. Outline 100 G managed ring status June 23th Network Design Meeting
Current Status
Next Steps
June 23th Network Design Meeting

3. 100G managed ring status

4. AmLight-ExP’s Bandwidth Evolution (1/3)
2013 to December 2015: 6 x 10G
40 Gbps between South America and Miami
10 Gbps between Sao Paulo and Fortaleza/Brazil
10 Gbps between Brazil and Chile
January 2016: 2 x 10G added
60 Gbps between South America and Miami

5. AmLight-ExP’s Bandwidth Evolution (2/3)
April 2016: 1 x 100G “delivered”
Miami to Sao Paulo via Atlantic
First 100G link in the SACS cable
Very unstable
Problems with cross-connections, switches and optical devices
June 2016: 1 x 100G added
Miami to Sao Paulo via Pacific
Second 100G link in the SACS cable
Also unstable, same reasons
100G Atlantic link still being tested, still unstable

6. AmLight-ExP’s Bandwidth Evolution (3/3)
End of June 2016: Finally, stability!
Brocade 100G modules replaced due to interoperability issues with CIENA
CIENA cards replaced and operational system replaced
Terremark’s cross-connections and IFCs re-cleaned (n-times)
July 2016:
AMPATH switches upgraded with new modules and OS
100G links added to AmLight-ExP network
2 x 10G links between Sao Paulo and Miami decommissioned
Total Capacity:
240 Gbps between South America and Miami
10 Gbps between Sao Paulo and Fortaleza/Brazil
10 Gbps between Brazil and Chile

7. AmLight-ExP’s Current Topology
Aug 2016

8. Next Steps 100G Atlantic link will be “opened” in Fortaleza/Brazil
2 x 10G links to be decommissioned:
Fortaleza - Sao Paulo
Fortaleza – Miami
Planned for October/16
100G Pacific link will be “opened” in Santiago/Chile:
Santiago – Sao Paulo
Santiago – Miami
Planned for November/16
Total Capacity before Jan 2017:
220 Gbps between South America and Miami
100 Gbps between Sao Paulo and Fortaleza/Brazil
100 Gbps between Brazil and Chile

9. AmLight-ExP’s Topology Planned by mid 2017

10. Bandwidth Utilization
Current Traffic:
Testing Traffic:

11. AmLight-ExP – Plans for 2017
Activation of Spectrum between Sao Paulo and Miami
Vendors and solutions are being evaluated
Activation of Spectrum between Miami and Jacksonville
Single solution planned for both activations

12. June 23th Network Design Meeting

13. June 23th Network Design Meeting
Two presentations were provided:
Jeronimo Bezerra presented some optical definitions, inter-domain connectivity options, redundancy approaches and interfaces to be evaluated by the LSST Engineering Team
Sandra Jaque presented REUNA’s network expansion plan in course, including services to be offered to the LSST community

14. Connectivity Presentation (1/5)
Main Connectivity Goal:
End-to-End optical connectivity as the primary path
Redundancy using Ethernet circuits

15. Connectivity Presentation (2/5)
Provisioning Challenges:
Is end to end optical across optical domains possible without “domain-based” regeneration?
How can spectral width and frequency requirements be coordinated?
How can channels be modeled and supported across domains with varying optical power and signal to noise requirements?
Can super-channel capabilities be leveraged?
Can the interconnected optical networks be protected from one another?
There are obviously many others as well.

16. Connectivity Presentation (3/5) Provisioning Challenges
Suggestion 01: End to end optical network
Each domain interconnection would have back-to-back terminal transponders:
Terminal Transponders could be from different vendors, each TT facing a different carrier (regenerators are a single device)
Alien Wave is an option in case of different vendors
Between Terminal Transponders a standard interface will be available (Ethernet, for instance)
In case of packet loss, circuit could be manually opened to isolate spans
Huge OPEX and network outages in order to isolate a problem
For example, the first AmLight’s 100G activation took 45 days to isolate the problem
with just one network in the path!
Having optical devices per-site implies huge CAPEX
Redundancy is completely isolated from the optical circuit (linear circuits)
Technology-independent (RDMA, Ethernet, FibreChannel, etc.)
Suggestion 02: Optical + Dedicated Ethernet network
Each domain interconnection would have a dedicated 100G switch with tunable transceivers
Tunable transceivers would be alien waves
Dedicated switches mean dedicated services (no bandwidth will be shared – almost deterministic)
In case of packet loss, perfSonar servers connected to these switches could be used
Problem isolation is simplified
Having switches per-site implies less CAPEX than optical devices
Better integration with the protected environment
Technology-dependent: Ethernet

17. Connectivity Presentation (4/5)
Operational Challenges:
End-to-End optical connectivity as the primary path
Network troubleshooting will be a complex activity
Redundancy won’t be optical (linear circuits)
Redundancy using Ethernet circuits
New services require coordination between multiple teams

18. Connectivity Presentation (5/5)
Complex Troubleshooting:

19. Simplified View of the end to end path

20. Next Steps?
LSST Eng Team will need to decide for the final provisioning approach
LSST Eng Team will need to define the vendor(s) to work with
Each Optical Domain will need to provide information about its fibers and optical measurements
Each Optical Domain will need to assign one or more channels to be used by LSST
Device’s specifications need to be finished by mid-2017
Field trials need to start in the end of 2017

21. AmLight-ExP: Collaboration at 100G Questions?
LSST 2016 Project and Community Workshop
Tucson, AZ – Aug 18th
Long Haul Networks session
AmLight-ExP: Collaboration at 100G Questions?
Jeronimo Bezerra