1. LHCOPN lambda and fibre routing Episode 4 (the path to resilience…)
Michael Enrico
Network Engineering & Planning Manager
DANTE
LHC Meeting, Geneva,

2. ?

3. T0-T1 Lambda routing (schematic, previously)
Connect. Communicate. Collaborate
T0-T1 Lambda routing (schematic, previously)
ASGC
Copenhagen
TRIUMF
Via SMW-3 or 4 (?) T1 DK T1
NDGF
??? T1
T0-T1s:
CERN-RAL
CERN-PIC
CERN-IN2P3
CERN-CNAF
CERN-GRIDKA
CERN-NDGF
CERN-SARA
CERN-TRIUMF
CERN-ASGC
USLHCNET NY (AC-2)
USLHCNET NY (VSNL N)
USLHCNET Chicago (VSNL S)
???
BNL T1
RAL
Hamburg
SURFnet
SARA T1 T1
MAN LAN
London NY
Amsterdam NL UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

4. Add T1-T1 Lambda routing (schematic, previously)
Connect. Communicate. Collaborate
Add T1-T1 Lambda routing (schematic, previously)
T0-T1 Lambda routing (schematic, previously)
ASGC
Copenhagen
TRIUMF T1
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1
T0-T1s:
CERN-RAL
CERN-PIC
CERN-IN2P3
CERN-CNAF
CERN-GRIDKA
CERN-NDGF
CERN-SARA
CERN-TRIUMF
CERN-ASGC
USLHCNET NY (AC-2)
USLHCNET NY (VSNL N)
USLHCNET Chicago (VSNL S)
???
T1-T1s:
BNL T1
RAL
SARA
Hamburg
GRIDKA-CNAF
SURFnet T1
GRIDKA-IN2P3 T1
GRIDKA-SARA
MAN LAN
London NY
Amsterdam NL
SARA-NDGF UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

5. SRLG analysis Connect. Communicate. Collaborate DE T0 KEY Atlantic
ASGC
Copenhagen
TRIUMF
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1 T1
???
KEY
GEANT2
NREN
USLHCNET
Via SURFnet
T1-T1 (CBF)
BNL T1
RAL
SARA
Hamburg
SURFnet T1 T1
MAN LAN
London NY NL UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

6. Failure scenarios (2 of them)

7. e.g. Big fibre cut Basel-Geneva
Connect. Communicate. Collaborate
e.g. Big fibre cut Basel-Geneva
ASGC
Copenhagen
TRIUMF T1
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1 T1
MAN LAN
London NY NL UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

8. e.g. Big fibre cut Basel-Geneva
Connect. Communicate. Collaborate
e.g. Big fibre cut Basel-Geneva ?

9. e.g. GÉANT2 Fra-Gen EDFA failure
Connect. Communicate. Collaborate
e.g. GÉANT2 Fra-Gen EDFA failure
ASGC
Copenhagen
TRIUMF T1
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1 T1
MAN LAN
London NY NL UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

10. e.g. GÉANT2 Fra-Gen EDFA failure
Connect. Communicate. Collaborate
e.g. GÉANT2 Fra-Gen EDFA failure ?

11. (Reminder of) Conclusions (from last time)
RAL & PIC need backups!!!
CNAF needs a 3rd route into CERN
SARA-NDGF should be routed on northern leg of NORDUnet’s “Southern Cross” (for Cop-Ham part)
via Padborg, Kolding, Odense
Reroute 2 Amsterdam-CERN lambdas via Lyon
Reroute USLHCNET (FNAL-CERN) via Lyon
Further investigations required in particular concerning:
Physical routing of GRIDKA-IN2P3 in Paris area
Leased lambdas passing through London

12. Mitigations (on the way)…

13. Route SARA-CERN via Lyon
Connect. Communicate. Collaborate
Route SARA-CERN via Lyon
ASGC
Copenhagen
TRIUMF
??? T1
Via SMW-3 or 4 (?) DK T1
NDGF T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1 T1
MAN LAN
London NY NL UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

14. Add USLHCNET lambda #4 Connect. Communicate. Collaborate DE T0
ASGC
Copenhagen
TRIUMF
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1 T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1
MAN LAN T1
London NY NL
AC-1 UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

15. Big fibre cut Basel-Geneva
Connect. Communicate. Collaborate
Big fibre cut Basel-Geneva ? ? ?

16. Big fibre cut Lyon-Geneva
Connect. Communicate. Collaborate
Big fibre cut Lyon-Geneva ? ?

17. Big fibre cut Zurich-Basel
Connect. Communicate. Collaborate
Big fibre cut Zurich-Basel ?

18. Still need to…

19. Add RAL backup via Frankfurt
Connect. Communicate. Collaborate
Add RAL backup via Frankfurt
ASGC
Copenhagen
TRIUMF
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1 T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1
MAN LAN T1
London NY NL
AC-1 UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart T1
FNAL
Atlantic
Ocean
Zurich
Basel
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

20. Add CNAF backup via Vienna
Connect. Communicate. Collaborate
Add CNAF backup via Vienna
ASGC
Copenhagen
TRIUMF T1
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1
MAN LAN T1
London NY NL
AC-1 UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart
Prague T1
FNAL
Atlantic
Ocean +1
Zurich
Basel
Vienna
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

21. And add PIC backup via Paris (or CBF arrangement)
Can use GÉANT+ (for 1 or 2 Gbit/s circuit) temporarily
Lease lambda Madrid-Paris
Extend lambda via London-Amsterdam-FFM to CERN
Or maybe use CBF for Paris-Strasbourg/Kehl-GRIDKA?
OR can extend lambda down to Geneva via Lyon
Via GÉANT2 fibre or RENATER fibre
OR can use LCAS semi-protected 10GE service…
CBF prospect (via Perpignan) has been mooted in the past but prospect not currently looking good…

22. Add PIC backup (GÉANT+)
Connect. Communicate. Collaborate
Add PIC backup (GÉANT+)
ASGC
Copenhagen
TRIUMF T1
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1
MAN LAN T1
London NY NL
AC-1 UK
Frankfurt
AC-2/Yellow DE
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart
Prague T1
FNAL
Atlantic
Ocean +1
Zurich
Basel
Vienna
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

23. Add PIC backup (full lambda)
Connect. Communicate. Collaborate
Add PIC backup (full lambda)
ASGC
Copenhagen
TRIUMF T1
Via SMW-3 or 4 (?) DK T1
NDGF
??? T1
???
BNL T1
RAL
SARA
Hamburg
SURFnet T1
MAN LAN T1
London NY NL
AC-1 UK
Frankfurt
AC-2/Yellow DE
Then where?
Various options
exist
VSNL N CH
VSNL S
Paris T1
Starlight
GRIDKA FR
Strasbourg/Kehl
Stuttgart
Prague T1
Needs to be leased
FNAL
Atlantic
Ocean +1
Zurich
Basel
Vienna
Lyon T0
Madrid
Barcelona T1 ES
GENEVA
Milan IT
IN2P3 T1 T1
PIC
CNAF

24. LCAS semi-protected 10GE (is technically possible)
Connect. Communicate. Collaborate
LCAS semi-protected 10GE (is technically possible)
Caveats:
OWD is the longest of the two paths (~13ms for Amsterdam-Geneva)
not quite full 10GE LAN PHY rate (~9.6Gbps)
capacity reduced by 50% during single outage
10-20s of full outage following failure event (further testing under way)
“protection” is only across GÉANT2
only possible for Ethernet LAN-PHY services
(more details at end)
10GE LAN
MCC 5G
(32 x VC-4) 5G
(32 x VC-4)
MCC
10GE LAN

25. Updated “what-if” scenarios…

26. Big fibre cut Basel-Geneva
Connect. Communicate. Collaborate
Big fibre cut Basel-Geneva ? ? ?

27. Big fibre cut Lyon-Geneva
Connect. Communicate. Collaborate
Big fibre cut Lyon-Geneva ? ?

28. Big fibre cut Zurich-Basel
Connect. Communicate. Collaborate
Big fibre cut Zurich-Basel ?

29. Big fibre cut Strasbourg area
Connect. Communicate. Collaborate
Big fibre cut Strasbourg area ?

30. Latest Conclusions
SARA-CERN lambda has been rerouted (or soon will be)
4th diverse USLHCNET lambda will be added
Can do “LCAS protected 10GE” for ASGC
RAL & PIC still need backups!!!
LCAS “semi-protected” 10GE is technically possible but with caveats
CNAF needs a 3rd route into CERN
Long route around “eastern ring” OR
New CBF solution(s)…
Further investigations required in particular concerning:
Physical routing of GRIDKA-IN2P3 in Paris area
Leased lambdas passing through UK

31. Questions?

32. Additional slides

33. LCAS testing (10GE)

34. LCAS Testbed Measurements
LCAS evaluation is performed on the DANTE testbed
Iperf servers are connected directly to MCC 10GE ports
Alcatel’s ‘diverse routing’ option alternates VC-4 trails between two diverse paths
VCG supports up to 32ms of path length difference –note that the net path latency is equal to the latency of the LONGEST path!

35. LCAS Testbed Measurements
First diverse path:
London – Amsterdam – Frankfurt – London has a one way delay of 15.5ms

36. LCAS Testbed Measurements
Second diverse path
London – Amsterdam – Prague – Paris - London has a one way delay 27.7 ms
Path latency difference is 12.2 ms

37. LCAS Testbed Measurements
Throughput measurements are performed using Iperf on two servers with 10GE NICs in London
TCP and UDP measurements
3 types of outages:
loss of optical signal on one leg
Remove cross-connects
Adjust bandwidth using LCAS

38. TCP throughput: Fibre break
Outage time
TCP slow recovery
LCAS recovery