LHCOPN lambda and fibre routing Episode 4 (the path to resilience…) Michael Enrico Network Engineering & Planning Manager DANTE LHC Meeting, Geneva, 06-07.11.2007

?

T0-T1 Lambda routing (schematic, previously) Connect. Communicate. Collaborate T0-T1 Lambda routing (schematic, previously) ASGC Copenhagen TRIUMF Via SMW-3 or 4 (?) T1 DK T1 NDGF ??? T1 T0-T1s: CERN-RAL CERN-PIC CERN-IN2P3 CERN-CNAF CERN-GRIDKA CERN-NDGF CERN-SARA CERN-TRIUMF CERN-ASGC USLHCNET NY (AC-2) USLHCNET NY (VSNL N) USLHCNET Chicago (VSNL S) ??? BNL T1 RAL Hamburg SURFnet SARA T1 T1 MAN LAN London NY Amsterdam NL UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

Add T1-T1 Lambda routing (schematic, previously) Connect. Communicate. Collaborate Add T1-T1 Lambda routing (schematic, previously) T0-T1 Lambda routing (schematic, previously) ASGC Copenhagen TRIUMF T1 Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 T0-T1s: CERN-RAL CERN-PIC CERN-IN2P3 CERN-CNAF CERN-GRIDKA CERN-NDGF CERN-SARA CERN-TRIUMF CERN-ASGC USLHCNET NY (AC-2) USLHCNET NY (VSNL N) USLHCNET Chicago (VSNL S) ??? T1-T1s: BNL T1 RAL SARA Hamburg GRIDKA-CNAF SURFnet T1 GRIDKA-IN2P3 T1 GRIDKA-SARA MAN LAN London NY Amsterdam NL SARA-NDGF UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

SRLG analysis Connect. Communicate. Collaborate DE T0 KEY Atlantic ASGC Copenhagen TRIUMF Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 T1 ??? KEY GEANT2 NREN USLHCNET Via SURFnet T1-T1 (CBF) BNL T1 RAL SARA Hamburg SURFnet T1 T1 MAN LAN London NY NL UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

Failure scenarios (2 of them)

e.g. Big fibre cut Basel-Geneva Connect. Communicate. Collaborate e.g. Big fibre cut Basel-Geneva ASGC Copenhagen TRIUMF T1 Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 T1 MAN LAN London NY NL UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

e.g. Big fibre cut Basel-Geneva Connect. Communicate. Collaborate e.g. Big fibre cut Basel-Geneva ?

e.g. GÉANT2 Fra-Gen EDFA failure Connect. Communicate. Collaborate e.g. GÉANT2 Fra-Gen EDFA failure ASGC Copenhagen TRIUMF T1 Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 T1 MAN LAN London NY NL UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

e.g. GÉANT2 Fra-Gen EDFA failure Connect. Communicate. Collaborate e.g. GÉANT2 Fra-Gen EDFA failure ?

(Reminder of) Conclusions (from last time) RAL & PIC need backups!!! CNAF needs a 3rd route into CERN SARA-NDGF should be routed on northern leg of NORDUnet’s “Southern Cross” (for Cop-Ham part) via Padborg, Kolding, Odense Reroute 2 Amsterdam-CERN lambdas via Lyon Reroute USLHCNET (FNAL-CERN) via Lyon Further investigations required in particular concerning: Physical routing of GRIDKA-IN2P3 in Paris area Leased lambdas passing through London

Mitigations (on the way)…

Route SARA-CERN via Lyon Connect. Communicate. Collaborate Route SARA-CERN via Lyon ASGC Copenhagen TRIUMF ??? T1 Via SMW-3 or 4 (?) DK T1 NDGF T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 T1 MAN LAN London NY NL UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

Add USLHCNET lambda #4 Connect. Communicate. Collaborate DE T0 ASGC Copenhagen TRIUMF Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 MAN LAN T1 London NY NL AC-1 UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

Big fibre cut Basel-Geneva Connect. Communicate. Collaborate Big fibre cut Basel-Geneva ? ? ?

Big fibre cut Lyon-Geneva Connect. Communicate. Collaborate Big fibre cut Lyon-Geneva ? ?

Big fibre cut Zurich-Basel Connect. Communicate. Collaborate Big fibre cut Zurich-Basel ?

Still need to…

Add RAL backup via Frankfurt Connect. Communicate. Collaborate Add RAL backup via Frankfurt ASGC Copenhagen TRIUMF Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 MAN LAN T1 London NY NL AC-1 UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart T1 FNAL Atlantic Ocean Zurich Basel Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

Add CNAF backup via Vienna Connect. Communicate. Collaborate Add CNAF backup via Vienna ASGC Copenhagen TRIUMF T1 Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 MAN LAN T1 London NY NL AC-1 UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart Prague T1 FNAL Atlantic Ocean +1 Zurich Basel Vienna Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

And add PIC backup via Paris (or CBF arrangement) Can use GÉANT+ (for 1 or 2 Gbit/s circuit) temporarily Lease lambda Madrid-Paris Extend lambda via London-Amsterdam-FFM to CERN Or maybe use CBF for Paris-Strasbourg/Kehl-GRIDKA? OR can extend lambda down to Geneva via Lyon Via GÉANT2 fibre or RENATER fibre OR can use LCAS semi-protected 10GE service… CBF prospect (via Perpignan) has been mooted in the past but prospect not currently looking good…

Add PIC backup (GÉANT+) Connect. Communicate. Collaborate Add PIC backup (GÉANT+) ASGC Copenhagen TRIUMF T1 Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 MAN LAN T1 London NY NL AC-1 UK Frankfurt AC-2/Yellow DE VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart Prague T1 FNAL Atlantic Ocean +1 Zurich Basel Vienna Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

Add PIC backup (full lambda) Connect. Communicate. Collaborate Add PIC backup (full lambda) ASGC Copenhagen TRIUMF T1 Via SMW-3 or 4 (?) DK T1 NDGF ??? T1 ??? BNL T1 RAL SARA Hamburg SURFnet T1 MAN LAN T1 London NY NL AC-1 UK Frankfurt AC-2/Yellow DE Then where? Various options exist VSNL N CH VSNL S Paris T1 Starlight GRIDKA FR Strasbourg/Kehl Stuttgart Prague T1 Needs to be leased FNAL Atlantic Ocean +1 Zurich Basel Vienna Lyon T0 Madrid Barcelona T1 ES GENEVA Milan IT IN2P3 T1 T1 PIC CNAF

LCAS semi-protected 10GE (is technically possible) Connect. Communicate. Collaborate LCAS semi-protected 10GE (is technically possible) Caveats: OWD is the longest of the two paths (~13ms for Amsterdam-Geneva) not quite full 10GE LAN PHY rate (~9.6Gbps) capacity reduced by 50% during single outage 10-20s of full outage following failure event (further testing under way) “protection” is only across GÉANT2 only possible for Ethernet LAN-PHY services (more details at end) 10GE LAN MCC 5G (32 x VC-4) 5G (32 x VC-4) MCC 10GE LAN

Updated “what-if” scenarios…

Big fibre cut Basel-Geneva Connect. Communicate. Collaborate Big fibre cut Basel-Geneva ? ? ?

Big fibre cut Lyon-Geneva Connect. Communicate. Collaborate Big fibre cut Lyon-Geneva ? ?

Big fibre cut Zurich-Basel Connect. Communicate. Collaborate Big fibre cut Zurich-Basel ?

Big fibre cut Strasbourg area Connect. Communicate. Collaborate Big fibre cut Strasbourg area ?

Latest Conclusions SARA-CERN lambda has been rerouted (or soon will be) 4th diverse USLHCNET lambda will be added Can do “LCAS protected 10GE” for ASGC RAL & PIC still need backups!!! LCAS “semi-protected” 10GE is technically possible but with caveats CNAF needs a 3rd route into CERN Long route around “eastern ring” OR New CBF solution(s)… Further investigations required in particular concerning: Physical routing of GRIDKA-IN2P3 in Paris area Leased lambdas passing through UK

Questions?

Additional slides

LCAS testing (10GE)

LCAS Testbed Measurements LCAS evaluation is performed on the DANTE testbed Iperf servers are connected directly to MCC 10GE ports Alcatel’s ‘diverse routing’ option alternates VC-4 trails between two diverse paths VCG supports up to 32ms of path length difference –note that the net path latency is equal to the latency of the LONGEST path!

LCAS Testbed Measurements First diverse path: London – Amsterdam – Frankfurt – London has a one way delay of 15.5ms

LCAS Testbed Measurements Second diverse path London – Amsterdam – Prague – Paris - London has a one way delay 27.7 ms Path latency difference is 12.2 ms

LCAS Testbed Measurements Throughput measurements are performed using Iperf on two servers with 10GE NICs in London TCP and UDP measurements 3 types of outages: loss of optical signal on one leg Remove cross-connects Adjust bandwidth using LCAS

TCP throughput: Fibre break Outage time TCP slow recovery LCAS recovery