1. DE-KIT(GridKa) -- 100G @ LHCONE
Bruno Hoeft / KIT
Andreas Petzold / KIT

2. Agenda History 100G @ (DE-)KIT NREN proposal DE-KIT proposal
100G between KIT and Jülich (2010)
SC13 – 100G from KIT to SC13 (Caltech-booth)
(DE-)KIT
Motivation
NREN proposal
DE-KIT proposal
KIT -- proposed Deployment Steps
Upgrade Sketch
At Upgrade Involved
Splitting into Parts
Market survey
(DE-)KIT deployment
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

3. 2010 – KIT – Jülich (DFN/Huawei/Cisco/Gasline)
History
2010 – KIT – Jülich (DFN/Huawei/Cisco/Gasline)
Düsseldorf
Köln
Bonn
Koblenz
Frankfurt
Wiesbaden
Mainz
Darmstadt
Mannheim
FZ Jülich
KIT Karlsruhe
Dernbach
Gernsheim
Porz
Naurod
88 km
60 km
96 km
64 km
29 km
93 km
17 km J– … J–
Huawei 100GE DWDM
hades-fzj1
ms-fzj1
max. 96,5Gbps
Fiber distance 440 km
Heidelberg K– … K–
Huawei 100GE DWDM
ms-kit1
ti-kit1
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

4. KIT @ Caltech SC13 WAN Demo
Storage to storage file transfer
File transfer via FastDataTransfer (FDT) application
@KIT
20 Fileserver different capacities
10G NIC
@Caltech booth
3 Fileserver
1 or 2 40G NIC
SSD Harddisk
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

5. … WAN – KIT SC13 @KIT: Nexus 7010 : 100G linecard (LR-4)
Echostream3
40G
sc3
sc5
Caltech
SC13 booth
DFN contributes 100G link Karlsruhe - Amsterdam
DFN PoP Karlsruhe ECI 100G to DFN PoP Frankfurt
@Frankfurt : peering to Géant
Géant PoP Frankfurt to Géant PoP Amsterdam
@Amsterdam Géant Peering via MX960 with SURFnet T4000
@KIT: Nexus 7010 : 100G linecard (LR-4) …
ANA-100G (transatlantic link)
@NewYork MANLAN (Internet2) and via Esnet to SC13 Denver
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

6. Throughput: Monalisa WAN – KIT SC13 70Gbps at monalisa
Echostream3
40G
sc3
sc5
Caltech
SC13 booth
Throughput:
Monalisa
70Gbps at monalisa
peaks up to 75Gbps
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

7. Agenda History 100G @ (DE-)KIT NREN proposal DE-KIT proposal
100G between KIT and Jülich (2010)
SC13 – 100G from KIT to SC13 (Caltech-booth)
(DE-)KIT
Motivation
NREN proposal
DE-KIT proposal
KIT -- proposed Deployment Steps
Upgrade Sketch
At Upgrade Involved
Splitting into Parts
Market survey
(DE-)KIT deployment
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

8. DE-KIT / GridKa German LHC Tier-1 status
GridKa (tier-1) Provider Edge Router
DWDM
DFN
R-ir-cn-1-1
KIT Provider Edge Router
IT-INFN-CNAF (T1)
FR-CCIN2P3 (T1)
FZU (Prague) (T2)
CERN (T0)
Poland (T2)
NL-T1 (T1)
R-ir-cn-1-2
KIT Provider Edge Router
Legend:
10G Ethernet
10G VPN End-to-End
LHCOPN: T0 – T1
Direct connected sites: T1 – T[12]
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

9. Géant initiated merging test of dedicated tier-1 connections to LHCONE
DE-KIT
Slide of Vincenzo Capone, url:
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

10. LHCOPN link was saturated even during LS1
Upgrade Motivation
LHCOPN link was saturated even during LS1
LHCone  max. usage 15Gbps DE-KIT to Tier-[123]
General Purpose Internet  max. usage 10Gbps
proposed upgrade
General purpose internet traffic 20 GE 50 GE
LHCone utilization  15Gbps 20 GE
LHCOPN utilization  10Gbps
70+20 GE
Proposed upgraded datarate after LS1
Expected 500MB/sec  approx. 5Gbps + additional experiment data  2 * 10GE LHCOPN
Increased datarate between Tier-1 – Tier-1 communication
Increased datarate between Tier-1 and Tier-[23] 1 * 10 GE
GridKa
-->
LHCOPN T0 3 * 10 GE
Gridka
-->
DE-KIT to T1 15 2 * 10 GE
GridKa
-->
LHCONE (T[123]) 15 +
Internetconnection - KIT 10 GE 40 GE
Status and Trends in Networking at LHC Tier1 Facilities (Andrey Bobyshev (FNAL) at CHEP 2012)
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

11. Current GridKa deployment
GridKa (tier-1) Provider Edge Router
DWDM
DFN
R-ir-cn-1-1
R-ir-cn-1-2
IT-INFN-CNAF (T1)
IT-INFN-CNAF (T1)
FR-CCIN2P3 (T1)
FR-CCIN2P3 (T1)
FZU (Prague) (T2)
NL-T1 (T1)
FZU (Prague) (T2)
CERN (T0)
CERN (T0)
Poland (T2)
Poland (T2)
NL-T1 (T1)
Direct link DE-KIT to CERN (2*10G) LHCOPN
Migration to-LHCONE
Direct link DE-KIT to FR-CCIN2P3 (10G) to LHCONE (100G)
Direct link DE-KIT to IT-INFN-CNAF (10G) to LHCONE (100G)
Direct link DE-KIT to Poznan (1G) to LHCONE (100G)
Direct link DE-KIT to FZU (Prague) (10G) to LHCONE (100G)
Direct link DE-KIT to NL-T1 (10G) to LHCONE (100G)
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

12. 100G @ KIT – upgrade perimeter
DFN
Areas needed to be included in the 100G upgrade:
2 border routers (one VSS) (provider edge) (2* KIT-CN)
6* 100GE interfaces
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

13. 100G @ KIT – upgrade perimeter
DFN
Belwue
Areas needed to be included in the 100G Upgrade:
2 border routers (one VSS) (provider edge) (2* KIT-CN)
6* 100GE interfaces
2 border routers (one VSS) (provider edge) (2* KIT-CS)
14* 100GE interfaces
4* 100GE transponder
firewall
22* 100GE interfaces
6* 100GE transponder
4 backbone routers (2* VSS) (KIT core router)
16* 100GE interfaces
routers of different projects (per project):
5* 100GE interfaces
DWDM
DWDM
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

14. 100G @ KIT – upgrade perimeter
DFN
Belwue
Areas needed to be included in the 100G Upgrade:
border router (provider edge) (KIT-CN)
4* 100GE interfaces
border routers (provider edge) (KIT-CS)
7* 100GE interfaces
2* 100GE transponder
firewall
14* 100GE interfaces
6* 100GE transponder
2 backbone routers (KIT core router)
6* 100GE interfaces
1* 100GE transponder
routers of different projects (per project):
2* 100GE interfaces
DWDM
DWDM
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

15. Splitting the 100G upgrade into affordable parts
2013
Project LSDF 100G link to BioQuant Heidelberg via Belwue
2014
Project GridKa 100G link to LHCONE via DFN
2015/6
KIT-CN – provider edge router
2016/7
KIT-CS – provider edge router
DWDM
2017/8
? Firewall ?
KIT core router (6* router)
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

16. 100G-WAN@KIT Result of market survey 100GE – cfp2
Brocade MLXe
100GE – cfp2
All required protocols
bgp (2 * internet routing table and still space)
ospf
pbr
IPv4 + IPv6
VLAN
... adaptation required
snmp  cacti netgraphs
sflow  currently netflow
netdoc  home grown inventory and configuration database …
First deployments steps are in progress
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

17. GridKa deployment after KIT 100G upgrade
second deployment phase (2015):
1* 100G to Frankfurt
1* 100G to Erlangen
“traffic shaping“ to 50Gpbs at each 100G link
Border router of KIT receiving 100G uplinks
Project will get WAN access through KIT provider edge router
2* 10G LHCOPN link will stay separate VPN DE-KIT -- CERN
IT-INFN-CNAF (T1)
FR-CCIN2P3 (T1)
NL-T1 (T1)
FZU (Prag) (T2)
CERN (T0)
Poland (T2)
LHCONE
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

18. Planned 100G Upgrade of DE-KIT (GridKa) (2016/17)
EDGE
MPO/MPT
r-ig-I
r-ig-II
Raum
168
198
r-grid145
r-grid3
r-grid2
r-grid6
FileServer TOR
ECMP
ComputeNode TOR
Nexus 7000
Nexus 7700
Current four Catalyst 6509 will get replaced by two Nexus7700
Nexus N2K-C2348TQ Aggregation Switches for ComputeNodes
FileServer aggregation switch Nexus 5696Q introduced
Edge router upgrade catalyst 6509 to Nexus 7700 planned for 2017
CORE …
Legend:
10Gbps
40Gbps
100Gpbs
100G / ISGC 2016,
Bruno Hoeft / Andreas Petzold (KIT)

19. 100G Deployment@(DE-KIT)
thanks for your attention ?
Questions
100G
Bruno Hoeft / KIT
Andreas Petzold / KIT