1. Provision of networks for the LHC experiments in Germany - Status 03/2006 - K. Schauerhammer, K. Ullmann, (DFN) 2. / 3.3.2006 Meeting GridKa TAB Karlsruhe

2. Seite 2 Contents I. Technical options X-WiN and Geant2 II. Architecture LCG - Network (T0 - T1) III. Performance Measurements DFNInternet IV. Connecting T2 centres in Germany

3. Seite 3 I. Technical options X-WiN and Geant2

4. Seite 4 X-WiN - status as of 1. 1. 2006 Technical concept backbone (after Europe wide tender) –backbone consists of (dark) fibre and leased circuits –operational responsibility: DFN –more (than before) bought in (partial) services –much more (than before) DFN PoPs Access-Network very similar as in the past Economic concept –backbone put together from different service offerings from the market and integrate that under DFN responsibility –most effective usage of competition on the market

5. Seite 5 20.02.2006 X-WiN (optical platform) GAR ERL BAY MUE FZJ AAC BIR DES HAM POT TUB FZK GSI DUI BRE HAN BRA MAG BIE FRA HEI STU REG DRE CHE ZIB ILM KIE ROS LEI JEN WEI ESF HUB ADH KAI GOE KAS MAR GIE Dark fibre Leased circuits GRE 80km to Frankfurt/O (Pionier/PL) 120km to Zürich (Switch/CH, GARR/IT) 50km to Kehl/Straßburg (Renater/FR) 50km to Enschede (Surfnet/NL) 43 DFN PoPs

6. Seite 6 Optical platform based services Toolbox for the provision of –DFNInternet (2Mbps to 10 GE) –VPN-Services based on optical links –services like DFNVC, DFNPKI, DFN-CERT, DFNRoaming, DFNNews etc. unchanged New cost structures for optical networks enable economic solutions for specialised services (Optical Private Networks / OPNs) i.e. Grids

7. Seite 7 General targets X-WiN design More performance performance increase by a factor of 4 since 01/06 more performance available (for the same price) as of 01/07 More flexibility no volume charging (and limitation) Ethernet as additional access technology Hybrid PoPs enable VPNs More availibility during design for backbone implicitly taken into account

8. Seite 8 Availibility backbone by means of... GAR ERL BAY MUE FZJ AAC BIR DES HAM POT TUB FZK GSI DUI BRE HAN BRA MAG BIE FRA HEI STU REG DRE CHE ZIB ILM KIE ROS LEI JEN ESF HUB ADH strongly protected physical topology –protected fibre paths –meshed fibre ring structure –more mathematical optimisation will show, how to meaningful complement physical topology

9. Seite 9 Availibility backbone by means of... Fault tolerant logical topology, especially for DFNInternet Well-known optimisation for –logical links between PoPs –best mapping to physical topology Optimisation targets, especially –minimal delay –minimal loss of connectivity in case of severe failures –best (smooth) utilisation in case of failures

10. Seite 10 X-WiN DFNInternet as of 01/06 Logical IP backbone

11. Seite 11 worldwide Internet Availibility backbone by means of... User X-WiN At least two accesses to global IP networks, more redundant carriers Full throughput in case of failures Additional roughly 100 direct peerings Redundant global connectivity by means of independant paths and PoPs assures protection against problems with Carriers (e.g. network failures or insolvency) Redundant global connectivity

12. Seite 12 Dark Fibre Technical definition –ITU-T conform and "WDM-applicable" –some minimal requirements on technical parameters for example attenuation Service: –Support : 24/7 hotline and debug –provision of colocation space –End-to-end operational responsibility –usual time for reaction –integration into DFN defined processes

13. Seite 13 Equipment to light Dark Fibre Technical definition: –equip with (several optional) digital interfaces –provision of a management system Service: –includes installation and maintenance –support: 24/7 hotline and debug –usual time for reaction –integration into DFN defined processes

14. Seite 14 Operations Definition –Monitoring of management data –pursue well defined procedures failure identification (devices? - links?) alarm of responsible support groups supervision of debug if necessary escalation –all procedures DFN defined –regular (i.e. monthly) reporting –open for integration of new devices

15. Seite 15 PoP A PoP BPoP C Operational model X-WiN platform dark fiber Leased circuit Access to management data of DFN owned devices Operations Group alarms and triggers support (links, devices)

16. Seite 16 GÉANT2 footprint LU RU EE LV LT DK IT FR BE CH SI AT HR PL DE CZ PT ES IE NL UK RO BG TR CY IL MT GR HU SK 2 2 3 3 3 3 2 2 2 2 2 3 3 2 2 2 2 2 2 1 1 1 1 1 1

17. Seite 17 GÉANT2 Hybrid POP Dark Fibre N x 10G 10G N x 10G GÉANT2 POP to NREN Leased Service

18. Seite 18 Gigabit Ethernet p2p Service GÉANT Border GÉANT Border Interconnect: N x physical GEth GÉANT2 Interconnect: N x 10GEth VLAN/10GEth (LAN or WAN) Physical GEth GEth (NREN transport) GEth (GÉANT2 transport) NREN A NREN B

19. Seite 19 II. Architecture LCG - Network (T0 - T1)

20. Seite 20 Architecture LCG network (1) Assumptions, design criteria: –bandwidths 10Gbit/s per T1 resp. multi 10Gbit/s for T0 –continuous stream of data –keep it simple (as possible) –organise transport if possible in layer 2 –security already considered during design system should be protected from unauthorised access access from trusted sources not limited

21. Seite 21 Architecture LCG Network (2) Optical Private Network, consists of dedicated 10G links from T0 and any T1, T1 access in two options: –Light path T1 –Routed T1 Back-up for T0-T1 is part of the design T0 interface very likely 10GE LAN-PHY

22. Seite 22 Scalability of the architecture Options for transfer: –10 Gbit/s ~ 10 14 byte/d or 100 Tbyte/d –eleven 10G links -> more than 1 Pbyte/d or roughly 0.5 Exabyte/a If one 10G link per T1 is not sufficient: –Installation of just another 10G link T0 - T1 if possible on physically separated fibre path –architecture covers this completely

23. Seite 23 LCG T0 - T1 OPN

24. Seite 24 III. Performance Measurements DFNInternet

25. Seite 25 Issues in setting up a 10Gbit connetion between GridKa and CERN openlab Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany http://www.gridka.de Bruno Hoeft Taken from: Measurements DFNInternet

26. Seite 26 CERN GridFTP server GridFTP server WAN 2003/4 -- Gigabit GridKa – CERN Géant 10 Gbps DFN 2.4 Gbps GridFTP tested over 1 Gbps Karlsruhe Frankfurt 2x 1 Gbps 98% of 1Gbit Geneva 1000 0

27. Seite 27 10Gigabit WAN SC GridKa – CERN CERN GridFTP server GridFTP server Géant 10 Gbps DFN 10 Gbps Karlsruhe Frankfurt - Bandwidth evaluation (tcp/udp) - MPLS via France (MPLS - MultiProtocol Label Switching) - LBE (Least Best Effort) - GridFTP server pool HD to HD Storage to Storage - SRM Geneva

28. Seite 28 Evaluation of max. throughput 700 1400 5600 4900 4200 3500 2800 2100 7000 6300 Mbit/s 18:00 20:00 9 nodes each site -8 * 845 Mbit -1 * 540 Mbit higher speed at one stream is resulting in a packet loss

29. Seite 29 IV. Connecting T2 centres in Germany

30. Seite 30 LCG network services in Germany T0-T1 problem –10 Gbit/s over European VPN (GridKa - CERN) –backup via CNAF or SARA T1-T2 problem –within DE no problem as most LHC sites are close to X-WiN PoPs –for T2 in DE rough guess of traffic flows are needed (source, destination, transferred volume per day, peak requirements?)

31. Seite 31 DES HAM ZIB HAN LEI DUI BIR FRA ERL GAR STU MUE BIE ROS DOR KAS KOE AAC BON FZK TUM AUG REG BAM CHE DRE BOC WUP HEI GSI MAI FRE MAN SIE BRE X-WiN-Nodes LHC User Locations at X-WiN-Nodes LHC User locations to be connected to X-WiN nodes LCG Sites DE and X-WiN