Download presentation
Presentation is loading. Please wait.
Published byMarcus Merritt Modified over 8 years ago
1
KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association STEINBUCH CENTRE FOR COMPUTING - SCC www.kit.edu 100 GE in the wild – first experiences Bruno Hoeft (KIT/SCC) Co-authors: Aurelie Reymund (KIT / SCC) Robert Stoy (DFN) Frank Schröder (DFN) Ralf Niederberger (FZJ) Olaf Mextorf (FZJ) Sabine Werner (FZJ)
2
2 Agenda Introduction Motivation for setting up the 100GE testbed Testbed layout 100GE test phase Experiences Bruno Hoeft -- SCC/NET 19.10.2010
3
3 Agenda Introduction Motivation for setting up the 100GE testbed Testbed layout 100GE test phase Experiences Bruno Hoeft -- SCC/NET 19.10.2010
4
4 Introduction Forschungszentrum Jülich Karlsruhe Institute of Technology Both institutions are: members of the Helmholtz Association connected via the national research network Germany (DFN) Distance approx. 250 km (distance by air) Bruno Hoeft -- SCC/NET 19.10.2010 Karlsruhe Institute of Technology Forschungszentrum Jülich Berlin
5
5 Agenda Bruno Hoeft -- SCC/NET 19.10.2010 Introduction Motivation for setting up the 100GE testbed Testbed layout 100GE test phase Experiences
6
6 Motivation for setting up the 100GE testbed (1) Fiber infrastructure GasLINE contributed Fiber (447 km) Prove 100 G using standard fiber infrastructure Evaluate interference of different wavelengths in one fiber (“light modulation”/”cross talking“) Bruno Hoeft -- SCC/NET 19.10.2010 IP equipment Cisco CRS-3 at a first field test Interoperability of the Equipment Optical equipment Huawei: DWDM (Dense Wavelength Division Multiplexing ) transmission system capable of 100 G deployment over wide area distance
7
7 Motivation for setting up the 100GE testbed (2) Overall goal: Bring the whole system into operation Fill the 100G Pipe with IP traffic Stress test Stability validation Obtain experience, gather information from unexpected events Bruno Hoeft -- SCC/NET 19.10.2010
8
8 Agenda Introduction Motivation for setting up the 100GE testbed Testbed layout 100GE test phase Experiences Bruno Hoeft -- SCC/NET 19.10.2010
9
9 GasLINE Fiber Bruno Hoeft -- SCC/NET 19.10.2010 Forschungszentrum Jülich / research centre Jülich Karlsruhe Institute of Technology distance by air 250 km Fiber length 447 km 7 fiber segments 6 amplifier (boosting and correcting the signal)
10
10 Deployment of Optical Transmission system Successful fiber calibration All 7 segments well prepared splice / calibration measurement system of Exfo Model: FTB 85100G, 100GE Ethernet test module (client site) 30h longterm 100GE tests no errors at "2010/07/12 11:02:42","History Reset Statistics","","" = start measurement at "2010/07/13 17:43:13","Test Stopped","","" No error during the whole measurement period The quality of the fiber connection is outstanding. Bruno Hoeft -- SCC/NET 19.10.2010
11
11 Testbed layout (1) Bruno Hoeft -- SCC/NET 19.10.2010 Huawei 100GE DWDM Huawei DWDM Transponder : 100Gbit/s Wavelength Conversion Board Order-#: sDQPSK with 2 Wavelength each 56Gbps Cfp of Finisar (LR4 (4x25G)) Cfp 100 Gigabit Small Form Factor Pluggable
12
12 Testbed layout (2) Bruno Hoeft -- SCC/NET 19.10.2010 Huawei 100GE DWDM Huawei DWDM Transponder : 100Gbit/s Wavelength Conversion Board Order-#: sDQPSK with 2 Wavelength each 56Gbps Cfp of Finisar (LR4 (4x25G)) Cisco CRS-3 4 slot system 4 Fabric Cards (140Gbps) Modular Services Cards (MSC) (140Gbps) 1 * 100GE PLIM ( Physical Layer Interface Module ) CFP, Vendor OPNEXT (100GBASE-LR4) 3 * 4*10GE PLIM
13
13 Testbed at KIT (1) Te0/1 Eth2 Eth-Ipmi Eth1 xr-fzk1 Gi9/31 Gi9/30 Gi9/39 Gi9/40 Gi9/41 Gi9/42 X-WiN ms-kit1 10GE Measuring Station (UDP,TCP) hades-kit2 Hades Station for OWD measurements ti-kit1Traffic Injector trc-kit1Cisco Router CRS-3 tsc-kit1, tsc-kit2Cisco Switch as mediaconverter SM-MM xr-fzk1X-WiN Router Eth0 Eth-Eric Eth0 Eth-Eric Te2 (v) DFN-PF-175:12 (h) (v) DFN-PF-174:9 (h) k-000-101-452 k-000-102-051 k-001-000-951 ms-kit1 hades-kit2 ti-kit1 RP2 EthRP1 Eth Te0/3/0/2 Te0/3/0/3 Cisco CRS-3 Router trc-kit1 He0/0/0/0 Te0/3/0/1 Gi9/29 Gi9/25 Te1/4 Te1/2 Gi9/27 Gi9/28 Gi9/26 k-000-104-518k-000-011-014 Bruno Hoeft -- SCC/NET 19.10.2010 - Admin connection -10 Gbps Single Mode -Copper Cat 5e data injection DWDM
14
14 Testbed layout (3) Compute Nodes at FZJ Type 1 (three nodes [J1 to J3]): 2 * Dual Core (AMD Opteron (P 265)), 2GB RAM Model: "Myricom Myri-10G Dual-Protocol NIC (10G-PCIE-8A)" Type 2 (one node [J4]): 2 * Dual Core (Intel® Xeon™ CPU 3.00GHz, 2 GB RAM Model: "Myricom Myri-10G Dual-Protocol NIC (10G-PCIE-8A)" Type 3 (five nodes [J5 to J9]): Dual QuadCore (Intel® Core™ i7 CPU 860 @2.80GHz 3 GB RAM Model: "Myricom Myri-10G Dual-Protocol NIC (10G-PCIE-8A)" Bruno Hoeft -- SCC/NET 19.10.2010 J– 09 … J– 01 Huawei 100GE DWDM
15
15 Testbed layout (4) Compute Nodes at KIT 9 nodes [K1 to K9]: Dual QuadCore (Intel® Xeon® CPU X5560 @2.80GHz 32 GB Memory Model: "Broadcom Corporation NetXtreme II BCM57711 10- Gigabit PCIe" Bruno Hoeft -- SCC/NET 19.10.2010 K– 01 … K– 09 J– 09 … J– 01 Huawei 100GE DWDM
16
16 Testbed at KIT (2) Te0/1 Eth2 Eth-Ipmi Eth1 xr-fzk1 Gi9/31 Gi9/30 Gi9/39 Gi9/40 Gi9/41 Gi9/42 X-WiN ms-kit1 10GE Measuring Station (UDP,TCP) hades-kit2 Hades Station for OWD measurements ti-kit1Traffic Injector K– 01 to 10Compute Node (10 GE NIC) trc-kit1Cisco Router CRS-3 tsc-kit1, tsc-kit2Cisco Switch as mediaconverter SM-MM xr-fzk1X-WiN Router Eth0 Eth-Eric Eth0 Eth-Eric Te2 (v) DFN-PF-175:12 (h) (v) DFN-PF-174:9 (h) k-000-101-452 k-000-102-051 k-001-000-951 DFN-PF-175 123456789 123456789 10 DWDM ms-kit1 hades-kit2 ti-kit1 RP2 EthRP1 Eth Te 0/1/0/ 1 Te 0/1/0/ 2 Te 0/1/0/ 3 Te 0/2/0/ 0 Te 0/2/0/ 1 Te 0/2/0/ 2 Te 0/2/0/ 3 Te 0/3/0/ 0 Te0/3/0/2 Te0/3/0/3 He0/0/0/0 Media Converter 1 Fe0 Te 2/ 1 Te 2/ 3 Te 1/ 3 Te 1/ 1 Te 1/ 5 Te 2/ 2 Te 2/ 4 Te 1/ 4 Te 1/ 2 Te 1/ 6 Te0/3/0/1 Te 0/1/0/ 0 Te 0/3/0/ 1 Gi9/29 Gi9/25 Te1/4 Te1/2 Gi9/27 Gi9/28 Gi9/26 Te 2/ 2 Te 2/ 4 Te 1/ 4 Te 1/ 2 Te 1/ 6 Te 2/ 1 Te 2/ 3 Te 1/ 3 Te 1/ 1 Te 1/ 5 k-000-104-518k-000-011-014 Bruno Hoeft -- SCC/NET 19.10.2010 K– 01 K– 02 K– 03K– 05K– 07K– 09 K– 04K– 06K– 08 - Admin connection -10 Gbps Multi Mode -10 Gbps Single Mode -Copper Cat 5e data injection Cisco CRS-3 Router trc-kit1 Media Converter 2
17
17 100 G link up? Challenges in setting up the testbed: Light between the two DWDM systems at Jülich and Karlsruhe Huawei developer saw light at both ends after several hours of work Installation and “booting up” of the Cisco CRS-3 Router Setting-up 100GE Interface between Cisco Routers and Huawei Transponders - preproduction CFPs - optical interfaces of one vendor - new releases will solve the vendor incompatability Bruno Hoeft -- SCC/NET 19.10.2010 07/22/2010 Time
18
18 Challenging – Interface errors Input errors increased at 100GE „KIT“ interface Standard ping (MTU 1400 Byte) 2% packet loss Error investigation result: bad input signal at Cisco CRS-3 resulting in „link flap“ After the link was stable (initially) Parameter adjustment caused “link flap” (link instable over weekend) Further adjustment at transmission system stabilised link (2 days of work) Link was stable for the duration of the testbed (3 weeks) Input error at KIT resolved and a small input error ratio at „FZJ“100GE (less than 10 -7 ) cause: old preproduction CFP at KIT First tests of testplan were executed: „Ramping up UDP load on `routing loop´” Bruno Hoeft -- SCC/NET 19.10.2010 07/23/201007/27/2010 Time
19
19 Agenda Introduction Motivation for setting up the 100GE testbed Testbed layout 100GE test phase Experiences Bruno Hoeft -- SCC/NET 19.10.2010
20
20 Testplan (1) UDP Load on Routing Loop between routers, Test 1.x end systems, administration: KIT, FZJ, 10GE Interface: 10GBase SR end systems, administration: DFN, 10GE Interface: 10GBase LR 1.1 Single UDP/IP flow on routing loop, traffic in both directions 1.2. Multiple UDP/IP flows on routing loop, traffic in both directions trc-kit1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/1 Te0/3/0/3 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 tsc-kit1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-kit2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 trc-fzj1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 Hu0/0/0/0 KarlsruheJülich distance ca. 400km K1 K2 K3 K4 K5 K6 K7 K8 K9 J1 J2 J3 J4 J5 J6 J7 J8 J9 ms-kit1 Te0/3/0/2 hades- kit2 Te0/3/0/2 Te0/3/0/3 Te0/3/0/1 ms-fzj1 hades- fzj2 ti-kit1 x 127 tsc-fzj1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-fzj2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 07/22/201008/06/2010 Time
21
21 1. UDP Load on Routing Loop Bruno Hoeft -- SCC/NET 19.10.2010 1.1 Single UDP/IP flow on routing loop, traffic in both directions 1.1.1 Ramp up of data rate from 0 to 90 Gbit/s, increase 1Gbit/s per step 100 GE interface statistic FZJ: substantial error rate: Vendor investigation in error rate traffic injector: 07/22/2010 Time
22
22 1. UDP Load on Routing Loop 1.2. Multiple UDP/IP flows on routing loop, traffic in both directions 1.2.1 Two parallel UDP/IP flows on routing loop Bruno Hoeft -- SCC/NET 19.10.2010 07/26/2010 Time
23
23 1. UDP Load on Routing Loop 1.2. Multiple UDP/IP flows on routing loop, traffic in both directions 1.2.2 Ten parallel UDP/IP flows on routing loop Bruno Hoeft -- SCC/NET 19.10.2010 traffic injector: moderate error ratio < 10 -7 08/06/2010 Time
24
24 Testplan (2) TCP iperf Tests between connected end systems, Test 2.x end systems, administration: KIT, FZJ, 10GE Interface: 10GBase SR end systems, administration: DFN, 10GE Interface: 10GBase LR trc-kit1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/1 Te0/3/0/3 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 tsc-kit1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-kit2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-fzj1 4900M Te tsc-fzj2 4900M Te trc-fzj1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 Hu0/0/0/0 KarlsruheJülich distance ca. 400km K1 K2 K3 K4 K5 K6 K7 K8 K9 J1 J2 J3 J4 J5 J6 J7 J8 J9 ms-kit1 Te0/3/0/2 hades- kit2 Te0/3/0/2 Te0/3/0/3 Te0/3/0/1 ms-fzj1 hades- fzj2 ti-kit1 2.1. Single TCP/IP flow between connected iperf stations 2.2. Multiple TCP/IP flows 08/09/2010 Time
25
25 2. TCP iperf Tests 10GE Endnodes 2.2 Multiple TCP/IP flows between Multiple 10GE 2.2.2 simplex from KIT to FZJ, successive activation of TCP flows Bruno Hoeft -- SCC/NET 19.10.2010 08/09/2010 Time
26
26 Testplan (3) TCP iperf Tests between end systems and parallel UDP background traffic load on routing loop, Test 3.x end systems, administration: KIT, FZJ, 10GE Interface: 10GBase SR end systems, administration: DFN, 10GE Interface: 10GBase LR trc-kit1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/1 Te0/3/0/3 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 tsc-kit1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-kit2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-fzj1 4900M Te tsc-fzj2 4900M Te trc-fzj1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 Hu0/0/0/0 KarlsruheJülich distance ca. 400km K1 K2 K3 K4 K5 K6 K7 K8 K9 J1 J2 J3 J4 J5 J6 J7 J8 J9 ms-kit1 Te0/3/0/2 hades- kit2 Te0/3/0/2 Te0/3/0/3 Te0/3/0/1 ms-fzj1 hades- fzj2 ti-kit1 3.1 Constant UDP load on Routing Loop and ramping up of TCP data between end systems 3.2 Nine TCP streams between end systems and ramping up and down of UDP background load x 127 08/09/2010 Time
27
27 9 TCP Streams, + UDP Background Load on Routing Loop, + UDP Loss Measurement (Test 3.2.1) 08/10/2010 Time
28
28 9 TCP Streams, + UDP Background Load on Routing Loop, + UDP Loss Measurement (Test 3.2.1) 08/10/2010 Time
29
29 Testplan (4) max. TCP streams, end systems KIT -> FZJ, incl. DFN Systems, Test 3.x end systems, administration: KIT, FZJ, 10GE Interface: 10GBase SR end systems, administration: DFN, 10GE Interface: 10GBase LR trc-kit1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/1 Te0/3/0/3 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 tsc-kit1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-kit2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-fzj1 4900M Te tsc-fzj2 4900M Te trc-fzj1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/2 Te0/3/0/3 Te0/3/0/1 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 Te0/3/0/2 Hu0/0/0/0 KarlsruheJülich distance ca. 400km K1 K2 K3 K4 K5 K6 K7 K8 K9 J1 J2 J3 J4 J5 J6 J7 J8 J9 dfn-kit1 dfn-kit2 dfn-fzj1 dfn-fzj2 08/10/2010 4.2.1 TCP Streams KIT FZJ 4.2.2 TCP Streams FZJ KIT Time
30
30 TCP Throughput FZJ -> KIT, 11 Streams (Test 4.2.2-0) 08/13/2010 Time
31
31 Agenda Introduction Motivation for setting up the 100GE testbed Testbed layout 100GE test phase Experiences Bruno Hoeft -- SCC/NET 19.10.2010
32
32 Experience obtained from the tests Bruno Hoeft -- SCC/NET 19.10.2010 GasLINE fiber infrastructure is ready for 100GE Huawei Transmission system was very stable and error free Early field tests equipment needed adjustment (no plug an play) Cisco and Huawei engineers executed debug commands Single vendor cfp combination was required (pre production cfp have been used) CRC error ratio (below 10 -7 ) only one direction (kit fzj) Caused by one pre production cfp at KIT router error rate did not reduce the TCP throughput 100GE was tested successfully
33
33 Acknowledgements Thanks to the Project Partners: Bruno Hoeft -- SCC/NET 19.10.2010
34
KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association STEINBUCH CENTRE FOR COMPUTING - SCC www.kit.edu 100 GE in the wild – first experiences Bruno Hoeft
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.