1. Ethernet over SONET Dr. John S. Graham University of London Computer Centre

2. Ethernet vs SONET FeatureEthernetSONET Guaranteed Delivery No (Packet Switched) Yes (Circuit Switched) Guaranteed BWNoYes Performance Monitoring CRC CountBER ProtectionNone (STP?)APS Fault LocationNoneAIS Access CostsLowHigh

3. SONET/SDH Model RR MUX DXC MUX Path Layer Connection Line Layer Section Layer

4. SONET Network Elements

5. SONET STS-1 Frame Section Line PTR SPE 3 5 86 columns Path 30599041 Payload Capacity = 756 Bytes = 48.384 Mb/s

6. Section Overhead Bytes Data Com D3 Data Com D2 Data Com D1 User F1 Orderwire E1 BIP-8 B1 STS-1 ID C1 Framing A2 Framing A1 Section

7. Line Overhead Bytes Line Orderwire E2 Line FEBE Z2 Sync Stat Z1 Data Com D12 Data Com D11 Data Com D10 Data Com D9 Data Com D8 Data Com D7 Data Com D6 Data Com D5 Data Com D4 APS K2 APS K1 BIP-8 B2

8. Path Overhead Bytes Path Multi-frame H4 User Channel F2 Path Status G1 Signal Label C2 BIP-8 B3 Path Trace J1

9. Accurate Bit-Level Timing Line Coding –AMI, AMI RZ –HDB3 Block Coding –4B/5B (FDDI) –5B/6B (E3) –8B/10B (Gigabit Ethernet) Scrambling –Frame Synchronous –Self Synchronous

10. 8B/10B Encoding (1/2) Ensures equal number of 1’s and 0’s for –Clock recovery –DC Balance Detection of single bit transmission errors Predefined ‘comma’ (0011111 and 1100000) characters

11. 8B/10B Encoding (2/2) Data (D) and commands (K) encoded separately Octet split into 5 MSB and 3 LSB Code groups generated from lookup tables Decoded ABCDEFGH 01000000 20 Encoded ABCDEiFGHj 1011010100 D2.0

12. Control Words (‘Ordered Sets’) SymbolEncodingPurpose FK28.5 D21.5LINK_NOT_AVAILABLE CK28.5 D10.5 config_regLINK_CONFIGURATION I1K28.5 D5.6Idle I2K28.5 D16.2Idle SK27.7SoF TK29.7EoF RK23.7CARRIER_EXTEND HK30.7ERROR

13. Scrambling in SONET/SDH SDH STM-1 uses x 7 + x 6 +1 polynomial

14. Concatenation STS-3155 Mb/s 145.152 Mb/s STS-3c/VC-4155 Mb/s 149.76 Mb/s STS-12622 Mb/s 580.608 Mb/s VC-4-4622 Mb/s 599 Mb/s STS-12c622 Mb/s 599.04 Mb/s

15. Provisioning Example A D B E C F 3 - 5 7 9 10 -12 1 ??? 1 – 12(622 Mb/s)

16. Bandwidth Fragmentation STS-3c STS-12 Blocked! Drop Interface (Node F) Ring

17. Virtual Concatenation STS-3c STS-12 STS-1-3v Drop Interface (Node F) Ring

18. The Magic of VCAT Protocol Client Rate Contiguous Concatenation Virtual Concatenation Fast Ethernet100STS-3c65%STS-1-2v98% ESCON160STS-12c26%STS-1-4v78% Fibre Channel850STS-48c34%STS-3c-6v92% Gigabit Ethernet1000STS-48c40%STS-3c-7v92% Fibre Channel 21700STS-48c66%STS-3c-12v92%

19. Differential Delay TXTX TyTy Source STS-1-12v Sink Core Network (Partial Mesh)

20. Multiframe Indicator Bit 1Bit 2Bit 3Bit 4Bit 5Bit 6Bit 7Bit 8 MFI2 MSBs (bits 1 - 4)0000 MFI2 LSBs (bits 5 – 8)0001 CTRL0010 GID0011 0100 0101 CRC-80110 0111 MST1000 1001 1010 1011 1100 1101 SQ MSBs (bits 1 – 4)1110 SQ LSBs (bits 5 – 8)1111

21. VCAT Example GID-a, SQ#=01 GID-a, SQ#=12 GID-a, SQ#=23 6 5 4 12 11 GID-b, SQ#=310 GID-b, SQ#=29 GID-b, SQ#=18 GID-a, SQ#=37 STS-1-4vSTS-1-3v STS-12

22. VCAT Puzzle An STS-1-2v between Chicago and London One STS-1 sent via geostationary satellite The other STS-1 sent via transatlantic fibre Will it work? ;-)

23. VCAT: Buffer Requirements VC Path Type Transport Signal Number Paths Buffer Size STS-1 VC-3 STS-12 STM-4 1218 MB STS-1 VC-3 STS-48 STM-16 4871 MB STS-3c VC-4 STS-12 STM-4 418 MB STS-3c VC-4 STS-48 STM-16 1255 MB

24. Link Capacity Adjustment Scheme Modify membership of a VCG On-demand and hitless bandwidth changes Automatic removal of failed VCG members Interworking of LCAS-enabled VCG with non-LCAS VCG Old-style ‘bridge and roll’ requires double bandwidth

25. LCAS Messaging Member Status (MST) Re-Sequence Acknowledge (RS-Ack) Control (CTRL) –Fixed –Add –Norm –EOS –Idle –DNU Group ID CRC

26. LCAS Signalling Flows (1/3) CTRL = ADD SQ = 255 Source PTESTS-1 #1 CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 MST = OK (#3) CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 STS-1 #2STS-1 #3 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 CTRL = EOS SQ = 3 CTRL = IDLE SQ = 255 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 RS-ACK CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 CTRL = EOS SQ = 3 MST = FAIL (#3)

27. LCAS Signalling Flows (2/3) CTRL = EOS SQ = 3 Source PTESTS-1 #1 CTRL = NORM SQ = 2 CTRL = IDLE SQ = 255 CTRL = EOS SQ = 3 CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 MST = FAIL (#1) CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 STS-1 #2STS-1 #3 RS-ACK CTRL = EOS SQ = 2 CTRL = NORM SQ = 1

28. LCAS Signalling Flows (3/3) CTRL = IDLE SQ = 255 Source PTESTS-1 #1 CTRL = NORM SQ = 2 CTRL = EOS SQ = 3 CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 MST = FAIL (#3) CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 STS-1 #2STS-1 #3 RS-ACK CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 CTRL = NORM SQ = 1

29. Gigabit Ethernet Implementations

30. Gigabit Ethernet (802.3z) FC-0 Interface & Media FC-1 Encode/Decode FC-2 Signalling FC-3 Common Services FC-4 Upper Layer Mapping IEEE 802.3 PHY IEEE 802.3 CSMA/CD IEEE 802.2 LLC ANSI X3T11 Fibre ChannelIEEE 802.3 EthernetIEEE 802.3z Gigabit Ethernet MDL IEE 802.3 MAC IEEE 802.2 LLC PMD PMA PCS

31. 1000BASE-LX/SX

32. Sequence of Events LINK_NOT_AVAILABLE ~LINK_CONFIGURATION ~ IDLE~ SOPPREAMBLE SFDDALLC DataSALENGTH PADDINGFCS

33. Generic Framing Procedure ITU-T G.7041 and ANSI T1.105.02 Defines mapping for many types of service onto SONET/SDH or OTN: –Ethernet, IP/PPP –GbE, Fibre Channel (inc. DVB), FICON etc Excellent bandwidth utilization; efficiency tailored to suit different client types Simple delineation and robust error control Extensible

34. Motivation for GFP (1/2) ATM –Cell overhead causes 10% bandwidth inflation –Adaptation functions needlessly complex Packet over SONET (POS) –Requires all frames to be converted to PPP over HDLC –Byte stuffing causes non-deterministic bandwidth inflation –QoS hard to monitor or guarantee

35. Motivation for GFP (2/2) Minimal overhead Transport of client PDUs in native format Designed for optimized processing Easy aggregation of frames from multiple client and multiple protocols into shared bandwidth channels Low latency capabilities for SAN

36. GFP: Types of Frame GFP Client Frames Control Frames Data Frames Management Frames Idle Frames OA&M Frames

37. GFP: Functional Model Source = IEEE Communications Magazine, May 2002

38. GFP: Frame Structure Source = IEEE Communications Magazine, May 2002

39. Payload Types Value of UPI FieldProtocolGFP Mode 0x01EthernetF 0x02PPP (IP & MPLS)F 0x03Fibre ChannelT 0x04FICONT 0x05ESCONT 0x06GbET 0x07reserved 0x08MAPOSF 0x09 – 0xFEreserved 0x00 & 0xFFunavailable

40. GFP: Client-Independent Processes Synchronization –Bit Level –Frame Delineation Scrambling Multiplexing –GFP Frame –Client PDU

41. GFP: Synchronization HuntPresync Sync cHEC Mismatch 2 nd cHEC Match cHEC Match No 2 nd cHEC Match cHEC Match No CHEC Match

42. GFP: Frame Delineation PayloadPLIcHECPLIcHECPayload PLIcHECPayloadPLIcHECPayloadPLIcHEC PayloadPLIcHEC

43. GFP: Client PDU Multiplexing SourceSink STS-3c-2v GFP Framer 1 2 3 1 2 3 1 2 3

44. GFP-F Encapsulation (1/2) Preamble SFD Destination Source Length Data FCS PLI (MSB) PLI (LSB) cHEC (MSB cHEC (LSB) Payload Header Payload FCS 18 Bit Transmission Order Transmission Order 7 1 6 6 2 4 4 X = 4 to 64 0 to 65,535 - X Ethernet MAC Frame

45. GFP-F Encapsulation (2/2) Flag (0x7E) Address Control PPP Type Data FCS PLI (MSB) PLI (LSB) cHEC (MSB cHEC (LSB) Payload Header Payload FCS 18 Bit Transmission Order Transmission Order 4 X = 4 to 64 0 to 65,535 - X PPP/HDLC Frame 1 1 2 4 1

46. GFP-T: 64B/65B Payload D1K1D2D3K2D4D5D6 000001010011101110111F 65B Sequence1 001 C1D2D3D4D5D610 101 C2D1 100Octet Number 64B Sequence 000001010011100101110111Octet Number

47. GFP-T Error Detection Leading flag bit is errored Error affects ‘Last control-code’ indicator Control-code location address received in error Error causes 4-bit control code to be modified

48. GFP-T: Superblocks 1,11,2…1,71,8 8,18,28,7…8,8 CRC (MSB) CRC (LSB) 1,1 1,2 … 1,7 1,8 8,1 8,2 … 8,7 8,8 536-bit Byte-Aligned

49. GFP-T: Bandwidth Requirements Client Signal Client Signal Bandwidth Minimum Transport Channel Size Nominal Transport Channel Bandwidth Number Superblocks per GFP Frame ESCON160 Mb/s STS-1-4v VC-3-4v 193.536 Mb/s1 Fibre Channel 850 Mb/s STS-3c-4v VC-4-6v 898.56 Mb/s13 Gigabit Ethernet 1000 Mb/s STS-3c-7v VC-4-7v 1.04832 Gb/s95

50. Pros and Cons: GFP-F  Higher bandwidth efficiency  Higher Latency  More buffer memory required  Core header fields must be calculated

51. Pros and Cons: GFP-T  Supports many protocols  Low latency  Ingress core header fields need not be calculated  Less bandwidth efficient  More logic required

52. 10 G Ethernet – 802.3ae 10GBASE-R10GBASE-W S = 850 nmMM300 m L = 1310 nmSM10 km E = 1550 nmSM40 km Full Duplex 802.3 MAC 64B/66B PCS Serial PMA WIS ELSELS XGMII PMD

53. 10 GbE Overview Usual 802.3 MAC and frame format –Jumbo frames not included in standard Full duplex only –No shared media –No CSMA/CD Only optical fibre –No copper interface –LAN-PHY or WAN-PHY at various reaches

54. The STS-192c WIS Frame J1 B3 C2 G1 F2 H4 Z3 Z4 Z5 Fixed Stuff Synchronous Payload Envelope (SPE) = 16704 Columns 63 columns 9.58464 Gbs -1 Capacity 802.3 FrameIdle 802.3 FrameIdle … …

55. Section & Line Overhead A1 B1 D1 A1 … A2 … Section A2 E1 D2 J0 F1 D3 Z0 … Overhead … H1 B2 D4 D7 D10 S1 H1 B2 Z1 … … … H1 B2 Z1 H2 K1 D5 D8 D11 Z2 … … H2 Z2 H2 M1 H3 K2 D6 D9 D12 E2 … H3 LineOverhead … 12192193194384385386576 Defined by WAN PHY as Fixed Value Supported by WAN PHY Unused by WAN PHY Optional for SONET/SDH Unused by WAN PHY Undefined for SONET/SDH

56. NetherLight Connectivity

57. UKLight – NetherLight Today MetroDirector UKLight 15454 NetherLight HDXc NetherLight 15454 CERN CoreDirector UKLight MetroDirector UKLight

58. UKLight – NetherLight Tomorrow? HDXc NetherLight CoreDirector UKLight MetroDirector UKLight 15454 CERN Requires that MSPPs are GFP-F enabled 15454 NetherLight

59. Peering Using 10 G WAN-PHY Force10 StarLight CoreDirector UKLight 10GBASE-SW CoreDirector UKLight STM-64 10GBASE-SW Switch UKLight London Chicago