Ethernet over SONET Dr. John S. Graham University of London Computer Centre

Ethernet vs SONET FeatureEthernetSONET Guaranteed Delivery No (Packet Switched) Yes (Circuit Switched) Guaranteed BWNoYes Performance Monitoring CRC CountBER ProtectionNone (STP?)APS Fault LocationNoneAIS Access CostsLowHigh

SONET/SDH Model RR MUX DXC MUX Path Layer Connection Line Layer Section Layer

SONET Network Elements

SONET STS-1 Frame Section Line PTR SPE columns Path Payload Capacity = 756 Bytes = Mb/s

Section Overhead Bytes Data Com D3 Data Com D2 Data Com D1 User F1 Orderwire E1 BIP-8 B1 STS-1 ID C1 Framing A2 Framing A1 Section

Line Overhead Bytes Line Orderwire E2 Line FEBE Z2 Sync Stat Z1 Data Com D12 Data Com D11 Data Com D10 Data Com D9 Data Com D8 Data Com D7 Data Com D6 Data Com D5 Data Com D4 APS K2 APS K1 BIP-8 B2

Path Overhead Bytes Path Multi-frame H4 User Channel F2 Path Status G1 Signal Label C2 BIP-8 B3 Path Trace J1

Accurate Bit-Level Timing Line Coding –AMI, AMI RZ –HDB3 Block Coding –4B/5B (FDDI) –5B/6B (E3) –8B/10B (Gigabit Ethernet) Scrambling –Frame Synchronous –Self Synchronous

8B/10B Encoding (1/2) Ensures equal number of 1’s and 0’s for –Clock recovery –DC Balance Detection of single bit transmission errors Predefined ‘comma’ ( and ) characters

8B/10B Encoding (2/2) Data (D) and commands (K) encoded separately Octet split into 5 MSB and 3 LSB Code groups generated from lookup tables Decoded ABCDEFGH Encoded ABCDEiFGHj D2.0

Control Words (‘Ordered Sets’) SymbolEncodingPurpose FK28.5 D21.5LINK_NOT_AVAILABLE CK28.5 D10.5 config_regLINK_CONFIGURATION I1K28.5 D5.6Idle I2K28.5 D16.2Idle SK27.7SoF TK29.7EoF RK23.7CARRIER_EXTEND HK30.7ERROR

Scrambling in SONET/SDH SDH STM-1 uses x 7 + x 6 +1 polynomial

Concatenation STS-3155 Mb/s Mb/s STS-3c/VC-4155 Mb/s Mb/s STS Mb/s Mb/s VC Mb/s 599 Mb/s STS-12c622 Mb/s Mb/s

Provisioning Example A D B E C F ??? 1 – 12(622 Mb/s)

Bandwidth Fragmentation STS-3c STS-12 Blocked! Drop Interface (Node F) Ring

Virtual Concatenation STS-3c STS-12 STS-1-3v Drop Interface (Node F) Ring

The Magic of VCAT Protocol Client Rate Contiguous Concatenation Virtual Concatenation Fast Ethernet100STS-3c65%STS-1-2v98% ESCON160STS-12c26%STS-1-4v78% Fibre Channel850STS-48c34%STS-3c-6v92% Gigabit Ethernet1000STS-48c40%STS-3c-7v92% Fibre Channel 21700STS-48c66%STS-3c-12v92%

Differential Delay TXTX TyTy Source STS-1-12v Sink Core Network (Partial Mesh)

Multiframe Indicator Bit 1Bit 2Bit 3Bit 4Bit 5Bit 6Bit 7Bit 8 MFI2 MSBs (bits 1 - 4)0000 MFI2 LSBs (bits 5 – 8)0001 CTRL0010 GID CRC MST SQ MSBs (bits 1 – 4)1110 SQ LSBs (bits 5 – 8)1111

VCAT Example GID-a, SQ#=01 GID-a, SQ#=12 GID-a, SQ#= GID-b, SQ#=310 GID-b, SQ#=29 GID-b, SQ#=18 GID-a, SQ#=37 STS-1-4vSTS-1-3v STS-12

VCAT Puzzle An STS-1-2v between Chicago and London One STS-1 sent via geostationary satellite The other STS-1 sent via transatlantic fibre Will it work? ;-)

VCAT: Buffer Requirements VC Path Type Transport Signal Number Paths Buffer Size STS-1 VC-3 STS-12 STM MB STS-1 VC-3 STS-48 STM MB STS-3c VC-4 STS-12 STM MB STS-3c VC-4 STS-48 STM MB

Link Capacity Adjustment Scheme Modify membership of a VCG On-demand and hitless bandwidth changes Automatic removal of failed VCG members Interworking of LCAS-enabled VCG with non-LCAS VCG Old-style ‘bridge and roll’ requires double bandwidth

LCAS Messaging Member Status (MST) Re-Sequence Acknowledge (RS-Ack) Control (CTRL) –Fixed –Add –Norm –EOS –Idle –DNU Group ID CRC

LCAS Signalling Flows (1/3) CTRL = ADD SQ = 255 Source PTESTS-1 #1 CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 MST = OK (#3) CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 STS-1 #2STS-1 #3 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 CTRL = EOS SQ = 3 CTRL = IDLE SQ = 255 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 RS-ACK CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 CTRL = EOS SQ = 3 MST = FAIL (#3)

LCAS Signalling Flows (2/3) CTRL = EOS SQ = 3 Source PTESTS-1 #1 CTRL = NORM SQ = 2 CTRL = IDLE SQ = 255 CTRL = EOS SQ = 3 CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 MST = FAIL (#1) CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 STS-1 #2STS-1 #3 RS-ACK CTRL = EOS SQ = 2 CTRL = NORM SQ = 1

LCAS Signalling Flows (3/3) CTRL = IDLE SQ = 255 Source PTESTS-1 #1 CTRL = NORM SQ = 2 CTRL = EOS SQ = 3 CTRL = NORM SQ = 2 CTRL = NORM SQ = 1 MST = FAIL (#3) CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 STS-1 #2STS-1 #3 RS-ACK CTRL = EOS SQ = 2 CTRL = NORM SQ = 1 CTRL = NORM SQ = 1

Gigabit Ethernet Implementations

Gigabit Ethernet (802.3z) FC-0 Interface & Media FC-1 Encode/Decode FC-2 Signalling FC-3 Common Services FC-4 Upper Layer Mapping IEEE PHY IEEE CSMA/CD IEEE LLC ANSI X3T11 Fibre ChannelIEEE EthernetIEEE 802.3z Gigabit Ethernet MDL IEE MAC IEEE LLC PMD PMA PCS

1000BASE-LX/SX

Sequence of Events LINK_NOT_AVAILABLE ~LINK_CONFIGURATION ~ IDLE~ SOPPREAMBLE SFDDALLC DataSALENGTH PADDINGFCS

Generic Framing Procedure ITU-T G.7041 and ANSI T Defines mapping for many types of service onto SONET/SDH or OTN: –Ethernet, IP/PPP –GbE, Fibre Channel (inc. DVB), FICON etc Excellent bandwidth utilization; efficiency tailored to suit different client types Simple delineation and robust error control Extensible

Motivation for GFP (1/2) ATM –Cell overhead causes 10% bandwidth inflation –Adaptation functions needlessly complex Packet over SONET (POS) –Requires all frames to be converted to PPP over HDLC –Byte stuffing causes non-deterministic bandwidth inflation –QoS hard to monitor or guarantee

Motivation for GFP (2/2) Minimal overhead Transport of client PDUs in native format Designed for optimized processing Easy aggregation of frames from multiple client and multiple protocols into shared bandwidth channels Low latency capabilities for SAN

GFP: Types of Frame GFP Client Frames Control Frames Data Frames Management Frames Idle Frames OA&M Frames

GFP: Functional Model Source = IEEE Communications Magazine, May 2002

GFP: Frame Structure Source = IEEE Communications Magazine, May 2002

Payload Types Value of UPI FieldProtocolGFP Mode 0x01EthernetF 0x02PPP (IP & MPLS)F 0x03Fibre ChannelT 0x04FICONT 0x05ESCONT 0x06GbET 0x07reserved 0x08MAPOSF 0x09 – 0xFEreserved 0x00 & 0xFFunavailable

GFP: Client-Independent Processes Synchronization –Bit Level –Frame Delineation Scrambling Multiplexing –GFP Frame –Client PDU

GFP: Synchronization HuntPresync Sync cHEC Mismatch 2 nd cHEC Match cHEC Match No 2 nd cHEC Match cHEC Match No CHEC Match

GFP: Frame Delineation PayloadPLIcHECPLIcHECPayload PLIcHECPayloadPLIcHECPayloadPLIcHEC PayloadPLIcHEC

GFP: Client PDU Multiplexing SourceSink STS-3c-2v GFP Framer

GFP-F Encapsulation (1/2) Preamble SFD Destination Source Length Data FCS PLI (MSB) PLI (LSB) cHEC (MSB cHEC (LSB) Payload Header Payload FCS 18 Bit Transmission Order Transmission Order X = 4 to 64 0 to 65,535 - X Ethernet MAC Frame

GFP-F Encapsulation (2/2) Flag (0x7E) Address Control PPP Type Data FCS PLI (MSB) PLI (LSB) cHEC (MSB cHEC (LSB) Payload Header Payload FCS 18 Bit Transmission Order Transmission Order 4 X = 4 to 64 0 to 65,535 - X PPP/HDLC Frame

GFP-T: 64B/65B Payload D1K1D2D3K2D4D5D F 65B Sequence1 001 C1D2D3D4D5D C2D1 100Octet Number 64B Sequence Octet Number

GFP-T Error Detection Leading flag bit is errored Error affects ‘Last control-code’ indicator Control-code location address received in error Error causes 4-bit control code to be modified

GFP-T: Superblocks 1,11,2…1,71,8 8,18,28,7…8,8 CRC (MSB) CRC (LSB) 1,1 1,2 … 1,7 1,8 8,1 8,2 … 8,7 8,8 536-bit Byte-Aligned

GFP-T: Bandwidth Requirements Client Signal Client Signal Bandwidth Minimum Transport Channel Size Nominal Transport Channel Bandwidth Number Superblocks per GFP Frame ESCON160 Mb/s STS-1-4v VC-3-4v Mb/s1 Fibre Channel 850 Mb/s STS-3c-4v VC-4-6v Mb/s13 Gigabit Ethernet 1000 Mb/s STS-3c-7v VC-4-7v Gb/s95

Pros and Cons: GFP-F  Higher bandwidth efficiency  Higher Latency  More buffer memory required  Core header fields must be calculated

Pros and Cons: GFP-T  Supports many protocols  Low latency  Ingress core header fields need not be calculated  Less bandwidth efficient  More logic required

10 G Ethernet – 802.3ae 10GBASE-R10GBASE-W S = 850 nmMM300 m L = 1310 nmSM10 km E = 1550 nmSM40 km Full Duplex MAC 64B/66B PCS Serial PMA WIS ELSELS XGMII PMD

10 GbE Overview Usual MAC and frame format –Jumbo frames not included in standard Full duplex only –No shared media –No CSMA/CD Only optical fibre –No copper interface –LAN-PHY or WAN-PHY at various reaches

The STS-192c WIS Frame J1 B3 C2 G1 F2 H4 Z3 Z4 Z5 Fixed Stuff Synchronous Payload Envelope (SPE) = Columns 63 columns Gbs -1 Capacity FrameIdle FrameIdle … …

Section & Line Overhead A1 B1 D1 A1 … A2 … Section A2 E1 D2 J0 F1 D3 Z0 … Overhead … H1 B2 D4 D7 D10 S1 H1 B2 Z1 … … … H1 B2 Z1 H2 K1 D5 D8 D11 Z2 … … H2 Z2 H2 M1 H3 K2 D6 D9 D12 E2 … H3 LineOverhead … Defined by WAN PHY as Fixed Value Supported by WAN PHY Unused by WAN PHY Optional for SONET/SDH Unused by WAN PHY Undefined for SONET/SDH

NetherLight Connectivity

UKLight – NetherLight Today MetroDirector UKLight NetherLight HDXc NetherLight CERN CoreDirector UKLight MetroDirector UKLight

UKLight – NetherLight Tomorrow? HDXc NetherLight CoreDirector UKLight MetroDirector UKLight CERN Requires that MSPPs are GFP-F enabled NetherLight

Peering Using 10 G WAN-PHY Force10 StarLight CoreDirector UKLight 10GBASE-SW CoreDirector UKLight STM-64 10GBASE-SW Switch UKLight London Chicago