Exchange Point Technology Nanog 20, Oct 22-24, 2000 Lane Patterson Member Research Staff Lane Patterson Member Research Staff

2 Intro Framework

3 IX Technology Universe 10/100 Ethernet, FDDI ATM OC3/12 (SAR limitations) GigE Private Copper/Fiber Cross Connect SONET ADM/DCS Frame Relay on POS OC48/192 LSR (over POS, Ethernet, …) 10GigE Optical Switching (w/ signaling of some sort) 10/100 Ethernet, FDDI ATM OC3/12 (SAR limitations) GigE Private Copper/Fiber Cross Connect SONET ADM/DCS Frame Relay on POS OC48/192 LSR (over POS, Ethernet, …) 10GigE Optical Switching (w/ signaling of some sort)

4 IX Connection Models Layer 2 Non-Broadcast Multi-Access (NBMA): TE and connection-oriented = more control at the expense of management and operational overhead Frame Relay or ATM PVCs MPLS LSPs Even 802.1p/q in a “VLAN per /30” model Layer 2 Shared Media FDDI, Ethernet, DPT/SRP Layer 1: SONET DCS style: STS-1, OC-N Wavelength, Dark Fiber Nailed Up TDM Today; GMPLS tomorrow? Layer 2 Non-Broadcast Multi-Access (NBMA): TE and connection-oriented = more control at the expense of management and operational overhead Frame Relay or ATM PVCs MPLS LSPs Even 802.1p/q in a “VLAN per /30” model Layer 2 Shared Media FDDI, Ethernet, DPT/SRP Layer 1: SONET DCS style: STS-1, OC-N Wavelength, Dark Fiber Nailed Up TDM Today; GMPLS tomorrow?

5 IX Applications: Current and Potential Unicast Peering Aggregation Private or Public Multicast Peering Aggregation Interdomain Interconnection Policy Human Enforced Clue Route Servers Interdomain MPLS/GMPLS? QoS Brokering? Not focusing outside of interconnection in this talk (e.g. Stratum 1 servers, other services) Unicast Peering Aggregation Private or Public Multicast Peering Aggregation Interdomain Interconnection Policy Human Enforced Clue Route Servers Interdomain MPLS/GMPLS? QoS Brokering? Not focusing outside of interconnection in this talk (e.g. Stratum 1 servers, other services)

6 What We’re Doing

7 Gigabit Ethernet The one big Unicast VLAN model But private VLANs and tagged interfaces OK Must Mitigate Shared Risk: IX Policy and Enforcement Switch Filters and Knobs Cost, simplicity, and operational advantages over ATM More traffic control features now on GigE lowers the contrast with ATM MTU a problem? Lots of “core Internet MTU” debate But more router data helpful—flow cache packet distributions, frag stats, better analysis of packets that are >1500, etc. Waiting on vendor support: MTU per VLAN tag Will be announced as a product I’m the R&D guy not the product guy—usual non-Nanog channels apply The one big Unicast VLAN model But private VLANs and tagged interfaces OK Must Mitigate Shared Risk: IX Policy and Enforcement Switch Filters and Knobs Cost, simplicity, and operational advantages over ATM More traffic control features now on GigE lowers the contrast with ATM MTU a problem? Lots of “core Internet MTU” debate But more router data helpful—flow cache packet distributions, frag stats, better analysis of packets that are >1500, etc. Waiting on vendor support: MTU per VLAN tag Will be announced as a product I’m the R&D guy not the product guy—usual non-Nanog channels apply

8 GigE Shared Risks No Brainers: Participants ARP: no ip proxy-arp Broadcast: no ip directed-broadcast ICMP Redirects: no ip redirects CDP/other noise: no cdp enable No IGP: passive interface No 3 rd party switches, BPDUs Multicast: keep off unicast exchange VLAN No Brainers: Participants ARP: no ip proxy-arp Broadcast: no ip directed-broadcast ICMP Redirects: no ip redirects CDP/other noise: no cdp enable No IGP: passive interface No 3 rd party switches, BPDUs Multicast: keep off unicast exchange VLAN

9 GigE Shared Risks (cont’d) IX-side STP tuning a must Upcoming 802.1w and STP improvements from today’s 2-4 second reconvergence to milliseconds Block BPDUs on non-IX-trunk ports Layer 2/3 filters per port Trend and alarm bridge tables (# and identity of MACs on each port) Some controls still needed that are easy Some controls may not be worth it: Too much maintenance overhead for ISPs Static ARP, MAC per PORT/VLAN Shaping/Policing policies Filter Maintenance If we really want this, go back to NBMA models IX-side STP tuning a must Upcoming 802.1w and STP improvements from today’s 2-4 second reconvergence to milliseconds Block BPDUs on non-IX-trunk ports Layer 2/3 filters per port Trend and alarm bridge tables (# and identity of MACs on each port) Some controls still needed that are easy Some controls may not be worth it: Too much maintenance overhead for ISPs Static ARP, MAC per PORT/VLAN Shaping/Policing policies Filter Maintenance If we really want this, go back to NBMA models

10 Future GigE Developments Vendors are active 10GigE Other proprietary features for control, scaling Active projects Still Need Multicast IX Features PIM snooping for Port/Group state Vendors are active 10GigE Other proprietary features for control, scaling Active projects Still Need Multicast IX Features PIM snooping for Port/Group state

11 What We’re Tracking/Testing

12 SONET Cross-Connects Aggregate private circuits through SONET DCS Assumes OC48/192 Ports Channelizable to OC-N (both on Router and DCS) Recent DCS Products: up to 512 OC48 down to STS-1 Worthwhile for co-located routers? Already precedent: FloridaMIX Good combo with DWDM into building Extends current SONET aggregation at DS3 & OC12 Cost of SONET DCS ports vs. Other Alternatives Policed Ethernet dot1q alternatives Cost/density/flexibility trade-offs versus continuing to burn DS3/OC3/OC12 router ports Will report back on future testing progress Input is welcome Aggregate private circuits through SONET DCS Assumes OC48/192 Ports Channelizable to OC-N (both on Router and DCS) Recent DCS Products: up to 512 OC48 down to STS-1 Worthwhile for co-located routers? Already precedent: FloridaMIX Good combo with DWDM into building Extends current SONET aggregation at DS3 & OC12 Cost of SONET DCS ports vs. Other Alternatives Policed Ethernet dot1q alternatives Cost/density/flexibility trade-offs versus continuing to burn DS3/OC3/OC12 router ports Will report back on future testing progress Input is welcome

13 NBMA Migration Paths Frame Relay at OC48/OC192 most promising “ATM Upgrade” today Mature standards Switches becoming available MPLS LSRs need more momentum Still intra-domain focused and limited set of ISPs Inter-domain signaling issues Promising longer term for new inter-provider features MPLS/BGP VPN Inter-provider QoS Dynamic B/W provisioning: circuit-em, signaling None of these proven yet Frame Relay at OC48/OC192 most promising “ATM Upgrade” today Mature standards Switches becoming available MPLS LSRs need more momentum Still intra-domain focused and limited set of ISPs Inter-domain signaling issues Promising longer term for new inter-provider features MPLS/BGP VPN Inter-provider QoS Dynamic B/W provisioning: circuit-em, signaling None of these proven yet

14 NBMA Migration Paths (cont’d) Equinix committed to more MPLS LSR Testing Current Inter-Domain Models Non-terminating IX LSR: Static Labels RSVP+ERO but no IGP—statics across directly-connected IX /30’s Terminating IX LSR: Add CCC-style to above combinations Need more robust Inter-Domain Signaling models MBGP NLRI for label exchange a first step Bilateral policy directly on LSRs or real 3 rd party gateway infrastructure? Same issues carry over to GMPLS and optical world Equinix committed to more MPLS LSR Testing Current Inter-Domain Models Non-terminating IX LSR: Static Labels RSVP+ERO but no IGP—statics across directly-connected IX /30’s Terminating IX LSR: Add CCC-style to above combinations Need more robust Inter-Domain Signaling models MBGP NLRI for label exchange a first step Bilateral policy directly on LSRs or real 3 rd party gateway infrastructure? Same issues carry over to GMPLS and optical world

15 Optical Switching Dense numbers of signaled wavelength and fiber cross connects Testing one OXC vendor near term Testing open to interested parties Still early in development For now, wavelength exchange is just a private cross connect between customer-owned DWDM equipment Fiber/wavelength density and signaling-driven applications will drive this onto OXCs Then Bill Norton can do some whiz-bang cost benefit graphs “Multiservice Exchange” Dense numbers of signaled wavelength and fiber cross connects Testing one OXC vendor near term Testing open to interested parties Still early in development For now, wavelength exchange is just a private cross connect between customer-owned DWDM equipment Fiber/wavelength density and signaling-driven applications will drive this onto OXCs Then Bill Norton can do some whiz-bang cost benefit graphs “Multiservice Exchange”

16 Virtual Device Trends… …or lack of them Useful for customer-driven, on-demand provisioning Today not many real virtual devices, just NMS/OSS integration through secure web front-end. Future boxes that can push this: 3 rd party policy brokers Register policy Integrate statistics for operation, billing Admission control in secure signaling domain Real virtual devices: Log in to your virtual instance on a shared box Bandwidth trading systems …or lack of them Useful for customer-driven, on-demand provisioning Today not many real virtual devices, just NMS/OSS integration through secure web front-end. Future boxes that can push this: 3 rd party policy brokers Register policy Integrate statistics for operation, billing Admission control in secure signaling domain Real virtual devices: Log in to your virtual instance on a shared box Bandwidth trading systems

17 Going Forward Large IX Facilities: Focus on removing space, power, and interconnection constraints from scaling equation Continued ability to accept new participants Only limit should be strands*wavelengths into the building/campus Today, Public Fabric: 10/100/1000 Ethernet moving to 10Gig Private Fabric: dark fiber cross connects Under Consideration: SONET cross-connects and aggregation onto OC12/48/192 Large IX Facilities: Focus on removing space, power, and interconnection constraints from scaling equation Continued ability to accept new participants Only limit should be strands*wavelengths into the building/campus Today, Public Fabric: 10/100/1000 Ethernet moving to 10Gig Private Fabric: dark fiber cross connects Under Consideration: SONET cross-connects and aggregation onto OC12/48/192

18 Last Thoughts When future cores are 80 wavelengths of OC192, and migrating to 320 wavelengths of OC768, what do exchange points need to look like? Other than more bandwidth, what inter-domain services will take hold? Will we have any hair left by then? When future cores are 80 wavelengths of OC192, and migrating to 320 wavelengths of OC768, what do exchange points need to look like? Other than more bandwidth, what inter-domain services will take hold? Will we have any hair left by then?