048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion Introduction

Spring – Packet Switch Architectures2 Course Content  Most of the results in the course are less than 10 years old!  Material is new and still changing…  No reference book available yet…  Uses slides by Profs. Nick McKeown and Balaji Prabahakar (Stanford)

Spring – Packet Switch Architectures3 Outline  Background  What is a router?  Why do we need faster routers?  Why are they hard to build?  Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

Spring – Packet Switch Architectures4 What is Routing? R3 A B C R1 R2 R4D E F R5 F R3E D Next HopDestination D

Spring – Packet Switch Architectures5 What is Routing? R3 A B C R1 R2 R4D E F R5 F R3E D Next HopDestination D Data Options (if any) Destination Address Source Address Header ChecksumProtocolTTL Fragment Offset Flags Fragment ID Total Packet LengthT.ServiceHLenVer 20 bytes

Spring – Packet Switch Architectures6 What is Routing? A B C R1 R2 R3 R4D E F R5

Spring – Packet Switch Architectures7 Points of Presence (POPs) A B C POP1 POP3 POP2 POP4 D E F POP5 POP6 POP7 POP8

Spring – Packet Switch Architectures8 Where High Performance Routers are Used R10 R11 R4 R13 R9 R5 R2 R1 R6 R3 R7 R12 R16 R15 R14 R8 (10 Gb/s)

Spring – Packet Switch Architectures9 What a Router Looks Like Cisco GSR 12416Juniper M160 6ft 19 ” 2ft Capacity: 160Gb/s Power: 4.2kW Full rack 3ft 2.5ft 19 ” Capacity: 80Gb/s Power: 2.6kW Half-a-rack

Spring – Packet Switch Architectures10 Core Router Market  Estimates cover different parameters  Dell’Oro (Feb. 17, 2005):  $1.2 billion in 2004 (up 66%)  Includes high-end (10Gbps) router market  Dell’Oro (Feb. 21, 2006):  $4.4 billion in 2005  Includes “worldwide service provider router market”  Infonetics (Feb. 21, 2006):  $5.2 billion in 2005 (up 31%)  Includes core/edge routers  Core router market  Relatively small (in comparison to enterprise market), but brings expertise to company  Vendors: Cisco, Juniper, Avici, Nortel, Lucent, Alcatel, Huawei, etc.

Spring – Packet Switch Architectures11 Router Market Source: Infonetics (Nov. 2004)

Spring – Packet Switch Architectures12 Basic Architectural Components of an IP Router Control Plane Datapath per-packet processing Switching Forwarding Table Routing Table Routing Protocols

Spring – Packet Switch Architectures13 Per-packet processing in an IP Router 1. Accept packet arriving on an incoming link. 2. Lookup: lookup packet destination address in the forwarding table, to identify outgoing port(s). 3. Header Processing: Manipulate packet header: e.g., decrement TTL, update header checksum. 4. Switching: Send packet to the outgoing port(s). 5. Buffering: Buffer packet in the queue. 6. Transmit packet onto outgoing link.

Spring – Packet Switch Architectures14 Generic Router Architecture Lookup IP Address Update Header Header Processing DataHdrDataHdr ~1M prefixes Off-chip DRAM Address Table Address Table IP AddressNext Hop Queue Packet Buffer Memory Buffer Memory ~1M packets Off-chip DRAM

Spring – Packet Switch Architectures15 Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table DataHdrDataHdrDataHdr Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory DataHdrDataHdrDataHdr

Spring – Packet Switch Architectures16 Outline  Background  What is a router?  Why do we need faster routers?  Why are they hard to build?  Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

Spring – Packet Switch Architectures17 Why Do We Need Faster Routers? 1. To prevent routers from becoming the bottleneck in the Internet. 2. To increase POP capacity, and to reduce cost, size and power.

Spring – Packet Switch Architectures18 Why We Need Faster Routers 1: To prevent routers from being the bottleneck Packet Processing Power 2x / 18 months Source: SPEC95Int & Coffman and Odlyzko. Single Fiber Capacity (commercial) ≥ 2x / year

Spring – Packet Switch Architectures19 POP with smaller routers Why we Need Faster Routers 2: To reduce cost, power & complexity of POPs POP with large routers  Ports: Price >$50k, Power > 400W.  It is common for 50-60% of ports to be for interconnection.

Spring – Packet Switch Architectures20 Why are Fast Routers Difficult to Make? 1. It’s hard to keep up with Moore’s Law:  The bottleneck is memory speed.  Memory speed is not keeping up with Moore’s Law.

Spring – Packet Switch Architectures21 Why are Fast Routers Difficult to Make? Speed of Commercial DRAM 1. It’s hard to keep up with Moore’s Law:  The bottleneck is memory speed.  Memory speed is not keeping up with Moore’s Law. Moore’s Law 2x / 18 months 1.1x / 18 months

Spring – Packet Switch Architectures22 Why are Fast Routers Difficult to Make? 1. It’s hard to keep up with Moore’s Law:  The bottleneck is memory speed.  Memory speed is not keeping up with Moore’s Law. 2. Moore’s Law is too slow:  Routers need to improve faster than Moore’s Law.

Spring – Packet Switch Architectures23 Router Performance Exceeds Moore’s Law Growth in capacity of commercial routers:  Capacity 1992 ~ 2Gb/s  Capacity 1995 ~ 10Gb/s  Capacity 1998 ~ 40Gb/s  Capacity 2001 ~ 160Gb/s  Capacity 2003 ~ 640Gb/s Average growth rate: 2.2x / 18 months.

Spring – Packet Switch Architectures24 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

Spring – Packet Switch Architectures25 Route Table CPU Buffer Memory Line Interface MAC Line Interface MAC Line Interface MAC Typically <0.5Gb/s aggregate capacity Shared Backplane Line Interface CPU Memory First-Generation Routers

Spring – Packet Switch Architectures26 Route Table CPU Line Card Buffer Memory Line Card MAC Buffer Memory Line Card MAC Buffer Memory Fwding Cache Fwding Cache Fwding Cache MAC Buffer Memory Typically <5Gb/s aggregate capacity Second-Generation Routers

Spring – Packet Switch Architectures27 Line Card MAC Local Buffer Memory CPU Card Line Card MAC Local Buffer Memory Switched Backplane Line Interface CPU Memory Fwding Table Routing Table Fwding Table Typically <50Gb/s aggregate capacity Third-Generation Routers

Spring – Packet Switch Architectures28 Switch Core Linecards Optical links 100s of metres Tb/s routers Fourth-Generation Routers Multi-Racks, Optical Links

Spring – Packet Switch Architectures29 Optical Switch Core Linecards Optical links 100s of metres Tb/s routers, in project (Future) Fifth-Generation Routers Optical Switch Core

Spring – Packet Switch Architectures30 Optical Switch Core Optical Linecards Optical links 100s of metres Tb/s routers, in the far future (Future) Sixth-Generation Routers All-Optical Routers

Spring – Packet Switch Architectures31 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

Spring – Packet Switch Architectures32 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Lookup IP Address Address Table Address Table Lookup IP Address Address Table Address Table Lookup IP Address Address Table Address Table

Spring – Packet Switch Architectures33 IP Address Lookup  Why it’s thought to be hard:  It’s not an exact match: it’s a longest prefix match.  The table is large: about 150,000 entries today, and growing.  The lookup must be fast: about 30ns for a 10Gb/s line.

Spring – Packet Switch Architectures34 IP Lookups find Longest Prefixes / / / / / / Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.

Spring – Packet Switch Architectures35 IP Addressing Classless Interdomain Routing (CIDR)  The IP address space is broken into line segments.  Each line segment is described by a prefix.  A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment.  e.g. The prefix 128.9/16 represents the line segment containing addresses in the range: … / /

Spring – Packet Switch Architectures36 IP Addressing Classless Interdomain Routing (CIDR) / / =2^7 9=2^3+2^0 16=2^4 14=2^3+2^2+2^ /16 = xxxxxxxx.xxxxxxxx  =

Spring – Packet Switch Architectures37 IP Address Lookup  Why it’s thought to be hard:  It’s not an exact match: it’s a longest prefix match.  The table is large: about 150,000 entries today, and growing.  The lookup must be fast: about 30ns for a 10Gb/s line.

Spring – Packet Switch Architectures38 Address Tables are Large Source:

Spring – Packet Switch Architectures39 IP Address Lookup  Why it’s thought to be hard:  It’s not an exact match: it’s a longest prefix match.  The table is large: about 150,000 entries today, and growing.  The lookup must be fast: about 30ns for a 10Gb/s line.

Spring – Packet Switch Architectures40 Lookups Must be Fast 12540Gb/s Gb/s Gb/s Mb/s B packets (Mpkt/s) LineYear Gb/s2005?

Spring – Packet Switch Architectures41 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

Spring – Packet Switch Architectures42 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory

Spring – Packet Switch Architectures43 Fast Packet Buffers Example: 40Gb/s packet buffer Size = RTT*BW = 10Gb; 40 byte packets Write Rate, R 1 packet every 8 ns Read Rate, R 1 packet every 8 ns Buffer Manager Buffer Memory Use SRAM? + fast enough random access time, but - too low density to store 10Gb of data. Use SRAM? + fast enough random access time, but - too low density to store 10Gb of data. Use DRAM? + high density means we can store data, but - too slow (50ns random access time). Use DRAM? + high density means we can store data, but - too slow (50ns random access time).

Spring – Packet Switch Architectures44 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

Spring – Packet Switch Architectures45 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory DataHdr DataHdr DataHdr 1 2 N 1 2 N N times line rate

Spring – Packet Switch Architectures46 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory DataHdr DataHdr DataHdr 1 2 N 1 2 N DataHdr DataHdr DataHdr Scheduler

Spring – Packet Switch Architectures47 Current Internet Router Technology Summary  There are three potential bottlenecks:  Address lookup,  Packet buffering, and  Switching. Most difficult – and focus of this course!