1. Opticomm 2001Nick McKeown1 Do Optics Belong in Internet Core Routers? Keynote, Opticomm 2001 Denver, Colorado Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford.edu http://www.stanford.edu/~nickm

2. Opticomm 2001Nick McKeown2 There seem to be 3 opinions… 1.“Optics and routers don’t belong together”  Optics are ill-suited to packet switching.  CMOS technology and architectural techniques will scale just fine. 2.“Optical circuit switches will kill off (core) routers”  Optical circuit switches are simpler and faster than routers.  We don’t need packet switching anymore. 3.“Optics and routers belong together”  Electronic switched “backplanes” will be replaced by optics.  Leads to lower power, higher density routers.

3. Opticomm 2001Nick McKeown3 1.“ Optics and routers don’t belong together” Optics are ill-suited to packet switching Buffering  Packet switches inherently require buffers for times of congestion,  Buffers provide statistical multiplexing for outgoing links,  Optical buffers are not economically feasible. Processing  Packet processing is too complex to be done optically.

4. Opticomm 2001Nick McKeown4 Buffering B A time rate x x A x B x A B C 2x C < 2x A+B time rate  The Internet is built on the assumption of expensive, congested links.  Statistical multiplexing enables sharing of expensive links.  All routers have big buffers.  Rule of thumb: buffer size ~= RTT * line-rate. At 10Gb/s: 2.5Gbits.  The Internet is built on the assumption of expensive, congested links.  Statistical multiplexing enables sharing of expensive links.  All routers have big buffers.  Rule of thumb: buffer size ~= RTT * line-rate. At 10Gb/s: 2.5Gbits.

5. Opticomm 2001Nick McKeown5 Processing Physical Layer Framing & Maintenance Packet Processing Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Buffer & State Memory Typical IP Router Linecard Lookup Tables Backplane Buffered or Bufferless Fabric Arbitration Optics OC192c linecard:  30M gates  2.5Gbits of memory  2 square feet of board  200W  $20k cost OC192c linecard:  30M gates  2.5Gbits of memory  2 square feet of board  200W  $20k cost

6. Opticomm 2001Nick McKeown6 1. “ Optics and routers don’t belong together” CMOS and router architectures will scale just fine Growth in capacity of electronic routers:  Capacity 1992 ~ 2Gb/s  Capacity 1995 ~ 10Gb/s  Capacity 1998 ~ 40Gb/s  Capacity 2001 ~ 160Gb/s  Capacity 2003 ~ 1-40Tb/s Main techniques for increasing capacity in electronic routers:  Separating linecards from switch cores.  Parallelism and load-balancing.

7. Opticomm 2001Nick McKeown7 Current “3 rd generation” Routers Switched Backplane Line Interface CPU Memory Line Card MAC Local Buffer Memory CPU Card Line Card MAC Local Buffer Memory Fwding Table Routing Table Fwding Table Typically <=160Gb/s aggregate capacity

8. Opticomm 2001Nick McKeown8 Arbiter 3 rd Generation Routers Queueing Structure Switch 1 write per “cell” time1 read per “cell” time Rate of writes/reads determined by switch fabric speedup Per-flow/class or per- output queues (VOQs) Per-flow/class or per- input queues Flow-control backpressure

9. Opticomm 2001Nick McKeown9 3 rd Generation Routers 19” or 23” 7’ Size-constrained: 19” or 23” wide. Power-constrained: 5kW for 640Gb/s is typical.

10. Opticomm 2001Nick McKeown10 Separating linecards from switch cores 4 th Generation Routers/Switches Switch Core Linecards 1000’s of feet The LCS Protocol 0.3 - 10Tb/s routers in development Optical links

11. Opticomm 2001Nick McKeown11 1. “ Optics and routers don’t belong together” CMOS and router architectures will scale just fine Growth in capacity of electronic routers:  Capacity 1992 ~= 2Gb/s  Capacity 1995 ~= 10Gb/s  Capacity 1998 ~= 40Gb/s  Capacity 2001 ~= 160Gb/s  Capacity 2003 ~1-40Tb/s Main techniques for increasing capacity in electronic routers:  Separating linecards from switch cores.  Parallelism and load-balancing.

12. Opticomm 2001Nick McKeown12 Parallelism and Load-Balancing Techniques in development for linecards at 10’s of Gb/s, but not discussed here:  Parallel packet buffers.  Parallel lookup tables. Discussed here:  Load-balancing across multiple parallel routers.

13. Opticomm 2001Nick McKeown13 Multiple parallel routers Big Router: R RR R The building blocks: R R R R NxN IP Router capacity 100s of Tb/s

14. Opticomm 2001Nick McKeown14 Multiple parallel routers Load Balancing architectures RR R 1 2 … … k R R R R/k R R R

15. Opticomm 2001Nick McKeown15 Method #1: Random packet load- balancing Method: As packets arrive they are randomly distributed, packet by packet over each router. Advantages:  Almost unlimited capacity  Load-balancer is simple  Load-balancer needs no packet buffering Disadvantages:  Random fluctuations in traffic  each router is loaded differently Packets within a flow may become mis-sequenced It is not possible to predict the system performance

16. Opticomm 2001Nick McKeown16 Method #2: Random flow load- balancing Method: Each new flow (e.g. TCP connection) is randomly assigned to a router. All packets in a flow follow the same path. Advantages:  Almost unlimited capacity  Load-balancer is simple (e.g. hashing of flow ID).  Load-balancer needs no packet buffering.  No mis-sequencing of packets within a flow. Disadvantages:  Random fluctuations in traffic  each router is loaded differently It is not possible to predict the system performance

17. Opticomm 2001Nick McKeown17 Observations Random load-balancing: It’s hard to predict system performance. Flow-by-flow load-balancing: Worst-case performance is very poor. If designers, system builders, network operators etc. need to know the worst case performance, random load-balancing will not suffice. (Conversely: If they don’t, then it will).

18. Opticomm 2001Nick McKeown18 Method #3: Intelligent packet load-balancing Goal: Each new packet is carefully assigned to a router so that: Packets are not mis-sequenced. The throughput is maximized and understood. Delay of each packet can be controlled. We call this “Parallel Packet Switching”

19. Opticomm 2001Nick McKeown19 Method #3: Intelligent packet load- balancing Parallel Packet Switching 1 2 k 1 N rate, R 1 N Router Bufferless R/k

20. Opticomm 2001Nick McKeown20 Parallel Packet Switching Advantages  Single-stage of buffering  No excess link capacity  k  power per subsystem   k  memory bandwidth   k  lookup rate 

21. Opticomm 2001Nick McKeown21 Example of an IP Router with Parallel Packet Switching 1 2 16 1 1024 160Gb/s rate, R 1 1024 10Tb/s router R/k Overall capacity 160Tb/s

22. Opticomm 2001Nick McKeown22 1.“Optics and routers don’t belong together” Summary If optics cannot buffer or process packets, and If electronic CMOS-based routers can be built that are fast enough, then Why would anyone try and build an optical router?

23. Opticomm 2001Nick McKeown23 There seem to be 3 opinions… 1.“Optics and routers don’t belong together”  Optics are ill-suited to packet switching.  CMOS technology and architectural techniques will scale just fine. 2.“Optical circuit switches will kill off (core) routers”  Optical circuit switches are simpler and faster than routers.  We don’t need packet switching anymore. 3.“Optics and routers belong together”  Electronic switched “backplanes” will be replaced by optics.  Leads to lower power, higher density routers.

24. Opticomm 2001Nick McKeown24 2. “Optical circuit switches will kill off (core) routers” Optical circuit switches are simpler and faster than routers. A survey of available equipment suggests that, with electronics, you can build a circuit switch that has about 10x the capacity of a packet switch. This is because a packet switch requires lots of complex per-packet processing, … While a circuit switch requires no per- packet processing.

25. Opticomm 2001Nick McKeown25 Processing steps IP Router Per packet:  IP lookup.  Update header & CRC.  Forward to correct output.  Schedule departure. Per route:  Maintain routing entry. Circuit Switch Continuously:  Transfer bits, bytes, photons from input to output. Per circuit:  Establish circuit  Remove circuit

26. Opticomm 2001Nick McKeown26 Why it’s hard for capacity to keep up with link rates 0,1 1 10 100 1000 10000 1985199019952000 Fiber Capacity (Gbit/s) TDMDWDM Packet processing PowerLink Speed 2x / 2 years2x / 7 months Source: SPEC95Int & David Miller, Stanford.

27. Opticomm 2001Nick McKeown27 Instructions per packet time Instructions per packet What we’d like: (more per-packet processing features) More efficient use of links, differentiated services, Multicast, Security, … What will happen

28. Opticomm 2001Nick McKeown28 Normalized number of instructions per packet

29. Opticomm 2001Nick McKeown29 2. “Optical circuit switches will kill off (core) routers” We don’t need packet switching anymore. Original reasons for packet switching no longer hold. There are new techniques, such MPLambaS, burst switching, and TCP Switching that all make it possible to use circuit switching in the core. Actually, most of the core is circuit switched already!

30. Opticomm 2001Nick McKeown30 Original reasons for packet switching 1.Efficient use of expensive links: “Circuit switching is rarely used for data networks,... because of very inefficient use of the links” – Gallager. 2.Resilience to failure of links & routers: ”For high reliability,... [the Internet] was to be a datagram subnet, so if some lines and [routers] were destroyed, messages could be... rerouted” – Tanenbaum. Source: Networking 101

31. Opticomm 2001Nick McKeown31 Neither reason is true today 1.Link capacity is abundant and under used  Most links are unused due to lack of switching capacity.  Most links are utilized < 10%.  Utilization continues to decrease. 2.Routers rarely fail  They are designed for <5s down-time per year.  They take >1min to recover when they do (circuit switches must recover in <50ms).

32. Opticomm 2001Nick McKeown32 How networking people think the Internet is Router

33. Opticomm 2001Nick McKeown33 How the Internet really is Circuit Switched (SONET) Packet Switched (IP routers) $35Bn $6Bn

34. Opticomm 2001Nick McKeown34 How the Internet really is SONET/SDH IP routers Your Local CO Your Local CO

35. Opticomm 2001Nick McKeown35 2. “Optical circuit switches will kill off (core) routers” Summary If the original rationale for packet switching no longer holds, and If circuit switching is inherently faster, and cheaper than packet switching, and If circuit switching is already working fine for most of the Internet already, then Packet switching doesn’t appear to have a long-term future.

36. Opticomm 2001Nick McKeown36 There seem to be 3 opinions… 1.“Optics and routers don’t belong together”  Optics are ill-suited to packet switching.  CMOS technology and architectural techniques will scale just fine. 2.“Optical circuit switches will kill off (core) routers”  Optical circuit switches are simpler and faster than routers.  We don’t need packet switching anymore. 3.“Optics and routers belong together”  Electronic switched “backplanes” will be replaced by optics.  Leads to lower power, higher density routers.

37. Opticomm 2001Nick McKeown37 3. “Optics and routers belong together” Electronic switched “backplanes” will be replaced by optics. The first step is already happening: physical separation of linecards and switch cores. Optical switching is feasible. Scheduling/arbitration is hard.

38. Opticomm 2001Nick McKeown38 Separating linecards from switch cores 4 th Generation Routers/Switches Switch Core Linecards Optical links

39. Opticomm 2001Nick McKeown39 Replacing the switch fabric with optics Switch Fabric Scheduler Physical Layer Framing & Maintenance Packet Processing Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Buffer & State Memory Typical IP Router Linecard Lookup Tables Optics Physical Layer Framing & Maintenance Packet Processing Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Buffer & State Memory Typical IP Router Linecard Lookup Tables Optics Electrical Switch Fabric Scheduler Physical Layer Framing & Maintenance Packet Processing Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Buffer & State Memory Lookup Tables Optics Physical Layer Framing & Maintenance Packet Processing Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Buffer & State Memory Lookup Tables Optics Optical Req/Grant Candidate technologies: MEMs, gratings, passive optical couplers + tunable lasers, holography,… Req/Grant But this is the difficult part…

40. Opticomm 2001Nick McKeown40 Architecture of most routers today Scheduler Per-output queues (VOQs) 1.Scheduler picks new configuration each “cell” time (<50ns for OC192). 2.Scheduling decisions are complex:  “Ideal” algorithm:  O(N 3 ) [maximum weight bipartite matching]  “Good” algorithm:  O(N 2 ) [maximal size bipartite matching]  Requires speedup which reduces cell time. 3.Scheduler chip is typically several million gates, 4.It is hard to use a distributed algorithm. The scheduler is often the bottleneck in the system.

41. Opticomm 2001Nick McKeown41 Overcoming the scheduler bottleneck 1.Increase the internal “cell” size to reduce rate of arbitration and reconfiguration.  Today: 64B is common.  Expect 100s or 1000s of bytes per cell [Kar].  Throughput is not affected.  When does it become circuit switching? 2.Eliminate the need for a scheduler  Two-stage switch [Chang].

42. Opticomm 2001Nick McKeown42 Two-Stage Switch Background 1 N 1 N OutputsInputs Simple Round-Robin It is known that if traffic is uniform and non-bursty, Then a single stage, with virtual output queues, and trivial round-robin (“TDM”) scheduling, gives 100% throughput. Of course, real traffic is non-uniform and bursty.

43. Opticomm 2001Nick McKeown43 Two-Stage Switch 1 N 1 N 1 N External Outputs Internal Inputs External Inputs First Round-RobinSecond Round-Robin Load Balancing Switch gives 100% throughput for non-uniform, bursty traffic, without a scheduler or speedup!

44. Opticomm 2001Nick McKeown44 An optical two-stage switch 1 2 3 Phase 2 Phase 1

45. Opticomm 2001Nick McKeown45 3. “Optics and routers belong together” Summary Optical switches can replace electronic crossbar switches now, Arbitration requires:  Faster (compromised?) schedulers, or  A 2-stage switch fabric.

46. Opticomm 2001Nick McKeown46 So which will it be…? 1.“Optics and routers don’t belong together”  Optics are ill-suited to packet switching.  CMOS technology and architectural techniques will scale just fine. 2.“Optical circuit switches will kill off (core) routers”  Optical circuit switches are simpler and faster than routers.  We don’t need packet switching anymore. 3.“Optics and routers belong together”  Electronic switched “backplanes” will be replaced by optics.  Leads to lower power, higher density routers. A bit of both for a few years:  Continued scaling of electronic routers.  Novel routers incorporating optics. A prediction:  By 2010, almost all of the Internet core will be optical and circuit switched.