1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz Optical Router Project:

2 Outline  Load-Balancing Background  Mis-sequencing Problem  Datapath Architecture  First stage - Segmentation  Second stage – Main Buffering  Third stage - Reassembly

3 Arbitration 160Gb/s Switch Fabric Line termination IP packet processing Packet buffering Line termination IP packet processing Packet buffering 160 Gb/s 160 Gb/s Electronic Linecard #1 Electronic Linecard #625 Request Grant (100Tb/s = 625 * 160Gb/s) 100Tb/s router

4 Load-Balanced Switch External Outputs Internal Inputs 1 N External Inputs Load-balancing cyclic shift Switching cyclic shift 1 N 1 N

5 160 Gbps Linecard Fixed-size Packets Reassembly Segmentation Lookup/ Processing R 1 N 2 VOQs Intermediate Input Block Load-balancing Switching Input Block Output Block R R R R R

6 Outline  Load-Balancing Background  Mis-sequencing Problem  Datapath Architecture  First stage - Segmentation  Second stage – Main Buffering  Third stage - Reassembly

7 Problem: Unbounded Mis-sequencing External Outputs Internal Inputs 1 N External Inputs Spanning Set of Permutations 1 N 1 N

8 Preventing Mis-sequencing  Uniform Frame Spreading:  Group cells by frames of N cells each (frame building)  Spread each frame across all middle linecards  Each middle stage receives the same type of packets => has the same queue occupancy state 111 N Middle stage NN 1N 1 N N 1 N 1

9 Outline  Load-Balancing Background  Missequencing Problem  Datapath Architecture  First stage - Segmentation  Second stage – Main Buffering  Third stage - Reassembly

10 Three stages on a linecard Segmentation/ Frame Building 1st stage 1 2 N Main Buffering 2nd stage 1 2 N R/N RRRR 3rd stage 1 2 N RR Reassembly

11 Technology Assumptions in 2005 DRAM Technology Access Time ~ 40 ns Size ~ 1 Gbits Memory Bandwidth ~ 16 Gbps (16 data pins) On-chip SRAM Technology Access Time ~ 2.5 ns Size ~ 64 Mbits Serial Link Technology Bandwidth ~ 10 Gb/s >100 serial links per chip

12 First Stage Segmentation 1 2 N R variable-size packets 128-byte cells 16-bytes 1 2 N 1 2 N 1 2 N R/8 16-bytes R/8 Frame Building

13 Segmentation Chip (1st stage) Segmentation 1 2 N R variable-size packets 128-byte cells R/8  Incoming: 16x10 Gb/s  Outgoing: 8x2x10 Gb/s  On-chip Memory: N x 1500 bytes = 7.2 Mbits 3.2ns SRAM 16-bytes

14 Frame Building Chip (1st stage)  Incoming: 2x10 Gb/s  Outgoing: 2x10 Gb/s  On-chip Memory: N^2 x 16 bytes = 48 Mbits 3.2ns SRAM 16-bytes 1 2 N 0-15 R/8 16-bytes 0-15 R/8 Frame Building

15 Three stages on a linecard Segmentation/ Frame Building 1st stage 1 2 N Main Buffering 2nd stage 1 2 N R/N RRRR 3rd stage 1 2 N RR Reassembly

16 Packet Buffering Problem Packet buffers for a 160Gb/s router linecard Buffer Memory Write Rate, R One 128B packet every 6.4ns Read Rate, R One 128B packet every 6.4ns 40Gbits Buffer Manager

17 Memory Technology  Use SRAM? + Fast enough random access time, but - Too low density to store 40Gbits of data.  Use DRAM? + High density means we can store data, but - Can’t meet random access time.

18 Can’t we just use lots of DRAMs in parallel? Write Rate, R One 128B packet every 6.4ns Read Rate, R One 128B packet every 6.4ns Buffer Manager Buffer Memory Read/write 1280B every 32ns 1280B Buffer Memory Buffer Memory Buffer Memory Buffer Memory … ………………

19 128B Works fine if there is only one FIFO Write Rate, R One 128B packet every 6.4ns Read Rate, R One 128B packet every 6.4ns Buffer Manager 1280B Buffer Memory 1280B 128B 1280B … ……………… 128B

20 In practice, buffer holds many FIFOs 1280B 1 2 Q e.g.  In an IP Router, Q might be 200.  In an ATM switch, Q might be Write Rate, R One 128B packet every 6.4ns Read Rate, R One 1280B packet every 6.4ns Buffer Manager 1280B 320B ?B 320B 1280B ?B How can we write multiple packets into different queues? … ………………

21 Arriving Packets R Arbiter or Scheduler Requests Departing Packets R 12 1 Q Small head SRAM cache for FIFO heads SRAM Hybrid Memory Hierarchy Large DRAM memory holds the body of FIFOs Q 2 Writing b bytes Reading b bytes cache for FIFO tails Q 2 Small tail SRAM DRAM

22 SRAM/DRAM results  How much SRAM buffering, given:  DRAM Trc = 40ns  Write and read a 128-byte cell every 6.4ns  Let Q = 625, b = 2*40ns/6.4ns = 12.5  Two Options [Iyer]  Zero Latency Qb[2+lnQ] = 61k cells = 66 Mbits  Some Latency Q(b-1) = 7.5k cells = 7.5 Mbits

23 Outline  Load-Balancing Background  Missequencing Problem  Datapath Architecture  First stage - Segmentation  Second stage – Main Buffering  Third stage - Reassembly

24 Problem Statement Queue Manager 40 Gb DRAM 160 Gb/s One 128B cell every 6.4ns One 128B cell every 6.4ns Write Rate, R Read Rate, R

25 Second Stage R/8 Main Buffering 1 2 N R/N 1 2 N 1 2 N R/8 16-bytes bytes

26 Queue Manager Chip (2nd stage)  Incoming: 2x10 Gb/s  Outgoing: 2x10 Gb/s  35 pins/DRAM x 5 DRAMs = 175 pins  SRAM/DRAM Memory: Q(b-1) = 2.8 Mbits 3.2ns SRAM  SRAM linked list = 1 Mbit 3.2ns SRAM 16-bytes 0-15 R/8 16-bytes 0-15 R/8 Main Buffering 1 2 N R/N 5 x 1Gb DRAM R/4

27 Outline  Load-Balancing Background  Missequencing Problem  Datapath Architecture  First stage - Segmentation  Second stage – Main Buffering  Third stage - Reassembly

28 Three stages on a linecard Segmentation/ Frame Building 1st stage 1 2 N Main Buffering 2nd stage 1 2 N R/N RRRR 3rd stage 1 2 N RR Reassembly

29 Third stage Reassembly 1 2 N R variable-size packets R/8  Incoming: 8x2x10 Gb/s  Outgoing: 16x10 Gb/s  On-chip Memory: N x 1500 bytes = 7.2 Mbits 3.2ns SRAM 16-bytes

30  1st stage  1 segmentation chip  8 frame building chips  2nd stage  8 queue manager chips  40 1 Gb DRAMs  3rd stage  1 reassembly chip  Total chip count  18 ASIC chips  40 1 Gb DRAMs Linecard Datapath Requirements