1. Routers with a Single Stage of Buffering Sundar Iyer, Rui Zhang, Nick McKeown High Performance Networking Group, Stanford University, http://yuba.stanford.edu ACM SIGCOMM Friday, Aug 23 rd 2002

2. 2 Contents 1. Background & motivation 2. An abstraction of a router with the pigeon-hole principle 3. Analysis of a FIFO work conserving router 4. Analysis of a PIFO work conserving router 5. Comparison and conclusion

3. 3 How do we design routers? We pick a capacity target & architecture Then & only then do we simulate Load Delay 0100 We hope customers will put up with unpredictable performance We can’t guarantee, that the router is work conserving

4. 4 How can we re-design routers? Then, we see how we can build a router We require that a router be work conserving We pick a capacity target & architecture We take the departure time of a packet as a requirement

5. 5 Motivation (A different way of thinking about routers)  This talk is about routers which are designed from the ground up.  We start with the departure times of the packets & take them as a requirement.  Then, we figure out the number of resources that we need to ensure that packets depart at their departure time.

6. 6 A look at single buffered routers A router with 3 ports with memories which run at the line rate R R R R C1 A1 B1 C2 C3 C2 C3 B1 A1 C1 C-4 FIFO Departure Time = 4 Destined to output C C4 Arriving packets Packets in buffers Empty buffers Final arriving packet

7. 7 Contents 1. Background & motivation 2. An abstraction of a router with the pigeon-hole principle 3. Analysis of a FIFO work conserving router 4. Analysis of a PIFO work conserving router 5. Comparison and conclusion

8. 8 An abstract model  There are P pigeon holes which can contain an infinite number of pigeons.  Assume that time is slotted, and in any one time slot  at most N pigeons can arrive and at most N can depart.  at most one pigeon can enter or leave a specific pigeon hole.  When a pigeon arrives, we know the exact time slot at which it will depart.  For any router:  What is the minimum P, such that all N pigeons can be immediately placed in a pigeon hole when they arrive, and can depart at the right time?

9. 9 Solving the abstract model  When a pigeon arrives in a time slot it must satisfy the following constraints  No more than N – 1 other pigeons arrive at that timeslot.  No more than N other pigeons depart at that timeslot.  No more than N - 1 other pigeons can depart at the same time as this pigeon departs (perhaps in future).  By the pigeon-hole principle,  3N –1 pigeon holes are sufficient for each pigeon to arrive and leave the pigeon holes without conflict.

10. 10 A take away.. If you want to build a work conserving router …  Evaluate the uniquely assigned departure time for every packet.  Define the constraints on every packet.  Apply the pigeon-hole principle.  Find the minimum resources required so that every packet departs at the right time.

11. 11 Contents 1. Background & motivation 2. An abstraction of a router with the pigeon-hole principle 3. Analysis of a FIFO work conserving router 4. Analysis of a PIFO work conserving router 5. Comparison and conclusion

12. 12 E.g. 1: Parallel Shared Memory Router R A router with 3 ports and k=7 memories running at rate R R R R R B A C k = 7 memories at rate R is not sufficient.... but 8 memories suffice R

13. 13 R A router with 3 ports and k=8 memories running at rate R R R R R B A C By the pigeon-hole principle, k  3N –1 memories at rate R, suffices. R How many memories for FIFO? (Parallel Shared Memory)

14. 14 Switch Fabric Line Card Line Card Line Card Buffer E.g. 2: DSM Router (Distributed Shared Memory) The central memories have been moved to distributed line cards.

15. 15 Why is the DSM router interesting?  The bandwidth in & out of each memory is at the line rate.  The memory is shared across inputs and outputs.  Memory and line cards can be added incrementally.

16. 16 A router with 3 ports and k=9 memories running at rate R By the pigeon-hole principle, k  3N -1 memories at rate R, suffices. Line Card Line Card Line Card Switch Fabric How many memories for FIFO? (Distributed Shared Memory)

17. 17 Previous Work (which use the counting/pigeon-hole principle)  Combined Input Output Queued (CIOQ) Router  Prabhakar and McKeown, -1997  Chuang, Goel, McKeown & Prabhakar -1998  Krishna, Patel, Charny, Simcoe –1998  Charny - 1998  Parallel Packet Switch (PPS)  Iyer, Awadallah & McKeown – 2000  Iyer & McKeown - 2001

18. 18 Contents 1. Background & motivation 2. An abstraction of a router with the pigeon-hole principle 3. Analysis of a FIFO work conserving router 4. Analysis of a PIFO work conserving router 5. Comparison and conclusion

19. 19 What is the Problem with non-FIFO? …1  First Problem:  The departure time of a cell is not fixed in policies such as strict priority, weighted fair queueing etc.  The counting technique depends on being able to predict the departure time and schedule it.  We will use an model called PIFO which captures a number of these scheduling policies for analysis

20. 20 A Push in First Out (PIFO) Queue  Arriving packets are “pushed-in” to an arbitrary location in the departure queue.  Packets depart from the head of line.  The relative ordering between packets in the queue does not change.

21. 21 Enabling QoS with PIFO queues C1 B2B1A3A2A1C1 WFQ with 3 per-flow queues, destined to same output, Weights 3:2:1 C2 B1B2 A1A2A3 C2 B2B1A3A2A1C1 B3 A single PIFO queue for an output, Weights 3:2:1 C2

22. 22 How many memories for PIFO? (Parallel Shared Memory) R A router with 3 ports and k=9 memories running at rate R R 2.5 R R R B A C R 2 4 1 3 k = 9 memories at rate R is not sufficient.. but 10 memories suffice 2.4

23. 23 How many memories are needed? (Distributed Shared Memory)  A cell which arrives at time t, destined to output port j must not be written to memories which 1.Are used to write the other N-1 arriving cells at t. 2.Are used to read the N departing cells at t. 3.Will be used to read the N-1 cells of output j before t. 4.Will be used to read the N-1 cells of output j after t.  There are four constraint sets  By the pigeon-hole principle, 4N-2 memories at rate R, or a memory bandwidth of 4NR is sufficient

24. 24 Contents 1. Background & motivation 2. An abstraction of a router with the pigeon-hole principle 3. Analysis of a FIFO work conserving router 4. Analysis of a PIFO work conserving router 5. Comparison and conclusion

25. 25 Comparison of FIFO & PIFO (Distributed Shared Memory) ArbiterBW of fabric BW of Mem. BW of fabric BW. Of Mem. Hard 4NR 3NR DSM Router 5NR Easy 4NR6NR3NR 8NR Easy 6NR4NR 6NR FIFO PIFO

26. 26 Comparison with the CIOQ Router CIOQDSM 2NR5NR Total Fabric BW 6NR4NR Total Memory BW  The DSM has a lower memory bandwidth than the CIOQ, which is the main bottleneck in routers.  The DSM arbiter is simpler than the CIOQ arbiter.

27. 27 Conclusion  There was a class of routers which were not analyzed earlier  Distributed/Parallel shared memory  Existing techniques were not applicable  We came up with a counting technique based on the pigeon-hole principle  Single buffered routers  Combined Input-Output Queued router