1. Analysis of a Statistics Counter Architecture Devavrat Shah, Sundar Iyer, Balaji Prabhakar & Nick McKeown (devavrat, sundaes, balaji, nickm)@stanford.edu Departments of Electrical Engineering & Computer Science, Stanford University

2. Stanford University 2 Motivation: Typical Line Card Architecture FRAMERFRAMER PACKET PROCESSOR L3/L4 LOOKUP TRAFFIC MANAGER MEMORY SWITCH FABRIC PHYPHY PHYPHY STATS MEMORY This talk

3. Stanford University 3 What is a Statistics Counter? When a packet arrives, it is first classified to determine the type of the packet. Depending on the type(s), one or more counters corresponding to the criteria are updated/incremented

4. Stanford University 4 Examples of Statistics Counters Packet switches maintain counters for –Route prefixes, ACLs, TCP connections –SNMP MIBs Counters are required for –Traffic Engineering Policing & Shaping –Intrusion detection –Performance Monitoring (RMON) –Network Tracing

5. Stanford University 5 Counter Requirements: A standard IP Router Number of Counters –1 Million: per prefix counters –128K : policing counters Size of Counters –32-64 bit counters

6. Stanford University 6 Determine and analyze techniques for building very high speed (>100Gb/s) statistics counters and support an extremely large number of counters. Motivation: How not to compromise

7. Stanford University 7 Why is Implementing this a Hard Problem? OC192c = 10Gb/s; Counters/pkt (C) = 10; R eff = 100Gb/s; Total Counters (N) = 1 million; Counter Size (M) = 64 bits; Counter Memory = N*M = 64Mb; 64 byte packets Read (X) at Rate, R eff 1 packet every 5.12 ns 1 packet every 5.12 ns Effective Arrival Rate, R eff Write (X+1) at Rate, R eff 1 packet every 5.12 ns Modify: X=X+1 Counter Memory (64Mb) 1 million 64 bits Modify: X=X+1

8. Stanford University 8 Why is Existing Memory Technology not Ideal? Use SRAM? + fast enough random access time, but - too expensive, and - too low density to store 64Mb of data. Use DRAM? + high density means we can store data, but - too slow (typically 50ns random access time) - Read-modify-write penalty

9. Stanford University 9 1 1 1 1 1 64 Large Slow Speed DRAM Memory with N =5 counters of size M=64 bits N = 5 Size M Memory Hierarchy Effective Arriving Rate R eff Small High Speed SRAM Memory with N =5 counters of size m =8 < M bits size m 81 81 81 81 81 Size m Counter Management Algorithm (CMA) Read M bitsWrite M bits Speed-down b DRAM Bandwidth = 2R eff /b Large Counter & Corresponding Small Counter

10. Stanford University 10 Questions How large does the SRAM need to be: –To deterministically guarantee that none of the counters in the SRAM overflow, irrespective of the arriving traffic pattern. What Counter Management Algorithm (CMA) should we use?

11. Stanford University 11 t = 1 Counters 001 000 000 000 000 A Bad Case for the Counters …1 N=5, m=3, b=5 t = 3 Counters 001 001 001 000 000 001 001 001 001 000 t = 4 Counters t = 2 Counters 001 001 000 000 000 t = 0 Counters 000 000 000 000 000 N =5 m=3 000 010 001 001 001 t = 6 Counters 000 010 010 001 001 t = 7 Counters 000 001 001 001 001 t = 5 Counters Choose

12. Stanford University 12 A Bad Case for the Counters …2 N=5, m=3, b=5 t = 9 Counters 000 010 010 010 010 t = 11 Counters 000 000 011 011 010 t = 12 Counters 000 000 011 100 010 t = 13 Counters 000 000 011 101 010 t = 8 Counters 000 010 010 010 001 t = 14 Counters 000 000 011 110 010 t = 10 Counters 000 000 011 010 010 t = 15 Counters 000 000 000 111 010 Overflow at t = 16 Choose

13. Stanford University 13 Largest Counter First (LCF-CMA) 2.Compute: Find out the counter with the largest count every b timeslots. green! 3.Update: Schedule update to counter in DRAM 010 000 010 110 001 4.Clear: Set value of the counter in SRAM to zero 010 000 010 000 001 1.In SRAM: N Counters of size m Counters 010 000 010 110 001 N m

14. Stanford University 14 Optimality of LCF-CMA Theorem: LCF-CMA, is optimal in the sense that it minimizes the size of the counter maintained in SRAM

15. Stanford University 15 Minimum Size of the SRAM Counter Theorem: (speed-down factor b>1) LCF-CMA, minimizes the size of counter (in bits) in the SRAM to: f(b, N) = log 2 ln [b/(b-1)] ln bN ________ ( )  (log log N)=

16. Stanford University 16 Implementation Numbers Example: OC192 Line Card:R = 10Gb/sec. No. of counters per packet:C = 10 R eff = R*C:100Gb/s Cell Size :64bytes T eff = 5.12/2 :2.56ns DRAM Random Access Time :51.2ns Speed-down Factor, b >= T/T eff :20 Total Number of Counters:1000, 1 million Size of Counters:64 bits, 8 bits

17. Stanford University 17 Implementation Examples OC192 Line Card Valuesf(b,N)Brute ForceLCF-CMA N=1000 M=8 8SRAM=8Kb DRAM=0 SRAM=8Kb DRAM=8Kb N=1000 M=64 8SRAM=64Kb DRAM=0 SRAM=8Kb DRAM=64Kb N=1000000 M=64 9SRAM=64Mb DRAM=0 SRAM=9Mb DRAM=64Mb

18. Stanford University 18 Conclusion The SRAM size required by LCF-CMA is a very slow growing function of N There is a minimal tradeoff in SRAM memory size The LCF-CMA technique allows a designer to arbitrarily scale the speed of a statistics counter architecture using existing memory technology