1. Jim Warner UC Santa Cruz February 2010
Speed match buffers
A little bit of packet loss can ruin your whole day when you’re trying to make TCP go fast.
Jim Warner
UC Santa Cruz
February 2010

2. Solution searching for problem
Campus 10G connection in Nov 2009
PerfSONAR might be useful
Q: What happens when a fast (10 Gb/s) source sends to a slow (Gb/s) client over a long delay path?
CENIC’s transition from Cisco to Foundry hinted this might be interesting.
Can a researcher with a 1 Gig net connection get 1000 Mb/s performance from a 10 gig backbone?
Iperf test to Atlanta: Cisco routers to a Gig-E receiver gave about 720 Mb/s inbound.
Approx the same setup, but with a Brocade/Foundry XMR at the speed match point gave about 250 Mb/s.
Measurements made in Oct 2009.

3. Many of Internet2 bwctl servers are 10G connected.
TCP does not attempt to space out packets, so big slugs packets permitted by the window will arrive at the access switch and queue for delivery.
If the access switch output buffer doesn’t have sufficient space, packets will get dropped. TCP will back off and users won’t see gig performance.

4. How much buffer to expect ??
This is a piece of the specifications for a mid-level access switch with a 10 G uplink and 1 G client ports. TCAM isn’t shown here; it holds address tables and access lists, not packets. This is *not* the switch used in later slides.
The answer is …

5. Not in the data sheet
The answer is that there is packet RAM associated with each of the port ASICs. One port ASIC controls 24 Gig-E ports or two 10G ports. Some of the memory is reserved so that no queue can be starved. Some is available for sharing for them that needs it. But it’s not much memory. Researchers at Berkeley report that that the Nortel 5500 has a max of 131K per gig-E port and a total of 786K shared among 48 ports (Chen, Griffith, Liu, Katz, Joseph, 2009). Cisco has more, but only by a factor of ~2.
As a general thing, output buffer capacity is a hard specification to find. It is almost like switch vendors don’t want to talk about it. So, for example, in Juniper’s white paper on Campus architecture, searching for the words “buffer” or “memory” comes up empty. The Brocade/Foundry GS data sheet gives the “Total Switching Performance” in Gbps, and lists 8 HW queues – but not the packet memory. This parameter is simply not a priority for LAN switches.
But wait. . . Maybe this isn’t a problem. And yet, here I am at the front of the room…

6. 1.6 Mbyte output buffer
Los Angeles is about 350 miles south of Santa Cruz. PerfSONAR runs 20 second bidirectional tests every hour. In Los Angeles, the computer I’m testing against has a 10 G interface and the entire network all the way to port on the Santa Cruz access switch is 10 Gb/s. The Santa Cruz target is the only gig-E interface and the buffer that has to speed match is in its access switch.
1.6 Mbytes of output buffer is generous by industry standards. The biggest buffers on Cisco Cat6500 GE line cards are 1.2 MB. HP has about 700 KB on 3500YL switch.
Green is inbound, exhausting the output buffer “just a little”. Blue is the reverse direction which, for the most part, runs as fast as could be expected.
Our little clip art friend reminds us that we had a fiber munch and fallback to our 1 gig connection during the data collection. The second dip in the blue dots is the feature that aligns with next picture which tests a longer path.

7. 1.6 MB to Seattle
The path to Seattle goes through Los Angeles. It appears that we’re averaging about 500 Mb/s inbound at Santa Cruz.

8. Buffer dialed up to 6 MB
Note: the green dots are still there – they are now underneath the blue ones.
Conclusion: More generous output buffering at the speed step eliminates packet loss that limited inbound data rate.
No time to say this in the talk:
The output port that does the speed step is a 1 Gb/s interface on a Catalyst 6500 Sup720-10G. It has 8.1 GB of output buffer. MLS QOS is turned on. The port has four output queues. About 10 percent is reserved for the high priority queue and cannot be touched. Allocation of memory to queues sets the amount available to the ‘best effort’ queue which is where our test traffic is riding.

9. Same for Seattle
We call this much much better, but not perfect.

10. Is there hope? Vendors will not provide bigger buffers. $$$ and power
Ethernet flow control might help
Change TCP to make it less bursty
About flow control:
If the access switch can send a pause command to its upstream 10 G source, it is in effect borrowing buffer space. Output buffers on 10 G line cards are at least 10X bigger than buffers on gig-E line cards, so this might work. It would, however, create campus debug scenarios that are unparalleled. And it sets up the classic head of line blocking scenario.
The number of proposals in the research literature that would tweak TCP is very large.