1 Content-Aware Device Benchmarking Methodology (draft-hamilton-bmwg-ca-bench-meth-04) BMWG Meeting Maastricht July 2010 Mike Hamilton BreakingPoint Systems

2 Agenda Why draft-hamilton? Charter objections/responses Goals reset Explicit goals of this draft Explicit non-goals of this draft

3 Why draft-hamilton? RFC 2544 doesn’t specifically apply to some modern devices Test vendors are already doing this in a one-off fashion  BreakingPoint, Spirent, Ixia, Agilent, etc.

4 Charter Objections “the scope of the BMWG is limited to technology characterization using simulated stimuli in a laboratory environment.” “Said differently, the BMWG does not attempt to produce benchmarks for live, operational networks This does not restrict BMWG from creating benchmark tests that are representative of VERY SPECIFIC live, operational networks

5 Goals Reset Create a series of benchmark tests to MOST accurately predict device performance under realistic conditions FOR A SPECIFIC SIMULATED NETWORK RFC 2544 Quotes Page 11, Section 18, “Multiple Frame Sizes” “The distribution MAY approximate the conditions on the network in which the DUT would be used.” “The authors do not have any idea how the results of such a test would be interpreted other than to directly compare multiple DUTs in some very specific simulated network”

6 Explicit Goals Repeatable Results Compare Multiple DUTs

7 Explicit Non-Goals Not a replacement of RFC 2544 Total Input Repeatability (discussion to follow)

8 Test Run Setup Methodologies Run RFC 2544 Throughput (64B B) RFC 3511 Throughput (1 kB kB) IMIX Throughput CAIDA Spirent Wikipedia Agilent-simple draft-hamilton-03 (random) draft-hamilton-04 (shell)

9 Test Results

10 Fuzzing Results

11 Draft-04 Highlights and Reasons “Shell” Methodology More reproducible Backoff on ‘realistic’ Compromise Dropped ‘security’ Difficult to scope and maintain currency Maintain ‘fuzzing’ aspect Random but repeatable