1. 1 Benchmarking Methodology WG (bmwg) 64th IETF – Vancouver, Canada Monday, November 7, 2005, 15:10-17:10 (Oak) Chairs: –Kevin Dubray (kdubray@juniper.net) –Al Morton (acmorton@att.com) If you are not subscribed to the BMWG mailing list and would like to be, put an asterisk (*) by your name when you sign the blue attendance sheet.

2. 2 BMWG Agenda Note-Taker(s), IPR, and Blue Sheets WG Status – with some discussion (Chairs, 20 min) IPsec Benchmarking Terms & New Methods (Kaeo, 20 min) IGP-Dataplane Convergence I-Ds. (Poretsky, 15 min) Core Router Accelerated Life Testing I-Ds (Poretsky, 15 min) Milestones Status (Chairs, 5 min) Revised Charter Discussion (Chairs, 15 min) New Work (Chairs, Poretsky, 15 min)

3. 3 BMWG Activity AD/IESG Review this space is blank for now WG Last Call, WGLC ended 10/10 LC ended 10/26, begin WGC shepherding will proceed with Meth I-Ds new WGLC requested. Ready for WGLC?

4. 4 BMWG Activity I-Ds, expected soon?

5. 5 BMWG Activity Expired BMWG I-Ds –, No Prog – Dead… – Pending term New Work proposals. –LDP Convergence – expired, but… –draft-poretsky-protection-term-00.txt (adopt?) –draft-poretsky-mpls-protection-meth-04.txt (adopt?) RFC Editor – none. Charter Update –Need more discussion. New RFC:

6. 6 Revised Milestones Fmr Past Due Milestones Dec 05 IPsec Device Benchmarking Terminology to AD Review Apr 06 Net Traffic Control Benchmarking Terminology to AD Review Apr 06 Methodology for FIB related Router Perf. Bench. to AD review. Apr 06 IGP/Data-Plane Terminology I-D to AD Review Apr 06 IGP/Data-Plane Methodology and Considerations I-Ds to AD Review Apr 06 Router Accelerated Test Terminology I-D to AD Review Jul 06 Basic BGP Convergence Benchmarking Methodology to AD Review. On Track, but running late Feb 06 Router Accelerated Test Method. and Consid. I-Ds to AD Review Feb 06 Hash and Stuffing I-D to AD Review Dec 05 Net Traffic Control Benchmarking Methodology to AD Review (no change) New Apr 06 Router Accelerated Test Method for EBGP to AD Review Apr 06 Router Accelerated Test Method for Operational Security to AD Review Dec 05 IPsec Device Benchmarking Methodology to AD Review

7. 7 Charter Text (Goal/Scope) Current goal is very broad: …to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

8. 8 Charter Text (continued) Should we tighten this coverage? …to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies, primarily those developed by the IETF; Could also say “preferably” or “exclusively” or ??

9. 9 What BMWG benchmarks are not, or cannot be (much discussion last time) Conformance tests –IETF consensus; avoids legal entanglements (?) –Benchmarking methodology assumes compliance, and the procedures should make it difficult to cheat. Reliability tests? –How well do these tests fall under our benchmark umbrella? Acceptance tests –Implication of success criteria must be avoided Operational Network tests (not controversial) –Division of Labor between other organizations.

10. 10 Backup

11. 11 Charter Text Central Tenet –Notion of a “Benchmark” BMWG Benchmark: –A prescribed measurement that’s a basis for performance evaluation AND comparison.

12. 12 Benchmark: Required Attributes? Relevant and meaningful. Unambiguous specification of an attribute (or attributes) being examined, its collection and reporting. Universal applicability within class. (No bias toward or against a specific implementation.) Repeatability.