Benchmarking%20Methodology%20WG%20(bmwg)%2063rd%20IETF%20

1
Benchmarking Methodology WG (bmwg)63rd IETF
Paris, France

• Tuesday, August 2, 2005, 1030-1230
• Chairs
• Kevin Dubray (kdubray_at_juniper.net)
• Al Morton (acmorton_at_att.com)
• If you are not subscribed to the BMWG mailing
  list and would like to be, put an asterisk () by
  your name when you sign the blue attendance
  sheet.  

2
BMWG Agenda

• Note-Taker(s), IPR, and Blue Sheets
• WG Status and Liaison notes (Chairs, 20 min)
• Milestones Status (Chairs, 5 min)
• Resource Reservation Terminology I-Ds. (Korn, 15
  min)
• IPsec Benchmarking Terms ( intro to methods)
  (Kaeo, 15 min)
• Core Router Accelerated Life Testing I-Ds
  (Poretsky, 15 min)
• Revised Charter Text (Chairs, 15 min)
• New Work

3
BMWG Activity

• AD/IESG Review
• this space is blank for now
• WG Last Call
• ltdraft-ietf-bmwg-benchres-term-06.txtgt, Back to
  AD review?
• ltdraft-ietf-bmwg-igp-dataplane-conv-term-07.txtgt
  Need to revise?
• ltdraft-ietf-bmwg-igp-dataplane-conv-meth-07.txtgt
  Need to revise?
• ltdraft-ietf-bmwg-igp-dataplane-conv-app-07.txtgt
  Need to revise?
• ltdraft-ietf-bmwg-dsmterm-11.txtgt, additional WGLC
  needed.
• ltdraft-ietf-bmwg-ipsec-term-06.txtgt WGLC,
  comments
• I-Ds
• ltdraft-ietf-bmwg-fib-meth-03.txtgt, No change
• ltdraft-ietf-bmwg-hash-stuffing-03.txtgt WGLC
  requested.
• ltdraft-ietf-bmwg-acc-bench-term-06.txtgt Ready for
  WGLC?
• ltdraft-ietf-bmwg-acc-bench-meth-03.txtgt Ready for
  WGLC?

4
BMWG Activity

• I-Ds
• ltdraft-ietf-bmwg-acc-bench-meth-ebgp-00.txtgt, new
• ltdraft-ietf-bmwg-acc-bench-meth-opsec-00.txtgt,
  new

5
BMWG Activity

• Expired BMWG I-Ds
• ltdraft-ietf-bmwg-bgpbas-XX.gt,
• ltdraft-ietf-bmwg-benchres-method-00.txtgt Pending
  term
• New Work proposals.
• LDP Convergence expired, but
• draft-poretsky-protection-term-00.txt (adopt?)
• draft-poretsky-mpls-protection-meth-04.txt
  (adopt?)
• RFC Editor none.
• Charter Update
• Need to adjust.
• New RFC
• RFCs 4061, 4062, 4063 on OSPF convergence
  benchmarking.
• RFC 4098 on BGP convergence terminology

6
802.11T - BMWG Liaison

• Detailed note from Tom Alexander posted to BMWG
  mail archive.
• 5 proposals for metrics methods (mostly RF)
  identified at San Francisco meeting 2 were
  accepted.
• Proposals to be integrated into a draft update.

7
Current Milestones Status

• Past Due Milestones
• Dec 04 IPsec Device Benchmarking Terminology to
  AD Review
• Apr 05 Net Traffic Control Benchmarking
  Terminology to AD Review
• Apr 05 Methodology for FIB related Router Perf.
  Bench. to AD review.
• Apr 05 IGP/Data-Plane Terminology I-D to AD
  Review
• Apr 05 IGP/Data-Plane Methodology and
  Considerations I-Ds to AD Review
• Apr 05 Router Accelerated Test Terminology I-D
  to AD Review
• Jul 05 Basic BGP Convergence Benchmarking
  Methodology to AD Review.
• On Track
• Oct 05 Router Accelerated Test Method. and
  Consid. I-Ds to AD Review
• Oct 05 Hash and Stuffing I-D to AD Review
  (added!)
• Dec 05 Net Traffic Control Benchmarking
  Methodology to AD Review.
• New
• ltdraft-ietf-bmwg-acc-bench-meth-ebgp-00.txtgt,
• ltdraft-ietf-bmwg-acc-bench-meth-opsec-00.txtgt

8
Charter Text

• Central Tenet
• Notion of a Benchmark
• BMWG Benchmark
• A prescribed measurement thats a basis for
  performance evaluation AND comparison.

9
Benchmark Required Attributes?

• Relevant and meaningful.
• Unambiguous specification of an attribute (or
  attributes) being examined, its collection and
  reporting.
• Universal applicability within class. (No bias
  toward or against a specific implementation.)
• Repeatability.

10
What BMWG benchmarks are not (or cannot be)

• Conformance tests
• IESG mandated
• Acceptance tests
• Implication of success criteria
• Practical mandate
• Operational tests
• Division of Labor between other organizations.