Application Performance Metrics APM BOF July 25, 2007 Alan Clark Al Morton IETF 69 – Chicago – July 2007

Agenda Note-Taker(s), Jabber, IPR (Yellow Sheet), Blue Sheets Note-Taker(s), Jabber, IPR (Yellow Sheet), Blue Sheets Problem Statement and Goals Problem Statement and Goals Potential Solutions – WG or Directorate or… Potential Solutions – WG or Directorate or… Gap Analysis & Existing Drafts Gap Analysis & Existing Drafts Pro’s and Con’s / Discussion Pro’s and Con’s / Discussion Consensus? Consensus?

Problem Statement and Goal Application performance measurement/ metrics is a specialized topic Application performance measurement/ metrics is a specialized topic WG’s may have application but not performance measurement expertise WG’s may have application but not performance measurement expertise Performance metric drafts tend to get less attention than application oriented drafts Performance metric drafts tend to get less attention than application oriented drafts No WG or Directorate focused on Application Performance Measurement No WG or Directorate focused on Application Performance Measurement GOAL: explore the need for a new WG and/or Directorate with the IETF community and capture the consensus (and solution direction) GOAL: explore the need for a new WG and/or Directorate with the IETF community and capture the consensus (and solution direction)

Proposed WG and or Directorate APM Directorate APM Directorate Directorate Role: Advise and Review work in other WGs and write BCP Directorate Role: Advise and Review work in other WGs and write BCP APM Working Group APM Working Group WG Role: Prepare RFCs in Coordination with other WG and write BCP WG Role: Prepare RFCs in Coordination with other WG and write BCP Short-lived APM WG – possibly evolving to one of above Short-lived APM WG – possibly evolving to one of above Writes a BCP or Framework RFC Writes a BCP or Framework RFC

Constraints Do Do Develop in cooperation with relevant WG’s Develop in cooperation with relevant WG’s Drafts related to IP based applications, with particular focus on IETF applications Drafts related to IP based applications, with particular focus on IETF applications Drafts related to transport protocol performance, with particular focus on reliable transport Drafts related to transport protocol performance, with particular focus on reliable transport Cooperate with other standards organizations working in related areas Cooperate with other standards organizations working in related areas

Constraints Don’t Don’t Develop metrics in areas where other standards organizations have established expertise Develop metrics in areas where other standards organizations have established expertise Examples: Examples: Voice, Audio, Video perceptual quality (expertise of ITU-T SG12, SG9, VQEG) Voice, Audio, Video perceptual quality (expertise of ITU-T SG12, SG9, VQEG) IP performance metrics (expertise of IPPM and ITU-T SG12) IP performance metrics (expertise of IPPM and ITU-T SG12)

Existing drafts draft-malas-performance-metrics-07.txt draft-malas-performance-metrics-07.txt draft-venna-ippm-app-loss-metrics-00.txt draft-venna-ippm-app-loss-metrics-00.txt draft-ietf-avt-rtcpxr-video-01.txt draft-ietf-avt-rtcpxr-video-01.txt draft-ietf-avt-rtcpxr-audio-00.txt draft-ietf-avt-rtcpxr-audio-00.txt draft-ietf-avt-rtcpxr-mpts-00.txt draft-ietf-avt-rtcpxr-mpts-00.txt draft-ietf-avt-rtcpxr-transport-00.txt draft-ietf-avt-rtcpxr-transport-00.txt draft-ietf-avt-rtcphr-01.txt draft-ietf-avt-rtcphr-01.txt draft-xie-ccamp-lsp-dppm-01.txt draft-xie-ccamp-lsp-dppm-01.txt draft-kikuchi-passive-measure-00.txt draft-kikuchi-passive-measure-00.txt Others? Others?

How well do current WG’s handle Performance Metrics? Feedback from authors on progress made within current WG’s? Feedback from authors on progress made within current WG’s? Feedback from WG Chairs / A-D’s on good/ difficult aspects of progressing performance measurement drafts within existing WG’s? Feedback from WG Chairs / A-D’s on good/ difficult aspects of progressing performance measurement drafts within existing WG’s? Any lessons to learn? Any lessons to learn?

SIP End-to-End Performance Metrics 69 th IETF – Chicago (APM BoF) Daryl Malas

Problem Statement With widespread implementation of SIP the following problems have surfaced: With widespread implementation of SIP the following problems have surfaced: No standard method for measuring SIP performance No standard method for measuring SIP performance Industry confusion on “How” Industry confusion on “How” Industry confusion on “Where” to measure Industry confusion on “Where” to measure Current reliance on PSTN (and other) metrics Current reliance on PSTN (and other) metrics Draft defines the “How” and the “Where” for common metrics applicable to ALL SIP applications. Draft defines the “How” and the “Where” for common metrics applicable to ALL SIP applications.

Metrics Registration Request Delay (RRD) Registration Request Delay (RRD) Session Request Delay (SRD) Session Request Delay (SRD) Session Disconnect Delay (SDD) Session Disconnect Delay (SDD) Session Duration Time (SDT) Session Duration Time (SDT) Average Hops per Request (AHR) Average Hops per Request (AHR) Session Establishment Rate (SER) Session Establishment Rate (SER) Session Establishment Efficiency Rate (SEER) Session Establishment Efficiency Rate (SEER) Session Defects (SD) Session Defects (SD) Ineffective Session Attempts (ISA) Ineffective Session Attempts (ISA) Session Disconnect Failures (SDF) Session Disconnect Failures (SDF) Session Completion Rate (SCR) Session Completion Rate (SCR) Session Success Rate (SSR) Session Success Rate (SSR)

RTCP XR Video Metrics RTCP XR – Video Metrics RTCP XR – Video Metrics RTCP XR – Audio Metrics RTCP XR – Audio Metrics RTCP XR – MPEG Transport Metrics RTCP XR – MPEG Transport Metrics RTCP XR – Transport Metrics RTCP XR – Transport Metrics Set of RTCP XR report blocks for IPTV (and potentially other IP Video) performance reporting Set of RTCP XR report blocks for IPTV (and potentially other IP Video) performance reporting Incorporates packet, transport, application metrics Incorporates packet, transport, application metrics

RTCP HR RTCP HR – intended use – carrier backbone VoIP services RTCP HR – intended use – carrier backbone VoIP services Useful in VoIP services with multiple interconnected segments (e.g. transcoding gateways) Useful in VoIP services with multiple interconnected segments (e.g. transcoding gateways) Reports range of packet loss distribution metrics, signal related metrics etc. Reports range of packet loss distribution metrics, signal related metrics etc. Strong ties to work in ITU-T.. H.248 Strong ties to work in ITU-T.. H.248

Perf. Metric Proposals Application Loss Pattern Metrics Application Loss Pattern Metrics Loss Pattern metrics to infer App. Performance and derive Errored Seconds. Loss Pattern metrics to infer App. Performance and derive Errored Seconds. LSP Dynamical Provisioning Performance Metrics in GMPLS LSP Dynamical Provisioning Performance Metrics in GMPLS LSP Setup (Uni/Bi-directional) and Graceful Rel. LSP Setup (Uni/Bi-directional) and Graceful Rel. Passive Measurement of e2e Quality Passive Measurement of e2e Quality Exploits a Seq. Num. Field to measure loss, duplicate, and reordering. Exploits a Seq. Num. Field to measure loss, duplicate, and reordering.

Other related activities BMWG – Defines Performance Metrics related to IP- based networking up and down the stack BMWG – Defines Performance Metrics related to IP- based networking up and down the stack Restricted to Lab Characterization Restricted to Lab Characterization IPPM – Active Performance Characterization of Live Networks at IP layer (TCP also addressed) IPPM – Active Performance Characterization of Live Networks at IP layer (TCP also addressed) No Passive Traffic Monitoring No Passive Traffic Monitoring OPSAWG - BCP guidelines for authors/reviewers of new IETF protocols for operational and manageability requirements + other small OPS projects OPSAWG - BCP guidelines for authors/reviewers of new IETF protocols for operational and manageability requirements + other small OPS projects RMON – concluded, with many metrics referenced for remote monitoring purposes RMON – concluded, with many metrics referenced for remote monitoring purposes

Proposal 1 – new APM Directorate Responsibilities: Advise protocol WGs initiating and developing APMs Responsibilities: Advise protocol WGs initiating and developing APMs correctness of metrics definitions correctness of metrics definitions Measurement and Reporting Methodologies Measurement and Reporting Methodologies Assist with Coordination with other Stds Orgs Assist with Coordination with other Stds Orgs Mode of operation Mode of operation Consult, review, and provide point of reference Consult, review, and provide point of reference Drafts would be chartered in the protocol WG. Drafts would be chartered in the protocol WG. Directorate would prepare a BCP/framework RFC Directorate would prepare a BCP/framework RFC

Proposal 2 – new APM WG Responsibilities: Responsibilities: Development of RFCs that characterize the performance of Layers above IP, especially those utilizing IETF protocols Development of RFCs that characterize the performance of Layers above IP, especially those utilizing IETF protocols Advance those RFCs along the Standards Track. Advance those RFCs along the Standards Track. Coordination with other Stds Orgs Coordination with other Stds Orgs Prepare a BCP/Framework RFC Prepare a BCP/Framework RFC Mode of Operation: Mode of Operation: Partner with a specific protocol development WG whenever possible. Partner with a specific protocol development WG whenever possible. Take on work ONLY with agreement from relevant protocol WG and IESG. Take on work ONLY with agreement from relevant protocol WG and IESG.

Proposal 3 – Short-Lived APM WG Responsibilities Responsibilities Prepare a BCP/Framework RFC Prepare a BCP/Framework RFC Act in the Role of the Directorate while in existence Act in the Role of the Directorate while in existence Then, evolve to Directorate or WG after completing the charter. Then, evolve to Directorate or WG after completing the charter.

Pro’s and Con’s APM Directorate APM Directorate Pro Pro Con Con APM WG APM WG Pro Pro Con Con BCP BCP Pro Pro Con Con

Key questions – 1 Should IETF commit its resources to the formal development of application performance metrics in one or more key areas? Should IETF commit its resources to the formal development of application performance metrics in one or more key areas?

Key questions – Preferred Direction 1. Protocol Development WGs with advice from an APM Directorate 2. APM WG with participation by experts from the relevant Protocol WG. 3. Short-lived WG to prepare Framework RFC/BCP, evolving to Dir or WG… 4. Neither an APM Directorate or an APM WG are needed

Key questions – 3 If the APM Directorate approach is desirable, is the proposed method of working acceptable, or are modifications needed? If the APM Directorate approach is desirable, is the proposed method of working acceptable, or are modifications needed?

Key questions – 4 If the (short-lived) APM WG approach is desirable, is the method of working acceptable, or are modifications needed? If the (short-lived) APM WG approach is desirable, is the method of working acceptable, or are modifications needed?

Summary Conclusions from this BOF? Conclusions from this BOF? Next steps Next steps

Contact info Alan Clark – Al Morton –