Application Performance Metrics APM BOF July 25, 2007 Alan Clark Al Morton IETF 69 – Chicago – July 2007.

Slides:



Advertisements
Similar presentations
IPPM WG. Note Well Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and any statement.
Advertisements

1 ISMS WG 79th IETF Beijing November 10, 2010 Goal:Creating a security model for SNMPv3 that will meet the security and operational needs of network administrators.
1 SIP End-to-End Performance Metrics (draft-ietf-pmol-sip-perf-metrics-00.txt)draft-ietf-pmol-sip-perf-metrics-00.txt 71 st IETF Conference PMOL Daryl.
Russ Housley IETF Chair 23 July 2012 Introduction to the IETF Standards Process.
1 Benchmarking Methodology WG (bmwg) 84th IETF Thursday, August 2, 2012 (1730 Vancouver Local Time, GMT-7:00) Chairs: –Al Morton If.
1 Benchmarking Methodology WG (bmwg) Virtual Interim Meeting prior to 76th IETF Friday, October 30, 2009, GMT Chairs: –Al Morton
SIP Performance Metrics 66 th IETF – Montreal Daryl Malas.
SIP End-to-End Performance Metrics draft-malas-performance-metrics-06.txt.
December 5, 2003FG3 Report FOCUS GROUP 3 Interoperability Report to NRIC VI Council December 5, 2003 Cliff Naughton (Boeing)
WG RAQMON Internet-Drafts RMON MIB WG Meeting Washington, Nov. 11, 2004.
MPTCP – Multipath TCP WG Meeting Toronto, IETF-90, 21 st July 2014 Philip Eardley Yoshifumi Nishida 1.
Framework for Performance Metric Development draft-morton-perf-metrics-framework-01.txt Alan Clark IETF 70 PMOL WG.
1 Proposal for BENCHMARKING SIP NETWORKING DEVICES draft-poretsky-sip-bench-term-01.txt draft-poretsky-sip-bench-meth-00.txt Co-authors are Scott Poretsky.
Discussion on IEEE metrics guidelines Document Number: IEEE R0 Date Submitted: Source: Antonio BovoVoice:
1 Benchmarking Methodology WG (bmwg) 82nd IETF Monday, November 14, 2011, (Taipei Local Time, GMT+8:00) Chairs: –Al Morton
Assuring Performance of Carrier-Class Networks and Enterprise Contact Centers SP-11: Ensuring Service Quality While Increasing Revenue February 4, 2009.
CCAMP Working Group Online Agenda and Slides at: Data tracker:
DIME WG IETF 82 Dime WG Agenda & Status THURSDAY, November 17, 2011 Jouni Korhonen & Lionel Morand.
MPTCP – MULTIPATH TCP Interim meeting #3 20 th October 2011 audio Yoshifumi Nishida Philip Eardley.
1 draft-ietf-ippm-loss-episode-metrics-00 Loss Episode Metrics for IPPM Nick Duffield, Al Morton, AT&T Joel Sommers, Colgate University IETF 79, Beijing,
Mary Barnes (WG co-chair) Cullen Jennings (WG co-chair) DISPATCH WG IETF 89.
1 Benchmarking Methodology WG (bmwg) 85th IETF Tuesday, November 6, 2012 (1520 Atlanta Local Time, GMT-5:00) Chairs: –Al Morton – PLEASE.
Routing Area WG (rtgwg) IETF 86 – Orlando Chairs: Alia Atlas Alvaro Retana
Draft-johnston-sipping-rtcp-summary-01.txt RTCP Summary Report Delivery to SIP Third Parties draft-johnston-sipping-rtcp-summary-01.txt Alan Johnston –
IPPM WG IETF 79. Note Well Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and.
CONEX BoF. Welcome to CONEX! Chairs: –Leslie Daigle –Philip Eardley Scribe Note well.
1 Benchmarking Methodology WG (bmwg) 86th IETF Tuesday, July 30, 2013 ( Berlin Local Time, GMT+2:00) Chairs: –Al Morton (acmorton(at)att.com)
IETF#64 – 7-11 November 2005 fecframe BOF Chair:Mark Watson Mailing List:
SIP Performance Benchmarking draft-ietf-bmwg-sip-bench-term-01 draft-ietf-bmwg-sip-bench-meth-01 March 22, 2010 Prof. Carol Davids, Illinois Inst. of Tech.
IETF-90 (Toronto) DHC WG Meeting Wednesday, July 23, GMT IETF-90 DHC WG1 Last Updated: 07/21/ :10 EDT.
CSE5803 Advanced Internet Protocols and Applications (14) Introduction Developed in recent years, for low cost phone calls (long distance in particular).
Page 1 IETF Speermint Working Group Speermint Requirements/Guidelines for SIP session peering draft-ietf-speermint-requirements-02 IETF 69 - Monday July.
Delay Variation Applicability Statement draft-morton-ippm-delay-var-as-03 July 24, 2007 Al Morton Benoit Claise.
DetNet WG 1 ST Meeting Chairs: Lou Berger Pat Thaler Secretary: Jouni Korhonen.
PAWS Protocol to Access White Space DB IETF 83, Paris Gabor Bajko, Brian Rosen.
1 SIP Performance Benchmarking draft-poretsky-sip-bench-term-03.txt draft-poretsky-bmwg-sip-bench-meth-01.txt BMWG, IETF-69 Chicago July 2007 Poretsky,
1 Benchmarking Methodology WG (bmwg) 79th IETF Thursday, November 11, 2010, CST (China Standard Time GMT +8:00) Chairs: –Al Morton
July 28, 2010IETF 78 – Maastricht, Netherlands1 IP Multicast Performance Monitoring: update on IPPM experience Vero Zheng Alberto Tempia Bonda.
1 MPLS Architectural Considerations for a Transport Profile ITU-T - IETF Joint Working Team Dave Ward, Malcolm Betts, ed. April 16, 2008.
Joint CCAMP, L2VPN, MPLS & PWE3 meeting on MPLS-TP Dublin
RObust Header Compression WG (ROHC) 66 th IETF Montreal, Canada, July 11, 2006 Meeting Chair: Carsten Bormann WG Chair: Lars-Erik Jonsson.
75 th IETF, Stockholm, Sweden July 26-31, 2009 BMWG SIP Benchmarking BMWG, Monday July 27, 2009 Scott Poretsky Carol Davids Vijay K. Gurbani.
Quality of Service for Real-Time Network Management Debbie Greenstreet Product Management Director Texas Instruments.
MPTCP – MULTIPATH TCP WG meeting #1 Nov 9 th, 2009 Hiroshima, ietf-76.
CLUE WG chair: Mary Barnes RTCWEB WG chair: Ted Hardie CLUE & RTCWEB WGs Adhoc Common (SDP/RTP) building blocks IETF-82.
Interface to the Routing System (IRS) BOF IETF 85, Atlanta November 2012.
Note Well Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and any statement made.
MPTCP – MULTIPATH TCP WG meeting Tuesday 23 rd & Friday 26 th March 2010 Anaheim, ietf-77.
EDU BOF IESG Plenary – IETF57, Vienna Margaret Wasserman
MODERN BoF Managing, Ordering, Distributing, Exposing, and Registering telephone Numbers IETF 92.
1 Benchmarking Methodology WG (bmwg) 78th IETF Thursday, July 29, 2010, CET (GMT – 0:00, due to DST in Europe) Chairs: –Al Morton
Pseudowire And LDP-enabled Services (PALS) WG Status IETF-92 Dallas Co-Chairs: Stewart Bryant and Andy Malis
© 2006 Cisco Systems, Inc. All rights reserved.Cisco Public 1 OSI transport layer CCNA Exploration Semester 1 – Chapter 4.
Source Packet Routing in Networking WG (spring) IETF 89 – London Chairs: John Scudder Alvaro Retana
1 Benchmarking Methodology WG (bmwg) 95th IETF Thursday, April 7, 2016 ( Local Time, UTC-3:00) Chairs: –Al Morton (acmorton(at)att.com) –Sarah.
SIPPING Working Group IETF 67 Mary Barnes Gonzalo Camarillo.
1 SIP End-to-End Performance Metrics 70 th IETF Conference PMOL Daryl Malas.
Jim McEachern Senior Technology Consultant ATIS July 8, 2015.
Evaluation Scenarios of the CJK NGN Test-bed
Nick Duffield, Al Morton, AT&T Joel Sommers, Colgate University
15th November 2016 Gorry Fairhurst (via webrtc) David Black WG chairs
CONEX BoF.
IP Performance Specifications - Progress and Next Steps
SIP Performance Metrics
Tuesday , 9:30-12:00 Morning session I, Buckingham
Binary Floor Control Protocol BIS (BFCPBIS)
DetNet WG Chairs: Lou Berger
Tuesday (July 23rd, 2019) Two sessions ( minutes)
Joint OPS Area and OPSAWG Meeting
Agenda Wednesday, March 30, :00 – 11:30 AM
Presentation transcript:

Application Performance Metrics APM BOF July 25, 2007 Alan Clark Al Morton IETF 69 – Chicago – July 2007

Agenda Note-Taker(s), Jabber, IPR (Yellow Sheet), Blue Sheets Note-Taker(s), Jabber, IPR (Yellow Sheet), Blue Sheets Problem Statement and Goals Problem Statement and Goals Potential Solutions – WG or Directorate or… Potential Solutions – WG or Directorate or… Gap Analysis & Existing Drafts Gap Analysis & Existing Drafts Pro’s and Con’s / Discussion Pro’s and Con’s / Discussion Consensus? Consensus?

Problem Statement and Goal Application performance measurement/ metrics is a specialized topic Application performance measurement/ metrics is a specialized topic WG’s may have application but not performance measurement expertise WG’s may have application but not performance measurement expertise Performance metric drafts tend to get less attention than application oriented drafts Performance metric drafts tend to get less attention than application oriented drafts No WG or Directorate focused on Application Performance Measurement No WG or Directorate focused on Application Performance Measurement GOAL: explore the need for a new WG and/or Directorate with the IETF community and capture the consensus (and solution direction) GOAL: explore the need for a new WG and/or Directorate with the IETF community and capture the consensus (and solution direction)

Proposed WG and or Directorate APM Directorate APM Directorate Directorate Role: Advise and Review work in other WGs and write BCP Directorate Role: Advise and Review work in other WGs and write BCP APM Working Group APM Working Group WG Role: Prepare RFCs in Coordination with other WG and write BCP WG Role: Prepare RFCs in Coordination with other WG and write BCP Short-lived APM WG – possibly evolving to one of above Short-lived APM WG – possibly evolving to one of above Writes a BCP or Framework RFC Writes a BCP or Framework RFC

Constraints Do Do Develop in cooperation with relevant WG’s Develop in cooperation with relevant WG’s Drafts related to IP based applications, with particular focus on IETF applications Drafts related to IP based applications, with particular focus on IETF applications Drafts related to transport protocol performance, with particular focus on reliable transport Drafts related to transport protocol performance, with particular focus on reliable transport Cooperate with other standards organizations working in related areas Cooperate with other standards organizations working in related areas

Constraints Don’t Don’t Develop metrics in areas where other standards organizations have established expertise Develop metrics in areas where other standards organizations have established expertise Examples: Examples: Voice, Audio, Video perceptual quality (expertise of ITU-T SG12, SG9, VQEG) Voice, Audio, Video perceptual quality (expertise of ITU-T SG12, SG9, VQEG) IP performance metrics (expertise of IPPM and ITU-T SG12) IP performance metrics (expertise of IPPM and ITU-T SG12)

Existing drafts draft-malas-performance-metrics-07.txt draft-malas-performance-metrics-07.txt draft-venna-ippm-app-loss-metrics-00.txt draft-venna-ippm-app-loss-metrics-00.txt draft-ietf-avt-rtcpxr-video-01.txt draft-ietf-avt-rtcpxr-video-01.txt draft-ietf-avt-rtcpxr-audio-00.txt draft-ietf-avt-rtcpxr-audio-00.txt draft-ietf-avt-rtcpxr-mpts-00.txt draft-ietf-avt-rtcpxr-mpts-00.txt draft-ietf-avt-rtcpxr-transport-00.txt draft-ietf-avt-rtcpxr-transport-00.txt draft-ietf-avt-rtcphr-01.txt draft-ietf-avt-rtcphr-01.txt draft-xie-ccamp-lsp-dppm-01.txt draft-xie-ccamp-lsp-dppm-01.txt draft-kikuchi-passive-measure-00.txt draft-kikuchi-passive-measure-00.txt Others? Others?

How well do current WG’s handle Performance Metrics? Feedback from authors on progress made within current WG’s? Feedback from authors on progress made within current WG’s? Feedback from WG Chairs / A-D’s on good/ difficult aspects of progressing performance measurement drafts within existing WG’s? Feedback from WG Chairs / A-D’s on good/ difficult aspects of progressing performance measurement drafts within existing WG’s? Any lessons to learn? Any lessons to learn?

SIP End-to-End Performance Metrics 69 th IETF – Chicago (APM BoF) Daryl Malas

Problem Statement With widespread implementation of SIP the following problems have surfaced: With widespread implementation of SIP the following problems have surfaced: No standard method for measuring SIP performance No standard method for measuring SIP performance Industry confusion on “How” Industry confusion on “How” Industry confusion on “Where” to measure Industry confusion on “Where” to measure Current reliance on PSTN (and other) metrics Current reliance on PSTN (and other) metrics Draft defines the “How” and the “Where” for common metrics applicable to ALL SIP applications. Draft defines the “How” and the “Where” for common metrics applicable to ALL SIP applications.

Metrics Registration Request Delay (RRD) Registration Request Delay (RRD) Session Request Delay (SRD) Session Request Delay (SRD) Session Disconnect Delay (SDD) Session Disconnect Delay (SDD) Session Duration Time (SDT) Session Duration Time (SDT) Average Hops per Request (AHR) Average Hops per Request (AHR) Session Establishment Rate (SER) Session Establishment Rate (SER) Session Establishment Efficiency Rate (SEER) Session Establishment Efficiency Rate (SEER) Session Defects (SD) Session Defects (SD) Ineffective Session Attempts (ISA) Ineffective Session Attempts (ISA) Session Disconnect Failures (SDF) Session Disconnect Failures (SDF) Session Completion Rate (SCR) Session Completion Rate (SCR) Session Success Rate (SSR) Session Success Rate (SSR)

RTCP XR Video Metrics RTCP XR – Video Metrics RTCP XR – Video Metrics RTCP XR – Audio Metrics RTCP XR – Audio Metrics RTCP XR – MPEG Transport Metrics RTCP XR – MPEG Transport Metrics RTCP XR – Transport Metrics RTCP XR – Transport Metrics Set of RTCP XR report blocks for IPTV (and potentially other IP Video) performance reporting Set of RTCP XR report blocks for IPTV (and potentially other IP Video) performance reporting Incorporates packet, transport, application metrics Incorporates packet, transport, application metrics

RTCP HR RTCP HR – intended use – carrier backbone VoIP services RTCP HR – intended use – carrier backbone VoIP services Useful in VoIP services with multiple interconnected segments (e.g. transcoding gateways) Useful in VoIP services with multiple interconnected segments (e.g. transcoding gateways) Reports range of packet loss distribution metrics, signal related metrics etc. Reports range of packet loss distribution metrics, signal related metrics etc. Strong ties to work in ITU-T.. H.248 Strong ties to work in ITU-T.. H.248

Perf. Metric Proposals Application Loss Pattern Metrics Application Loss Pattern Metrics Loss Pattern metrics to infer App. Performance and derive Errored Seconds. Loss Pattern metrics to infer App. Performance and derive Errored Seconds. LSP Dynamical Provisioning Performance Metrics in GMPLS LSP Dynamical Provisioning Performance Metrics in GMPLS LSP Setup (Uni/Bi-directional) and Graceful Rel. LSP Setup (Uni/Bi-directional) and Graceful Rel. Passive Measurement of e2e Quality Passive Measurement of e2e Quality Exploits a Seq. Num. Field to measure loss, duplicate, and reordering. Exploits a Seq. Num. Field to measure loss, duplicate, and reordering.

Other related activities BMWG – Defines Performance Metrics related to IP- based networking up and down the stack BMWG – Defines Performance Metrics related to IP- based networking up and down the stack Restricted to Lab Characterization Restricted to Lab Characterization IPPM – Active Performance Characterization of Live Networks at IP layer (TCP also addressed) IPPM – Active Performance Characterization of Live Networks at IP layer (TCP also addressed) No Passive Traffic Monitoring No Passive Traffic Monitoring OPSAWG - BCP guidelines for authors/reviewers of new IETF protocols for operational and manageability requirements + other small OPS projects OPSAWG - BCP guidelines for authors/reviewers of new IETF protocols for operational and manageability requirements + other small OPS projects RMON – concluded, with many metrics referenced for remote monitoring purposes RMON – concluded, with many metrics referenced for remote monitoring purposes

Proposal 1 – new APM Directorate Responsibilities: Advise protocol WGs initiating and developing APMs Responsibilities: Advise protocol WGs initiating and developing APMs correctness of metrics definitions correctness of metrics definitions Measurement and Reporting Methodologies Measurement and Reporting Methodologies Assist with Coordination with other Stds Orgs Assist with Coordination with other Stds Orgs Mode of operation Mode of operation Consult, review, and provide point of reference Consult, review, and provide point of reference Drafts would be chartered in the protocol WG. Drafts would be chartered in the protocol WG. Directorate would prepare a BCP/framework RFC Directorate would prepare a BCP/framework RFC

Proposal 2 – new APM WG Responsibilities: Responsibilities: Development of RFCs that characterize the performance of Layers above IP, especially those utilizing IETF protocols Development of RFCs that characterize the performance of Layers above IP, especially those utilizing IETF protocols Advance those RFCs along the Standards Track. Advance those RFCs along the Standards Track. Coordination with other Stds Orgs Coordination with other Stds Orgs Prepare a BCP/Framework RFC Prepare a BCP/Framework RFC Mode of Operation: Mode of Operation: Partner with a specific protocol development WG whenever possible. Partner with a specific protocol development WG whenever possible. Take on work ONLY with agreement from relevant protocol WG and IESG. Take on work ONLY with agreement from relevant protocol WG and IESG.

Proposal 3 – Short-Lived APM WG Responsibilities Responsibilities Prepare a BCP/Framework RFC Prepare a BCP/Framework RFC Act in the Role of the Directorate while in existence Act in the Role of the Directorate while in existence Then, evolve to Directorate or WG after completing the charter. Then, evolve to Directorate or WG after completing the charter.

Pro’s and Con’s APM Directorate APM Directorate Pro Pro Con Con APM WG APM WG Pro Pro Con Con BCP BCP Pro Pro Con Con

Key questions – 1 Should IETF commit its resources to the formal development of application performance metrics in one or more key areas? Should IETF commit its resources to the formal development of application performance metrics in one or more key areas?

Key questions – Preferred Direction 1. Protocol Development WGs with advice from an APM Directorate 2. APM WG with participation by experts from the relevant Protocol WG. 3. Short-lived WG to prepare Framework RFC/BCP, evolving to Dir or WG… 4. Neither an APM Directorate or an APM WG are needed

Key questions – 3 If the APM Directorate approach is desirable, is the proposed method of working acceptable, or are modifications needed? If the APM Directorate approach is desirable, is the proposed method of working acceptable, or are modifications needed?

Key questions – 4 If the (short-lived) APM WG approach is desirable, is the method of working acceptable, or are modifications needed? If the (short-lived) APM WG approach is desirable, is the method of working acceptable, or are modifications needed?

Summary Conclusions from this BOF? Conclusions from this BOF? Next steps Next steps

Contact info Alan Clark – Al Morton –