1. A Comprehensive Approach to Quality (COACH) IETF 57 Vienna, Austria Monday, July 14, 2003 13:00 – 15:00 http://www.drizzle.com/~aboba/IETF57/COACH/

2. Agenda Preliminaries (13:00 – 13:05) Agenda Bashing Bluesheets Minutes Introduction (13:05-13:15) Existing RFC editorial guidelines - Scott Bradner http://www.ietf.org/rfc/rfc2360.txt Quality: Overview and Framework (13:15-13:45) A Comprehensive Approach to Quality (15 minutes), Bernard Aboba http://www.ietf.org/internet-drafts/draft-loughney-coach-00.txt IETF Problem Resolution Processes (15 minutes) - Margaret Wasserman http://www.ietf.org/internet-drafts/draft-ietf-problem-process-00.txt Starting New Work (13:45-14:05) The BOF Process: A Critique (10 minutes) - Leslie Daigle IESG Overload and Quality of WGs (10 minutes) - John Klensin http://www.ietf.org/internet-drafts/draft-klensin-overload-00.txt

3. Agenda The WG Process (14:05-14:15) Decision points/milestones in the WG process (10 minutes) - Margaret Wasserman http://www.ietf.org/internet-drafts/draft-wasserman-wg-process- 00.txt http://www.ietf.org/internet-drafts/draft-wasserman-wg-process- 00.txt The Review Process (14:15-14:40) Careful Additional Review of Documents (CARD) (15 minutes) - Brian Carpenter http://www.ietf.org/internet-drafts/draft-carpenter-solution-sirs- 01.txt http://www.ietf.org/internet-drafts/draft-carpenter-solution-sirs- 01.txt The Review Process in Action: The DCCP WG (10 minutes) - Aaron Falk & Allison Mankin Summary and Discussion (14:40 – 15:00)

4. Some Groundrules A mindset. “The truth is out there” - Mulder A reading List. People who have read at least three of the documents, please sit in the first three rows. A format. Strict time limit for each topic & presentation. Q&A at the end. Expectations of Behavior. Please use the mike. Please state your name before speaking. Only one person at the mike at a time. Please do not interrupt the person at the mike. Please relinquish the mike when requested to do so by the chair. Please show courtesy to your fellow IETF participants.

5. A Comprehensive Approach to Quality (COACH) John Loughney Bernard Aboba Draft-loughney-coach-00.txt

6. Basic Principles IETF Working Groups are responsible for completing work… And are accountable for the quality of that work. If Working Groups are to improve the quality (and timeliness) of the work… It is necessary for them to plan for, and carry out, that improvement. If the IETF is to improve… It is necessary to do post-mortems on the plans.

7. A Theory Many reasons why Working Groups deliver poor quality documents. Unlikely that the plan was to deliver poor quality documents all along. More likely that the level of effort, skill and experience required is beyond the capability of working group participants. We can’t all be experts in everything. Poor quality not the result of bad intentions, but insufficient resources. Naïve optimism required to start the work… But not enough to finish it.

8. What is a WG Quality Plan? A public document no more than 5 pages long Made available on the IETF web site. Developed in concert with the Working Group charter Revised when the WG charter or schedule changes significantly. Since each WG is different, no “one size fits all” quality plan. Sharing of “best practices” is encouraged. Signed off by the AD and the WG founders Reviewed by the IETF community

9. Potential Components of a WG Quality Plan WG charter – the foundation. What the WG plans to do (documents), who will do it (chairs, editors, advisors), by when (milestones & schedule). Challenge assessment – what’s in the way (reality check) The technical issues. The dependencies. Areas of skill, knowledge or experience that are lacking. Tools plan. WG mailing list. WG web site. Issue tracking tools. Revision control systems. Document production and build environments. Review plan. Checkpoints. Required reviewer skill set. Review process. Reviewer recruitment.

10. Challenge Assessment The Charter: What the WG is trying to achieve, and the resources at its disposal. The Challenge Assessment: “what does the WG need to do in order to have a high probability of completing the work on schedule and with high quality?” Some questions Are the goals achievable? With infinite resources? With the resources available? What are the key assumptions? What are the risks? Within the core area of expertise of the WG? Outside the core area of expertise? How can the risks be mitigated?

11. The Tools Plan The technology and host for the Working Group mailing list. Anti-spam plan. Archive access. The Working Group web site. Permanence? BW limits? Document production system. Tools & templates Build environment Revision control system. Issue tracking and reporting system.

12. Issue Tracking & Reporting An important tool for: Tracking issues: encourages participants to raise issues, knowing that they won’t be “forgotten” Focusing WG discussion: encourages discussions that lead to document improvements. Measuring progress: Provides metrics for assessing current status:  Open/resolved  Accept/reject  Days to resolution Enables estimates of the task remaining

13. The Review Plan Challenge Assessment review. Review of the WG Charter Should include reviewers from outside the area of the Working Group Work item review. Review of documents before they become WG work items. Does the architecture make sense? Are there major issues lurking? Midterm assessment. Is the document on the right track? Has something important been missed? Late review. Is the document ready for WG/IETF last call? More on this later…

14. An Observation Most documents are competent in the Area in which the WG resides. Transport Area WGs understand transport. Security Area WGs understand security issues. Problems, when they occur, are generally in areas outside of the core competence of the WG Internet Area documents with security issues. Security documents with transport issues. Suggestion: Cross-Area review is an important aspect of the review plan.

15. The Post Mortem No more than three pages long. Quality assessment. An evaluation of the quality of working group output, written by the Area Director. Challenge Assessment. The unforseen challenges that the Working Group encountered. Tools assessment. An evaluation of the tools that were used and how well they worked. Reviews assessment. An assessment of the review process. Recommendations. Recommendations to future working groups looking to avoid similar problems.

16. Required Materials (What a WG might produce?) An archive of sample document build environments and templates. A website covering issue tracking and reporting tools. A document on use of tracking tools for document management, including metrics and reporting. A document on the review process, including Challenge Assessment. A website including sample quality plans and post- mortems.

17. Summary Quality doesn’t just happen, it can be planned. The WG charter provides what, who, when. The WG quality plan provides the how. The resources available need to match those required to finish the job. It isn’t what we know, it’s what we think we know that isn’t so.

18. Feedback?