1. Richard Rosenfeld MD, MPH
Why Quality and Confidence in Guidelines are Important What Makes a Guideline Trustworthy?
Richard Rosenfeld MD, MPH
Distinguished Professor & Chairman of Otolaryngology, SUNY Downstate Senior Advisor for Guidelines and Quality, AAO-HNS Foundation Founder & Chair Emeritus, Guidelines Int’l Network North America

2. Brooklyn has 2.6 million people
130 languages spoken
SUNY Downstate is in geographic center
Only academic medical center in borough

3. American Academy of Otolaryngology – Head and Neck Surgery
The American Academy of Otolaryngology—Head and Neck Surgery (AAO-HNS) is the world's largest organization representing specialists who treat the ear, nose, throat, and related structures of the head and neck.
The Academy represents more than 12,000 otolaryngologist – head and neck surgeons who diagnose and treat disorders of those areas.
Headquarters in Alexandria, VA

4. Why are quality and confidence in guidelines so important?
Because without trustworthy methods, recommendations may not be valid, useful, or generalizable

5. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 1. Establishing Transparency
1.1 The processes by which a clinical practice guideline is developed and funded should be detailed explicitly and publicly accessible.

6. Clinical Practice Guideline Development Manual: Third Edition
Rosenfeld, Shiffman, and Robertson
Pragmatic, transparent approach to creating guidelines for performance assessment
Evidence-based, multidisciplinary process leading to publication in months
Emphasizes a focused set of key action statements to promote quality improvement
Uses action statement profiles to summarize decisions in recommendations
Otolaryngol Head Neck Surg 2013; 148(Suppl):S1-55

7. AAO-HNS CPG Manual, 3rd ed
AAO-HNS CPG Manual, 3rd ed. Otolaryngol Head Neck Surg 2013; 148(Suppl):S3

8. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 2. Conflict of Interest
2.1 Guideline development group (GDG) members should declare all interests
2.2 All COIs should be disclosed and discussed within the GDG
2.4 Chair should not have a COI, only a minority of members should have COI, funders have no role in CPG development

9. Development Group Conflicts of Interest
How might potential conflicts of interest impact scope and focus?

10. Guidelines and Conflict of Interest
Manage when assembling group and writing action statements
Disclose
Declare
Declaration is the start, not the end
Disclosure is important, but not enough
Processes are needed for managing conflicts during development

11. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 3. Guideline Development Group (GDG) Composition
3.1 The GDG should be multidisciplinary and balanced, comprising a variety of methodological experts and clinicians, and populations expected to be affected by the guideline.

12. Multidisciplinary Guideline Panels
Why Bother to Diversify?
Increases the probability that all relevant scientific evidence will be located and critically evaluated
Increases the chances that the panel will address practical problems relating to application of the guidelines
Helps build support among the groups for whom the guideline is intended
May produce more reliable results by balancing biases on the panel
Shekelle et al. Clinical guidelines: developing guidelines. BMJ 1999; 318:593-6 Institute of Medicine. Clinical practice guidelines. Washington DC: Nat’l Academy Press, 1990

13. AAO-HNS Tympanostomy Tube Guideline
Guideline Development Group
ENT physicians
1 chair
2 asst. chairs
1 consultant
2 otologists
1 BOD officer
1 BOG officer
1 resident
Other physicians
1 pediatrician (general)
1 pediatrician (child behavior)
1 family physician
1 anesthesiologist
Non-physicians
1 audiologist
1 speech LP
1 nurse
1 PA
2 consumers
1 staff liaison
Rosenfeld et al, Otolaryngol Head Neck Surg 2013; 149(Suppl): S1-S35

14. Begin with the End in Mind
Habit #2, Stephen Covey
Members of the guideline development group do not have to all be content experts!

15. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 3. Guideline Development Group (GDG) Composition
3.2 Patient and public involvement should be facilitated by including (at least at the time of clinical question formulation and draft CPG review)
a current or former patient,
and a patient advocate or patient/consumer organization representative in the GDG

16. What are the Possibilities? What do Consumers contribute to GDGs?
Consumer Involvement in Guidelines
What are the Possibilities?
What do Consumers contribute to GDGs?
Passion
Perspective
Skepticism
Respect for harms
Patient education
Shared decisions

17. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 4. Systematic Reviews
4.1 CPG developers should use systematic reviews that meet IOM standards.
4.2 When reviews are conducted specifically to inform particular guidelines, the GDG and systematic review team should interact regarding the scope, approach, and output of both processes.

19. AAO-HNS Adult Sinusitis Guideline Update
Overview of Relevant Systematic Reviews
Cochrane Reviews (interventions)
Antibiotics for acute maxillary sinusitis in adults, 2014
Antibiotics for clinically diagnosed acute rhinosinusitis in adults, 2012
Systemic corticosteroids for acute sinusitis, 2011
Systemic antibiotics for chronic rhinosinusitis (CRS), 2011
Nasal saline irrigations for the symptoms of CRS, 2007
Topical steroids for CRS without polyps, 2011
Topical steroids for nasal polyps, 2012
Topical and systemic antifungal therapy for CRS, 2011
Functional endoscopic sinus surgery for CRS, 2009
Functional endoscopic balloon dilation for CRS, 2011
Rapid Systematic Reviews
8 total: 3 diagnosis, 3 interventions, 2 prognosis
Other Systematic Reviews
24 total: 2 diagnosis, 20 interventions, 2 miscellaneous

20. Rosenfeld & Wyer, Otolaryngol Head Neck Surg 2017; ePub ahead of print

21. What Should Drive Guideline Development?
Stakeholder Needs vs. Evidence Availability
Stakeholder-Driven Quality Improvement
High-Level Evidence for Guideline Topic
Stakeholder perceptions of needs, uncertainties, and knowledge gaps
Availability of randomized trials and systematic reviews relevant to the topic

22. Rosenfeld & Wyer, Otolaryngol Head Neck Surg 2017; ePub ahead of print
Stakeholder-driven approach
Topic-driven approach
Underlying focus
Stakeholder perception of needs, knowledge gaps, and clinical uncertainties
Topic-related evidence, especially randomized trials and systematic reviews
Process driving development
Clinical relevance and pragmatism with an overriding goal of QI
Epidemiological purity and rigor of evidence selection and synthesis
Determining the guideline scope
Based on practice variation, clinical concerns, and knowledge gaps in a specific population with the most to gain from helpful guidance
Based on a need to develop guidelines for large homogeneous populations similar to those selected for randomized trials and in systematic reviews
Composing the guideline development group
Includes broad, multidisciplinary stakeholder representation, including patients and consumers; promotes capacious, unconstrained thought about real world QI needs; content experts are a minority
Includes primarily specialists and content experts who are best suited to interpret the research literature; may or may not include other disciplines, allied health providers, consumers, or patients
Prioritizing ideas for key questions
Explicit ranking of ideas by the guideline development group based on the perceived clinical gaps and the potential QI impact
Self-evident process of prioritizing ideas with the highest level of evidence and deemphasizing those with low level evidence
Identifying key questions for recommendations
Formulated to address QI needs and pressing concerns of clinicians and patients, including understudied special populations, high risk subgroups, and patients with multiple chronic conditions
Formulated to address needs of “typical” patients, based on research that may not reflect the needs of at risk groups excluded from studies because of ethical issues or desire for purity of research design
Rosenfeld & Wyer, Otolaryngol Head Neck Surg 2017; ePub ahead of print

23. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 5. Evidence Foundations
5.1 For each recommendation provide:
Clear description of benefits & harms
Quality, quantity, and consistency of the available aggregate evidence
Role of values, opinion, theory, and clinical experience in deriving the recommendation
Rating of confidence in the evidence
Rating of the strength of recommendation
Explanation of any differences of opinion

24. Anatomy of AAO-HNS Guideline Statements
Key Action Statement
Anatomy of AAO-HNS Guideline Statements
Action Statement Profile
Supporting (Amplifying) Text

25. AAO-HNS Rhinoplasty Clinical Practice Guideline
Postoperative antibiotics: When a surgeon, or a surgeon’s designee, chooses to administer perioperative antibiotics for rhinoplasty, they should not routinely prescribe antibiotic therapy for a duration of more than 24 hours after surgery Recommendation against prescribing based on randomized trials and systematic reviews with a preponderance of harm over benefit.
Evidence profile:
Quality improvement opportunity: Reduce antibiotic prescribing after rhinoplasty and promote antibiotic stewardship (NQS domain: Patient Safety)
Aggregate evidence quality: Grade B, randomized trials and systematic reviews
Level of confidence in evidence: Medium based on indirectness of evidence about benefits beyond 24 hours and absence of evidence concerning benefits of antibiotic prophylaxis
Benefits: Promote selective antibiotic use; reduce cost, adverse events, bacterial resistance
Harms: Potential for infection in patients who might have benefited from antibiotic therapy
Benefits-harm assessment: Preponderance of benefit over harm
Value judgments: Perception by the GDG that antibiotics are commonly prescribed after rhinoplasty despite a lack of evidence to consistently support benefits beyond administering a single intraoperative dose, or more than 24 hours after surgery
Intentional vagueness: The word “routine” is used to avoid setting a legal standard of care
Patient preference: Small
Exclusions: Revision surgery, complicated surgery; patients receiving nasal implants; baseline nasal colonization with MRSA; immunocomprise
Differences of opinion: None
Otolaryngol Head Neck Surg 2017

26. Action Statement Profiles and Guideline Development
Key action statement with recommendation strength and justification
Action statement profile:
QI opportunity:
Aggregate evidence quality:
Confidence in evidence:
Benefit:
Risk, harm, cost:
Benefit-harm assessment:
Value judgments:
Intentional vagueness:
Role of patient preferences:
Differences of opinion:
Exclusions:
Supporting text for key action statement
Encourage an explicit and transparent approach to guideline writing
Force guideline developers to discuss and document the decision making process
Create “organizational memory” to avoid re-discussing already agreed upon issues
Allow guideline users to rapidly understand how and why statements were developed
Facilitate identifying aspects of guideline best suited to performance assessment

27. Aggregate Grades of Evidence for AAO-HNS Guidelines
Rosenfeld, Shiffman, Robertson. Guideline Manual, 3rd ed. Otolaryngol Head Neck Surg 2013

28. Classifying Recommendations for Practice Guidelines
AAP Steering Committee on Quality Improvement and Management
Evidence Grade
Preponderance of Benefit or Harm
Balance of Benefit and Harm
A (high quality)
Strong recommendation
Option
B (moderate quality)
Recommendation or strong recommendation
C (low quality)
Recommendation
D (very low quality)
X (exceptional circumstances)
Pediatrics 2004; 114:

29. Evidence Balance and Guideline Recommendations
Best evidence suggests large benefits and quality improvement if clinicians follow the recommendation
Best evidence suggests acceptable harm, risks, and costs
Our confidence in the underlying evidence is determined by the precision, consistency, generalizability (directness), and risk of bias

30. Assessing the Level of Confidence in Aggregate Evidence
Quality concern
Comment
Study limitations (risk of bias) for randomized trials
Lack of allocation concealment
Lack of blinding: patients, investigators, assessors
Loss to follow-up: imbalanced loss among groups; lack of ITT analysis
Selective or incomplete outcome reporting
Study limitations (risk of bias) for observational studies
Problems with selection criteria; no comparison group
Flawed measurement of exposure or outcome
Failure to control for confounding
Loss to follow-up: 20% of more of sample
Imprecision
Few studies and/or small sample sizes
Main outcomes are not described using 95% CIs or wide CIs
Values at the upper or lower range of the 95% CI might change the recommendation
Inconsistency
Results vary widely across studies
Significant heterogeneity found in systematic review
Limited generalizability (indirectness)
Differences between the study population(s) and target population
Differences in interventions used by the investigator(s) compared to the guideline intervention
Differences in outcome measure(s)
AAO-HNS Guideline Manual, 3rd Edition (based on GRADE)

31. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 6. Articulation of Recommendations
6.1 Recommendations should be articulated in a standardized form detailing precisely what the recommended action is, and under what circumstances it should be performed
6.2 Strong recommendations should be worded so that compliance with the recommendations can be evaluated.

32. Seidman et al, Otolaryngol Head Neck Surg 2015; 152 (Suppl)
Key Action Statements on Allergic Rhinitis (Intermittent and Persistent)
Allergic rhinitis is an inflammatory, IgE-mediated disease characterized by nasal congestion, nasal drainage, sneezing, and/or nasal itching
Clinicians should recommend intranasal steroids for patients with a clinical diagnosis of allergic rhinitis whose symptoms affect their quality of life.
Clinicians should recommend oral second-generation non-sedating antihistamines for patients with allergic rhinitis and primary complaints of sneezing and itching.
Clinicians may offer intranasal antihistamines for patients with seasonal, perennial, or episodic allergic rhinitis.
Clinicians may offer (or refer to a clinician who can offer) acupuncture for patients with allergic rhinitis who are interested in nonpharmacologic therapy.
Clinicians should not offer oral leukotriene receptor antagonists as primary therapy for patient with allergic rhinitis.
Seidman et al, Otolaryngol Head Neck Surg 2015; 152 (Suppl)

33. Building Better Guidelines with BRIDGE-Wiz
Shiffman and Rosenfeld et al, JAMIA 2012
Description of a software assistant for structured action statement creation to promote clarity, transparency and implementability
Choose an action type
Choose a verb
Define the object for the verb
Add actions
Check executability
Define conditions for the action
Check decidability
Describe benefits, risks, harms & costs
Judge the benefit-harms balance
Select aggregate evidence quality
Review proposed strength of recommendation and level of obligation
Define the actor
Choose recommendation style
Edit the final statement
J Am Med Inform Assoc 2002; 19:

34. Standards for Developing Trustworthy Clinical Practice Guidelines
Standard 7. External Review
7.1 External reviewers should comprise a full spectrum of relevant stakeholders, including scientific and clinical experts, organizations, agencies, patients, and representatives of the public.
7.3 The GDG should consider all external reviewer comments and keep a written record of the rationale for modifying or not modifying a CPG in response to comments.

35. Guideline Review Process
Internal Review
Guideline Development Group
Guideline Implementability Analysis
External Review
Invited Multidisciplinary Comments
Open Public Comment
Peer Review
Traditional Journal Editorial Review
Post-publication Peer Review

36. External Review of Draft Guideline
Principles for Rapid Guideline Development
Identify reviewers through organizational outreach; must include all relevant stakeholders
Reviewers complete COI and confidentiality forms; informed in advance of strict date for comment submission
Ensure understanding of the process: comment disposition, differences from standard peer-review, feedback to reviewers
Staff collate and organize all comments into a master grid with reviewer names removed
Chair and assistant chairs triage comments with group review

37. Guideline Adherence to IOM Standards
Kung et al, Arch Int Med 2012
Two reviewers independently screened 130 guidelines selected at random from the National Guideline Clearinghouse for compliance with IOM standards
Arch Int Med 2012; 172:

38. Ann Intern Med 2012; 156:

39. Quality in Healthcare Guidelines Defining Quality
Registries Tracking Quality
Measures Assessing Quality
Quality Improvement

40. Richard Rosenfeld MD, MPH
Why Quality and Confidence in Guidelines are Important What Makes a Guideline Trustworthy?
Richard Rosenfeld MD, MPH
Distinguished Professor & Chairman of Otolaryngology, SUNY Downstate Senior Advisor for Guidelines and Quality, AAO-HNS Foundation Founder & Chair Emeritus, Guidelines Int’l Network North America