1. GRADE Grading of Recommendations Assessment, Development and Evaluation British Association of Dermatologists April 2014

2. Previous grading system Level of evidence *Studies with a level of evidence ‘-’ should not be used as a basis for making a recommendation. Strength of recommendation RCT: randomised controlled trial; NICE: National Institute for Health and Care Excellence. ClassEvidence A  At least one meta-analysis, systematic review, or RCT rated as 1++, and directly applicable to the target population, or  A systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population and demonstrating overall consistency of results  Evidence drawn from a NICE technology appraisal B  A body of evidence including studies rated as 2++, directly applicable to the target population and demonstrating overall consistency of results, or  Extrapolated evidence from studies rated as 1++ or 1+ C  A body of evidence including studies rated as 2+, directly applicable to the target population and demonstrating overall consistency of results, or  Extrapolated evidence from studies rated as 2++ D  Evidence level 3 or 4, or  Extrapolated evidence from studies rated as 2+, or  Formal consensus D (GPP)  A good practice point (GPP) is a recommendation for best practice based on the experience of the guideline development group Level of evidence Type of evidence 1++ High-quality meta-analyses, systematic reviews of RCTs, or RCTs with a very low risk of bias 1+ Well-conducted meta-analyses, systematic reviews of RCTs, or RCTs with a low risk of bias 1- Meta-analyses, systematic reviews of RCTs, or RCTs with a high risk of bias* 2++ High-quality systematic reviews of case-control or cohort studies High-quality case-control or cohort studies with a very low risk of confounding, bias or chance and a high probability that the relationship is causal 2+ Well-conducted case-control or cohort studies with a low risk of confounding, bias or chance and a moderate probability that the relationship is causal 2- Case-control or cohort studies with a high risk of confounding, bias or chance and a significant risk that the relationship is not causal* 3Non-analytical studies (for example, case reports, case series) 4Expert opinion, formal consensus

3. Advantages of GRADE over other systems Developed by a widely representative group of international guideline developers Clear separation between quality of evidence and strength of recommendations Explicit evaluation of the importance of outcomes of alternative management strategies Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings Transparent process of moving from evidence to recommendations Explicit acknowledgment of values and preferences Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients and policy makers GRADE has been adopted by the WHO, Cochrane Collaboration, NICE, SIGN and nearly 50 other international organisations

4. a technique used in evidence-based practice to frame and answer a clinical question P : population/patient I : intervention C : comparator/control (if applicable) O : outcome “In an elderly man with acutely painful herpes zoster would treatment with antivirals or antivirals and corticosteroids lead to a more rapid resolution of pain?” PICO method

5. Evidence hierarchy

6. Quality of evidence Evidence: high quality (further research is very unlikely to change our confidence in the estimate of effect) moderate quality (further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate) low quality (further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate) very low quality (any estimate of effect is very uncertain) “The quality of evidence reflects the extent to which confidence in an estimate of the effect is adequate to support a particular recommendation”

7. How does GRADE classify the quality of evidence? Evidence based on RCTs begins as high quality evidence, but confidence may be decreased due to five factors: 1.study limitations 2.inconsistency of results 3.indirectness of evidence 4.imprecision 5.reporting/publication bias

8. 1. Domains for study limitations: selection bias —lack of allocation concealment or sequence generation lack of blinding attrition bias —loss of participants (dropouts; non-responders; stopping early for benefit; protocol deviators and failure to adhere to an intention-to-treat analysis) measurement bias — inaccuracy in the measurement instrument; bias in the expectations of study participants, carers or researchers outcome reporting bias —selective outcome-reporting or failure to report outcomes How does GRADE classify the quality of evidence?

9. 2. Inconsistency of results : widely differing estimates of the treatment effect across studies suggest true differences in the underlying treatment effect; variability in the results may arise from differences in the P, I, C, O heterogeneity without plausible explanation 3. Indirectness of evidence: indirect comparisons of the magnitude of effect of drug A vs. drug B when trials = drug A vs. placebo and drug B vs. placebo; differences between the P, I, C, O 4. Imprecision: studies with relatively few patients and few events – wide CI 5. Reporting/publication bias: failure to report studies or funding issue How does GRADE classify the quality of evidence?

10. For study limitations and indirectness (individually), the quality of evidence is assessed: – for each study for the particular outcome – then across all studies for the particular outcome (e.g. for meta- analyses), reporting the quality in the majority of the evidence For imprecision, inconsistency and publication bias, the quality of evidence is assessed for ALL the studies as a whole How does GRADE classify the quality of evidence?

11. Using RCTs as an example, for each outcome, obtain an overall quality rating: 0 if no problems with any of the five factors (high) -1 if problem in one factor (high  moderate) -2 if problem in two factors (high  low) -3 if problem in three factors (high  very low) How does GRADE classify the quality of evidence?

12. Observational studies start with low quality ratings; grading upwards may be warranted if: –the magnitude of the treatment effect is large or very large (large effect: RR > 2 or RR 5 or RR < 0.2) –evidence of a dose-response relationship –all plausible confounders/biases would have ordinarily decreased the magnitude of an apparent treatment effect Grading upwards is only possible if evidence has not already been downgraded (due to study limitations, inconsistency, imprecision) Upgrade to: – +1 when RR > 2 or RR < 0.5 = (low  moderate) – +2 when RR > 5 or RR < 0.2 (low  high) How does GRADE classify the quality of evidence?

13. Lower if… Quality of evidence High Moderate Low Very low Study limitations Inconsistency Indirectness Imprecision Publication bias Observational studies Study design Randomised trials Higher if… Treatment effect Dose-response relationship Confounders or biases

14. GRADE is outcome-centric Critical outcomes determine the overall quality of evidence 3 1++42- Previous system GRADE Quality: High Quality: Moderate Quality: Low Outcome #1 Outcome #2 Outcome #3 Outcome #4 Quality: Very low Defining outcomes

15. GRADE challenges guideline developers to: specify all outcomes of importance to patients differentiate outcomes that are critical for decision-making from those that are important but not critical, and those that are not important limit to a suggested maximum of 7 outcomes Defining outcomes

16. For decision-making need to know: the outcomes that GDG decides as being critical: —mortality and QoL —otherwise, outcomes most likely to affect QoL and mortality the magnitude of effect as absolute differences per outcome the overall evidence quality per outcome an evidence profile helps with decision-making For decision-making: look at the clinical importance for each outcome, and not statistical significance (need to know absolute changes, and direction) Judging clinical importance: GDG decides if the effect estimate represents a clinically important difference If it is a clinically important difference, need to indicate the direction (i.e. benefit or harm) Defining outcomes

17. Example from BAD guidelines on hidradenitis suppurativa: Quality of Life (9) Adverse effects – serious (9) Pain (8) Disease-specific physician score (6) Physician’s Global Assessment (5) Patient’s Global Assessment (5) Adverse effects – nuisance (4)

18. Factors determining the strength of recommendations: balance between desirable and undesirable effects quality of evidence variability (or uncertainty) in patient values and preferences costs (resource allocation) Desirable effects include: reduction in morbidity and mortality improvement in QoL reduction in the burden of treatment reduction in resource expenditures Undesirable effects include: adverse effects having a deleterious impact on morbidity, mortality, QoL or increase in resource expenditures How does GRADE classify the strength of recommendations?

19. strong (unconditional): desirable effects of an intervention clearly outweigh the undesirable effects (or clearly do not) high-quality methods with large, precise effect low variability or uncertainty in patient values and preferences low resource allocation weak (conditional): desirable effects not clearly greater or smaller than undesirable effects low quality evidence with imprecise estimate patient values and preferences very important high resource allocation N.B. Recommendations to use interventions in a research context may be appropriate How does GRADE classify the strength of recommendations?

20. NICE approach to recommendations: strong: (the benefits clearly outweigh the harms for most people and the intervention is likely to be cost-effective; the guideline panel believes that a vast majority of clinicians and patients would choose a particular intervention if they considered the evidence in the same way that the panel has)  OFFER weak: (there is a closer balance between benefits and harm; some patients are averse to some side effects whilst others are not)  CONSIDER “Management options associated with strong recommendations are candidates for quality criteria” “When recommendations are weak, discussing relative merits of alternative management options with patients and families may become a quality criterion” Strength of recommendation

21. Systematic review Guideline development PICOPICO Formulate question Rate importance Critical Important Critical Less important Create evidence profile Summary of findings & estimate of effect for each outcome GDG reviews the evidence profile for rating the overall quality of evidence RCT start high, observational data start low 1.Risk of bias 2.Inconsistency 3.Indirectness 4.Imprecision 5.Publication bias Grade down Grade up 1.Large effect 2.Dose response 3.Confounders Rate quality of evidence for each outcome High Moderate Low Very low Outcome Select outcomes Formulate recommendations: For or against (direction) Strong or weak (strength) By considering:  Quality of evidence  Balance benefits/harms  Values and preferences  Resource use (cost) NICE recommendations: “Offer…” “Consider…” “Do not offer…” Outcomes across studies

22. GRADE evidence profile