Chicago 2014 Peter Morley, Eddy Lang E3, GRADE expert Incorporating lower levels of evidence

Chicago 2014 COI #380 Commercial/industry Evidence Evaluation Expert (E3: ILCOR/AHA) Potential intellectual conflicts Acting Chair Australian Resuscitation Council (ARC) ANZCOR delegate on International Liaison Committee on Resuscitation (ILCOR)

Chicago 2014 EBP is to consider the best available evidence

Chicago 2014 Issues EBP is to consider the best available evidence GRADE and other approaches to systematic reviews only incorporate RCTs and cohort studies

Chicago 2014 Unfortunately in Resuscitation and First Aid science RCTs are lacking Good quality non-RCTs are also lacking Sometimes only case series, case reports or animal data (previous LOE 4 and 5)

Chicago 2014 GRADE allocation of quality of evidence across each outcome RCTs start as high can be downgraded Non-RCTs start as low Can be downgraded Can occasionally be upgraded

Chicago 2014 So where do Case Series fit? Case series are observational studies Same bias assessment tools as other observational studies Recommended only to include if 4 or more cases are included in a manuscript

Chicago 2014 Bias assessment for Case Series Were appropriate eligibility criteria developed and applied to both the cohort of interest and the comparison cohort? NO Was measurement of both exposure and outcome appropriate and consistently applied to both the cohort of interest and the comparison cohort? NO Was confounding adequately controlled for? Could be YES Was follow up complete? Could be YES

Chicago 2014 Determinants of quality What lowers quality of evidence? 5 factors: Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias RCTs start high Observational studies start low Case series start very low

Chicago 2014 Still go on to complete a GRADE Evidence Profile table Enter all studies you include in your final analysis for each outcome

Chicago

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Then populate the fields using the drop-down menus No serious limitations: Most information is from studies at low risk of bias. Serious limitations: Most information is from studies at moderate risk of bias. Very serious limitations: Most information is from studies at high risk of bias.

Chicago 2014 And don’t forget to add a comment to the footnotes each time you are downgrading based on your assessment

Chicago 2014 Implications of the assessment of limitations No serious limitations: Most information is from studies at low risk of bias No downgrading of evidence Serious limitations: Most information is from studies at moderate risk of bias Evidence would be normally downgraded 1 level (ie. for RCTs from high to moderate) Very serious limitations: Most information is from studies at high risk of bias Evidence would be normally downgraded 2 levels (ie. for RCTs from high to low)

Chicago 2014 Then populate the fields using the drop-down menus Reviewers should document limitations when (1) point estimates vary widely across studies, (2) confidence intervals show minimal or no overlap (ie. studies appear to have different effects), or (3) statistical tests of heterogeneity are suggestive

Chicago 2014 Concerns about directness therefore arise when there are differences in: The Population (eg. patients in cardiac arrest vs not in cardiac arrest), Intervention (eg. standard CPR using 2010 guidelines vs standard CPR using 2005 guidelines), or Outcomes (surrogates, eg. ROSC vs termination of VF for 5 seconds), or where no head to head Comparisons between interventions

Chicago 2014 Other concerns about indirectness Evidence from animal studies, manikins or other models would generally be rated as having very serious limitations (but this would be dependent on the key outcomes listed). Interventions may be delivered/implemented differently in different settings (eg. therapeutic hypothermia). Limitations should therefore be considered if the intervention cannot be implemented with the same rigor or technical sophistication as in the studies from which the data come. Limitations in more than one type of directness may suggest a need to rate the studies as having very serious limitations.

Chicago 2014 Concerns about imprecision arise when: Cis overlap with a clinical decision threshold, eg. 1% absolute difference Trials stopped early Early publications (particularly if small) Optimal Information Size (eg. <400 patients for continuous data) Outcomes: importance, the adverse effects, the burden to the patient, resource use, and the difficulty of introducing a measure into practice

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Publication bias Reviewers should allocate strongly suspected when the evidence consists of a number of small studies, especially if these are industry sponsored or if the investigators share another conflict of interest. The risk of publication bias in observational studies is probably larger than RCTs

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Magnitude of effect for non-RCTs A large magnitude effect would be considered as justification to increase the rating by one level (eg. low to moderate) if: relative risk [RR] = 2-5 or RR with no plausible confounders; or RR very large >5 or <0.2 and no serious problems with risk of bias or precision (sufficiently narrow confidence intervals); The reviewer would be more likely to rate up if the above size of effects were rapid and out of keeping with prior trajectory; they are usually supported by indirect evidence. If above criteria are all met, and the RR is very large (eg. >5-10), rating up by two levels (from low to high) could be considered, but obviously not if other concerns were present (such as risk of bias, imprecision, inconsistency, indirectness and publication bias).

Chicago 2014 Then populate the fields using the drop-down menus

Chicago 2014 Plausible confounding for non-RCTs If: all plausible prognostic factors are accurately measured in observational studies, and if all the observed residual confounders and biases would diminish the observed effect, then the effect estimate would be strengthened. In this setting, the rating up by one level could be considered.

Chicago 2014

Dose-response gradient for non-RCTs A dose-response gradient (such as increased effect with increased dose, or decreased time to intervention) increases the confidence in the findings of observational studies. In this setting, the rating up by one level could be considered

Chicago 2014 For dichotomous (yes/no) variables, for a single study, enter number with outcome, and total number of patients. Then click “apply” Outcomes

Chicago 2014 No control group For dichotomous (yes/no) variables, for a single study, enter number with outcome, and total number of patients. Then click “apply”

Chicago 2014 No relative effects Choose desired variable: usually Relative Risk (RR): single or multiple studies if meta-analysis performed

Chicago 2014 When and how to perform a meta-analysis (interventional studies) When More than one study, of Good methodological quality, addressing Near identical individual components of PICO Population, Intervention, Comparison, Outcome Rarely with non-RCTs How Send and data for listed outcomes for studies to Peter or Eddy

Chicago 2014 GRADE evidence profile (preferred)

Chicago 2014

?

Peter Morley, Eddy Lang, Laurie Morrison Methods Committee Cost and resource implications

Chicago 2014 COI #380 Commercial/industry Evidence Evaluation Expert (E3: ILCOR/AHA) Potential intellectual conflicts Acting Chair Australian Resuscitation Council (ARC) ANZCOR delegate on International Liaison Committee on Resuscitation (ILCOR)

Chicago 2014 ILCOR Methods Committee Guidance on the Consideration of Resources in the Creation of Treatment Recommendations as part of the CoSTR

Chicago 2014 Issues the requirement of significant methodological expertise and rigour around health economic evaluations the variability in cost and resource considerations across a myriad of settings

Chicago Cost/Resource Questions Are the resources required small? Is the incremental cost small relative to the net benefits?

Chicago 2014 Overarching principles The greater the resource requirements (cost, including but not limited to equipment price and training costs), the less likely it is that an option will be a priority. Resource costs, and the implications of these to the Councils implementing the guidelines, are likely to vary substantially across jurisdictions.

Chicago st question “Are the resources required small?”

Chicago 2014 “Are the resources required small?” No* Probably no Uncertain Probably yes Yes* * Need to be supported by formal cost analysis

Chicago 2014 “Are the resources required small?” No* Probably no Uncertain Probably yes Yes* * Need to be supported by formal cost analysis

Chicago 2014 “Are the resources required small?” No supported by formal cost analysis in relevant contexts

Chicago 2014 “Are the resources required small?” Probably no Significant resources are probably required, but no formal cost analysis identified. Incremental implementation costs are significant and likely to impact on the introduction of the new therapy/diagnostic technique. These could include significant drug, equipment or diagnostic test costs, significant educational requirements, or additional significant costs for employing more staff or purchase of monitoring devices. Please elaborate on the reasons you chose this option.

Chicago 2014 “Are the resources required small?” Uncertain Resource estimates vary considerably and no reliable source of information can be identified

Chicago 2014 “Are the resources required small?” Probably yes Resources probably not altered, but no formal cost analysis identified. Incremental implementation costs are small and unlikely to impact on the introduction of the new therapy/diagnostic technique.

Chicago 2014 “Are the resources required small?” Yes supported by formal cost analysis

Chicago 2014 “Are the resources required small?” No* Probably no Uncertain Probably yes Yes* * Need to be supported by formal cost analysis

Chicago nd question “Is the incremental cost small relative to the net benefits?”

Chicago 2014 “Is the incremental cost small relative to the net benefits?” No* Probably no Uncertain Probably yes Yes* * Need to be supported by formal cost-effectiveness analysis

Chicago 2014 “Is the incremental cost small relative to the net benefits?” No* Probably no Uncertain Probably yes Yes* * Need to be supported by formal cost-effectiveness analysis

Chicago 2014 “Is the incremental cost small relative to the net benefits?” No A formal cost-effectiveness analysis suggests that the intervention is not justified in terms of the resource allocated for benefit obtained.

Chicago 2014 “Is the incremental cost small relative to the net benefits?” Probably no Although no formal cost-effectiveness analysis was conducted, the resources required to implement the intervention or diagnostic strategy appear to be large or resource intensive relative to the estimate of the net benefit.

Chicago 2014 “Is the incremental cost small relative to the net benefits?” Uncertain The resource considerations are difficult to estimate and the benefits and risks/harms are uncertain so no reasonable consideration of cost- effectiveness is possible at this time.

Chicago 2014 “Is the incremental cost small relative to the net benefits?” Probably yes Although no formal cost analysis was conducted and efficacy estimates are uncertain, the resources required are deemed small enough that implementation might be considered reasonable from a cost effectiveness perspective. Potential example - ASA in suspected ACS.

Chicago 2014 “Is the incremental cost small relative to the net benefits?” Yes A formal cost-effectiveness analysis supports the view that the costs are redeemed by improved outcomes.

Chicago 2014 “Is the incremental cost small relative to the net benefits?” No* Probably no Uncertain Probably yes Yes* * Need to be supported by formal cost-effectiveness analysis

Chicago 2014 A couple of examples

Chicago 2014 Example 1 Cardiogel

Chicago 2014 Cardiogel systematic review identifying moderate quality evidence from 3 randomized controlled trials (Lang , Morrison , Morley ) downgraded for risk of bias demonstrating that in patients with cardiac arrest, “cardiogel” a hypothetical free radical scavenger improves survival to hospital discharge in comparison to placebo. The RR for this and other critical outcomes is % (CI )

Chicago 2014 Cardiogel cardiogel is perceived by committee members and the TFQO as expensive “Probably no” to “cost small” Q “Uncertain” to cost-effectiveness Q (as no formal analysis was conducted or identified)

Chicago 2014 Cardiogel: ILCOR guidance The treatment recommendation should not address resource considerations State “no formal economic analysis based on a systematic review was completed or identified” in the “research evidence” column to provide justification Furthermore the question of resource consumption should not weigh in the development of the treatment recommendation.

Chicago 2014 Example 2 “A novel bodywrap device”

Chicago 2014 Bodywrap systematic review identifying very low quality evidence ( Lang , Morrison , Morley ) downgraded for imprecision, risk of bias and indirectness suggesting that a novel bodywrap device to assist CPR fails to improve any critical or important outcome in comparison to standard CPR; RR % CI 0.4 – 1.6 no evidence of harm.

Chicago 2014 Bodywrap the device is perceived by the committee members and the TFQO to be significantly more expensive in all countries represented by the Taskforce than standard CPR and/or its implementation would consume significant resources in most health care settings “Probably no” to “cost small” Q an economic analysis based on a systematic review of addressing the incremental cost- effectiveness of CPR bodywrap was not done “Uncertain” to cost-effectiveness Q (as no formal analysis was conducted or identified)

Chicago 2014 Bodywrap: ILCOR guidance (1) In this setting it would be appropriate for the TFQO and EvRevs to incorporate consensus-based statements pertaining to resource considerations into the development of the treatment recommendation for this PICO. Specifically it would be appropriate to suggest against the use of bodywrap over standard CPR – weak recommendation

Chicago 2014 Bodywrap: ILCOR guidance (2) “In making this recommendation we place greatest value on not directing health care resources that were perceived by the committee to be high towards an unproven treatment except under experimental conditions.”

Chicago 2014 If bodywrap were: relatively inexpensive and/or already readily available to resuscitation teams in most or all settings the Task Force could consider making a weak recommendation in favour of CPR bodywrap over standard CPR “In making this recommendation we place a higher value on a potential clinical benefit in a setting with a high rate of poor outcomes over other considerations”.

Chicago 2014 Additional comments when a useless or harmful therapy is in wide use, developing recommendations against a management approach is also appropriate as would recommendations to use interventions only in research. Task Forces may also choose to divide their recommendations into ones that address high and low resource settings.

Chicago 2014 ?