What is “sufficient” evidence to inform combination HIV prevention programs Stefan Baral

Evidence Supporting Interventions Donabedian Approach  Process  The Traditional Gold Standard for M&E  Is system efficient?  Counting the actual products distributed, people trained, etc Condoms, Peer Educators, Paralegals, etc.  Structure  Structural Outcomes of the Intervention  Health Systems, Health Policies, etc  Outcome  Emerging Gold Standard…  What is happening with outcome of interest?  Impact! Efficacy vs Effectiveness 2

What is Sufficient Evidence? Evidence-based medicine is a global standard  Double-Blinded (DB) RCT is gold standard Evidence-based PH interventions should also be a global standard  Often limited evidence, PH decision still needs to be made  Precautionary Principle for PH?  When there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation To develop guidelines  Need to characterize  Efficacious  Effective  Sustainable and Scalable programs 3

Agency for Healthcare Research and Quality (AHRQ) US Preventive Services Task Force (USPSTF)  Three-Step System  Strength of Recommendation  Letter (A-D, I)  Level of Certainty  Low, Medium, High  Suggestions for Practice

Strength of Recommendation RatingStrength of RecommendationPractice Recommendations ARecommends the Service High Certainty that net benefit is substantial Offer/Provide This Service BRecommends the Service High certainty that net benefit is moderate Moderate certainty that net benefit is substantial Offer/Provide This Service CRecommends against Routine Provision of this service Special considerations for or against in each patient Moderate certainty that net benefit is low Offer/Provide only if special considerations support in individual DRecommends against the service Moderate or high certainty that service has no net benefit or harms outweigh the benefits Discourage the use of this service ICurrent Evidence is Insufficient to assess balance between benefits and harms Evidence is : lacking, poor quality, conflicting Never be offered

Level of Certainty Description HighEvidence from: Methodologically Sound Studies in Primary Care Populations (Generalizability) Health outcomes evaluation (effectiveness) Unlikely to be affected by future studies ModerateEnough evidence to determine effect of service on health outcomes, but limited confidence in estimate Evidence constrained by Number/size/quality of studies, Inconsistency, limited generalizability Recommendation may change based on future results LowEvidence is insufficient to assess health outcomes Insufficient because: Limited number/size of studies, flaws in study design, inconsistency, gaps in chain of evidence, not generalizable, inadequate info More data will allow estimation of effects on health outcomes

CDC Prevention Research Synthesis (PRS) Project  3 Domains  Study Design  Study Implementation and Analysis  Strength of Evidence  2 Tiers  Tier 1 – Best Evidence  Tier 2 – Good Evidence  Separate Criteria for  Individual-level interventions (ILI) and group-level interventions (GLI)  Community-Level Interventions (CLI)

CDC Prevention and Treatment of Opportunistic Infections  Two Step System  Letter – Strength of Recommendation related to Practice Recommendation  Efficacy Data  Clinical Benefit  Roman Numeral – Quality of Evidence Source: Kaplan, et al. Guidelines for Prevention and Treatment of Opportunistic Infections in HIV-Infected Adults and Adolescents. MMWR. 2009

Strength of Recommendation RatingStrength of RecommendationUse AStrong Efficacy Data Substantial Clinical Benefit Should always be offered BModerate Efficacy/Substantial Clinical Benefit High Efficacy/Limited Clinical Benefit Generally be offered CInsufficient Efficacy Data Good Efficacy Data/Efficacy Data does not outweigh adverse effects/actual cost/opportunity cost Optional DLack of Efficacy Data/Moderate adverse outcome data Generally not be offered EGood evidence for lack of efficacy or adverse outcome Never be offered

Quality of Evidence RatingQuality of Evidence IEvidence from at least one high-quality DB RCT IIEvidence from at least one Quasi-experimental clinical trial no randomization, no blinding, etc Cohort/Case-control data Ideally multiple centers Multiple Time-Series Studies Dramatic Results from Uncontrolled Experiments IIIEvidence from Expert Opinion Clinical experience Reports of Expert Committees/Authoritative Bodies Descriptive Studies

UK National Institute for Clinical Excellence (NICE) (Formerly the Health Development Agency)

Oxford Center For Evidence-Based Medicine Level of Evidence Rating within Level Details 1ABCABC SR of RCTs (homogeneity) Individual RCT All or None (no outcome either before or after intervention—ie parachutes) 2ABCABC SR of Cohort Studies (with homogeneity) Individual Cohort Outcomes Research/Ecological Work 3ABAB SR Of Case-Control Studies (with homogeneity) Individual Case-Control Studies 4Case-Series (or poor quality cohort/case-control) 5Expert Opinion 12

CEBM Grade of Recommendations 13 GradeCharacteristics AConsistent Level 1 Study BConsistent Level of 2 or 3 Studies or Extrapolating from Level 1 Studies (ie off label use) CLevel 4 Studies or Extrapolations from Level 2/3 DLevel 5 Evidence or troublingly inconsistent or inconclusive studies of any level

Grading of Recommendations Assessment, Development and Evaluation (GRADE) Score-based system designed for clinical interventions  Type of Evidence  Quality  Consistency  Directedness  Effect Size Includes Values and Preferences PICOTS Questions  Population, Intervention/Exposure, Comparison/Control, Outcome, Timing, Setting (PC, Specialty, In-Patient)

GRADE Characteristics Type of Evidence  RCT or SR of RCT  +4  Observational Evidence  +2 Quality  Blinding, retention, subjective outcomes  0 to -3 Directness  Generalizability  0 to -2 Effect Size  Measure of association >2 or >5  0 to +2

GRADE Score Strength of Recommendation  High  > or = 4  Medium  3  Low  2  Very Low  < or = 1 Values and Preferences

Highest Attainable Standard of Evidence System for HIV Interventions (HASTE)

Tension Between Internal and External Validity Challenges for Evidence Combination Prevention Internal Validity  Minimal study biases suggesting confidence in ultimate conclusion of the study  External Validity  Generalizability of ultimate findings to broader population Traditional Question for Clinicians/Programmers  Does it work? What is effect size?  Should I use it? Implementation Questions  How, when, why, and where does it work?  What factors influence effectiveness?  Should I use it? How should I use it?

Tension in Research about Validity Traditional Approach is to establish internal validity with certain study designs and then have studies focused on external validity  Internal Validity  Phase 1 (Safety), Phase 2a/b (tolerability, TOC), Phase 3 (Efficacy)  External Validity  Phase 4 (Post-Marketing)

Traditional Research Pathway Effectiveness Research (and guideline development) generally happens prior to implementation research  Are there more time-effective approaches to integrate implementation research with effectiveness/efficacy research  Assess barriers/facilitators to intervention uptake  acceptance/adoption/routinization  Diagnose quality gaps  Fidelity  Characterize Sustainability  Maintenance, Cost-Effectiveness