1. Building the Science of Health Care Quality Improvement Intervention Denise Dougherty, Ph.D. Senior Advisor, Child Health and Quality Improvement AHRQ Annual Conference 2010 Coordinator and Moderator Implementation, Change, and Improving Health Care Quality and Safety: Lessons Learned From AHRQ’s Implementation Science Awards September 28, 2010

2. Overview (15 minutes) Introductory Comments: Why and how does AHRQ Focus on Implementation Science? Introductory Comments: Why and how does AHRQ Focus on Implementation Science? An emerging framework An emerging framework Implementation Science grantees work Implementation Science grantees work – Rita Mangione-Smith – Carrie Byington Interactive Discussion Interactive Discussion

3. The “3T’s” Road Map to Transforming U.S. Health Care Key T1 activity to test what care works what care works Clinical efficacy research Key T2 activities to test who benefits from who benefits from promising care Outcomes research Comparative effectiveness research Health services research Key T3 activities to test how to deliver high-quality how to deliver high-quality care reliably and in care reliably and in all settings all settings Measurement and accountability of health care quality and cost care quality and cost Implementation of Interventions and health care system redesign care system redesign Scaling and spread of effective interventions Research in above domains T1T2T3 Basic biomedical science Clinical efficacy knowledge Clinical effectiveness knowledge Improved health care quality and value and population health Source: JAMA, May 21, 2008: D. Dougherty and P.H. Conway, pp. 2319-2321. The “3T’s Roadmap to Transform U.S. Health Care: The ‘How’ of High-Quality Care.”

4. Relevant Provisions in New Laws: American Recovery and Revitalization Act (ARRA) “Meaningful Use” of Health IT = “Meaningful Use” of Health IT = Beyond getting the electrons in place Beyond getting the electrons in place Using health IT to improve quality and safety of health care Using health IT to improve quality and safety of health care Funding for Comparative Effectiveness Research (CER) beyond purely clinical interventions Funding for Comparative Effectiveness Research (CER) beyond purely clinical interventions System redesign System redesign Enhanced registries for QI and CER Enhanced registries for QI and CER Accelerating Implementation of Comparative Effectiveness Findings on Clinical and Delivery System Interventions by Leveraging AHRQ Networks Accelerating Implementation of Comparative Effectiveness Findings on Clinical and Delivery System Interventions by Leveraging AHRQ Networks Other (http://www.ahrq.gov/fund/granarch.htm#RFA) Other (http://www.ahrq.gov/fund/granarch.htm#RFA)

5. New Laws: Patient Protection and Affordable Care Act Demonstration Projects for Quality Improvement Demonstration Projects for Quality Improvement “demonstration” — 312 mentions “demonstration” — 312 mentions “pilot” — 80 mentions# “pilot” — 80 mentions# Creation of the Center for Medicare and Medicaid Innovation (CMI) Creation of the Center for Medicare and Medicaid Innovation (CMI) National Strategy for Quality Improvement National Strategy for Quality Improvement More (see CRS report) More (see CRS report) # http://e-caremanagement.com/pilots-demonstrations-innovation-in-the-ppaca-healthcare-reform-legislation/ http://e-caremanagement.com/pilots-demonstrations-innovation-in-the-ppaca-healthcare-reform-legislation/ http://www.aamc.org/reform/summary/ph.pdf

6. CHIPRA – CMS Quality Demonstration Grants to States Aims: Aims: A) Experiment with, and evaluate the use of new measures for quality of Medicaid/CHIP children's health care; B) Promote the use of HIT for the delivery of care for children covered by Medicaid/CHIP; C) Evaluate provider-based models which improve the delivery of Medicaid/CHIP children's health care services; or D) Demonstrate the impact of the model Electronic Health Record format for children (developed and disseminated under section 401(f)) on improving pediatric health, and pediatric health care quality, as well as reducing health care costs. E) Broad systems approaches/medical home 10 awards made Feb. 2010 to individual States and consortia of States 10 awards made Feb. 2010 to individual States and consortia of States National Evaluation (in planning stage –AHRQ has lead) National Evaluation (in planning stage –AHRQ has lead)http://www.cms.gov/CHIPRA/15_StateDemo.asp

7. AHRQ Funding to Test and Disseminate Strategies to Improve Quality and Patient Safety ACTION II ACTION II Program Announcements Program Announcements Evaluation of Spread of the Keystone projects (health care associated infections) Evaluation of Spread of the Keystone projects (health care associated infections) Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (forthcoming report) Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (forthcoming report) Co-sponsorship of the NIH Science of D & I conference (2011) Co-sponsorship of the NIH Science of D & I conference (2011) Innovations Clearinghouse Innovations Clearinghouse Knowledge Transfer Projects Knowledge Transfer Projects Value Exchanges Value Exchanges PAR 08-136 PAR 08-136

8. What are We Trying to Learn from all This Work? Answer: Not Only the What, but the How and the Why of Healthcare Quality Improvement Answer: Not Only the What, but the How and the Why of Healthcare Quality Improvement

9. Current State of the Science: QII and Evaluation Designs—A Personal View Problem identification (vaguely defined) Problem identification (vaguely defined) Theory of action to solve the problem (often omitted, vague or in-name-only) Theory of action to solve the problem (often omitted, vague or in-name-only) Interventions Interventions – vaguely described; – not replicable; – conceptual confusion between “intervention” and “implementation” Focus on internal validity and related designs Focus on internal validity and related designs – Context of intervention/ implementation processes: not considered or not considered or considered post hoc and descriptive/idiosyncratic considered post hoc and descriptive/idiosyncratic effects of context/variation in context not considered in assessing results and variation in results effects of context/variation in context not considered in assessing results and variation in results “qualitative” research does not mean standards of the qualitative research field “qualitative” research does not mean standards of the qualitative research field Lack of validated measures of contextual variables (leadership, culture, teamwork, resources) Lack of validated measures of contextual variables (leadership, culture, teamwork, resources) – For publication: design driven by clinical hierarchy of evidence standards (RCTs at patient level) – If not for publication: threats to internal validity rarely considered; post only studies or simple pre-post without comparisons; implications for knowledge base not widely recognized. threats to internal validity rarely considered; post only studies or simple pre-post without comparisons; implications for knowledge base not widely recognized. Few comparison studies (one QI intervention to another; multiple settings) Few comparison studies (one QI intervention to another; multiple settings)

10. Specific Example: Context– Multiple potential influences on QII Results External factors – e.g.: External factors – e.g.: Regulatory requirement Regulatory requirement Payments or penalties Payments or penalties Local sentinel event Local sentinel event Structural/organizational characteristics (organization site) Structural/organizational characteristics (organization site) Culture, Teamwork, Leadership Culture, Teamwork, Leadership Implementation Processes and Tools Implementation Processes and Tools – Staff education and training – Audit and feedback Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, forthcoming. Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, forthcoming.

11. Specific Example: Quantitative Approaches to Context Heterogeneity - Progress Premise: Context often moderates intervention effectiveness Premise: Context often moderates intervention effectiveness This moderation effect can be represented statistically through the “intervention x context” interaction: This moderation effect can be represented statistically through the “intervention x context” interaction: – Yi = b0 + b1 × Ti + b2 × Ci + b12 × Ti × Ci + εi, where i denotes the unit of analysis (usually the various sites in the study, but can also be dyads of sites in matched comparisons), Yi denotes the outcome measure, Ti denotes the intervention status (Ti=1 for intervention, Ti=0 for control), Ci denotes the contextual factor, Ti × Ci denotes the “intervention × context” interaction, εi denotes random error, b0 denotes the intercept for the model, b1 denotes the main effect for the intervention, b2 denotes the main effect for the contextual factor, and b12 denotes the moderation effect for the contextual factor, i.e., the influence of the contextual factor on intervention effectiveness. where i denotes the unit of analysis (usually the various sites in the study, but can also be dyads of sites in matched comparisons), Yi denotes the outcome measure, Ti denotes the intervention status (Ti=1 for intervention, Ti=0 for control), Ci denotes the contextual factor, Ti × Ci denotes the “intervention × context” interaction, εi denotes random error, b0 denotes the intercept for the model, b1 denotes the main effect for the intervention, b2 denotes the main effect for the contextual factor, and b12 denotes the moderation effect for the contextual factor, i.e., the influence of the contextual factor on intervention effectiveness. Looks like progress, assuming we can quantify contextual variables Looks like progress, assuming we can quantify contextual variables Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, Chapter 12, Special contribution from Naihua Duan, Columbia University, New York, New York forthcoming Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, Chapter 12, Special contribution from Naihua Duan, Columbia University, New York, New York forthcoming

12. Meta-Science Issues IRBs IRBs Study sections Study sections Promotion and Tenure Promotion and Tenure Little collaborative research (understanding effects of variations in context) Little collaborative research (understanding effects of variations in context) Conflict between research and evaluation Conflict between research and evaluation Limited knowledge of evaluation “how to”? Limited knowledge of evaluation “how to”?

13. Research in Implementation and Change While Improving Quality – The Answer? PAR -08-136 PAR -08-136 – A relatively small attempt to specifically try to understand the how and why in a rigorous way $300K/year $300K/year – No dedicated pot of funds at AHRQ – highly competitive This session: This session: – Two examples – in process – methods and interim results, not definitive findings – Interactive discussion – What would you add? What would you add? What else do you need to know? What else do you need to know?