Using Multiple Data Sources to Understand Variable Interventions Bruce E. Landon, M.D., M.B.A. Harvard Medical School AcademyHealth Annual Research Meeting June 10, 2008

The Problem Quality improvement interventions often teach a method for improvement, rather than a specific intervention Variability in implementation across sites –Site specific needs –Resources, leadership, etc. –Specific interventions Variable evidence of success at the level of individual sites What can be learned from this variability?

Outline The HRSA Health Disparities Collaboratives –What do organizations do in a QIC? The EQHIV Study in Ryan White Funded HIV Clinics –What accounts for negative results? –How reliable are organizational assessments? Conclusions

IHI and the Breakthrough Series Collaborative method for improving the quality and value of health care Short term (6-18 months) programs that bring together learning teams from multiple organizations Developed by IHI in 1995 –Over 50 collaboratives (just by IHI) –Over 2000 improvement teams

Setting Aims Measuring Progress Selecting/Implementing Changes Testing Changes The IHI Learning Model Source: The Institute for Healthcare Improvement

Health Disparities Collaboratives Prevention and Screening PrePost Collaborative3951*** Collaborative – Int. Control = 6.5% *** Collaborative – Ext. Control = 5.5% *** Int. Control4145 *** Ext. Control3945 *** Disease Monitoring and Treatment Collaborative4757 *** Collaborative – Int. Control = 5.9% *** Collaborative – Ext. Control = 6.5% *** Int. Control4650 *** Ext. Control5054 *** Source: Landon BE, et al. NEJM 2007***p<.001

Looking Inside the Collaboratives Interventions systematically recorded by each site in “monthly reports” Coding instrument developed to categorize interventions based on the CCM and stage of implementation Coded by two trained abstractors independently

The Chronic Care Model Source: Wagner et. Al, The McColl Institute

Delivery System Design (Changes in the organization of human resources) Sub-Categories Care management roles Practice team Care delivery Proactive follow-up Planned visit Visit system changes Examples Nurses take on more patient follow-up tasks Multidisciplinary team meetings Standardize specialist referral process Call patients who are overdue for visit Targeted reminders in charts prior to visits Group visits

Decision Support (guidance for provider behavior or decision-making) Sub-categories Institutionalization of guidelines, protocols and prompts Provider education Expert consultation support Examples Structured forms to replace progress notes Podiatrist teaches nurses to do foot exams Case conferences with specialists

Quality Improvement Activities by CCM Categories CCM CategoryTotal Number of Activities Mean number of Activities per Center (Range) Delivery system design (2-15) Self-management support (1-19) Decision support (1-14) Information support (3-16) Community linkages (1-24) Health system organization (1-12) Total1, (8-84)

Intensity of Quality Improvement Activities, by CCM Categories CCM Category Number (%) of Activities Implemented (“Actions”) % of Actions Institutionalized and/or Refined % of Actions Evaluated % of Actions of “high” or “very high” Probable Impact Delivery system design 295 (86%)57%28%2% Self-management support 251 (90%)62%43%8% Decision support200 (87%)51%32%9% Information support313 (89%)50%35%1% Community linkages284 (83%)42%10%0% Health system organization 189 (91%)60%21%2% Total1,532 (87%)53%28%3%

Relationship between QI Activities and Quality Change No significant relationships between: –Total number of activities –Mean impact score of activities –% implemented –% institutionalized –Number graded as “high” or “very high” And changes in observed quality of care

Limitations to This Type of Analysis Dramatic loss of power when changing from a controlled design to an observational design with n=~40 One size might not fit all Specific interventions are designed to meet local needs and might not be transferrable How to account for local context?

Landon, B. E. et. al. Ann Intern Med 2004;140: The EQHIV Study Difference Intervention Clinic Difference Control Clinic Percentage pointsP Value Antiretroviral therapy Receipt of HAART on last visit for appropriate patients >0.2 Viral load controlled Screening and prophylaxis TB screening >0.2 Influenza shot7.36.8>0.2 Hepatitis C status5.56.2>0.2 Pap smear Prophylaxis against Pneumocystis carinii pneumonia >0.2 Access to care Visits in 3 or 4 quarters5.42.7>0.2

Possible Explanations The intervention did not work Clinics were not “prepared” for the intervention (or ready) Change might be required across the spectrum of the CCM to achieve results Structure and culture at individual clinics might impede change Improving chronic care requires broad across the board changes

Assessment of Organizational Change Pre/post surveys of: –Clinic directors –Clinicians Results show modest pre/post changes in 3 domains of the CCM, with little change in the other domains System changes were likely not sufficient to lead to broad improvements

How to Increase Reliability of Organizational Surveys? Scale (# items) ρrρr at n j =N/J Needed for =0.7 ρr(i)ρr(i) Cronbach's α Openness to QI (7) HIV knowledge (6) Research emphasis (3) Autonomy (3) Patient help (3) Guidelines emphasis (2) Barriers to QI (5) Patient load (3) Source: Marsden, Landon, et. al. HSR, ρr

Other Potential Methods Pre/post surveys of clinicians/support staff/leadership Site visits/qualitative data Pre surveys to assess “readiness for change” ….and so on

Conclusions Additional data sources are needed to understand what happens at the individual clinic level These data can be useful for understanding more about the implementation of the interventions Data are limited by small sample sizes, lack of a control group, and low reliability Many questions remain unanswered (e.g., What other components of care are important (leadership, resources, composition, etc.)?)

Final Thoughts Quality Improvement is difficult Studying Quality Improvement is particularly challenging Quality Improvement Implementation Research is a rich area that brings together many disciplines