Expanding Research and Evaluation Designs…for QII Carolyn M. Clancy, MD Director, AHRQ September 13, 2005

Are We Making Any Progress? Are We Making Any Progress? Second annual reports focus on quality of and disparities in health care in America Second annual reports focus on quality of and disparities in health care in America – Quality Report finds that quality is improving and identifies areas which are in need of major improvements – Disparities Report indicates that there are pervasive disparities related to race, ethnicity, and socioeconomic status

Preliminary NHQR Highlights 2005 Health Care Quality Continues To Improve at a Modest Pace Across Most Measures of Quality Health Care Quality Continues To Improve at a Modest Pace Across Most Measures of Quality – 10:1 ratio of measures improved to declined – Overall improvement rate: 2.8% Health Care Quality Improvement Is Variable, With Notable Areas of High Performance Health Care Quality Improvement Is Variable, With Notable Areas of High Performance – Patient safety: 10.2% – QIO Measures: 9.2% – Effectiveness measures: 2.8% High improvement subset: 5.4% High improvement subset: 5.4%

RAND Study: Quality of Health Care Often Not Optimal Doctors provide appropriate health care only about half the time Doctors provide appropriate health care only about half the time Percentage of time Alcohol dependence Hip fracture Peptic ulcer Diabetes Low back pain Prenatal care Breast cancer Cataracts 11% 23% 33% 45% 69% 73% 76% 79% E. McGlynn, S. Asch, J. Adams, et al., The Quality of Health Care Delivered to Adults in the United States, N Engl J Med, 2003

Public Perceptions * Percent who say they are dissatisfied with the quality of health care in this country… Has the quality of health care in this country… Gotten worse Gotten better Stayed about the same Don’t Know Source: Kaiser Family Foundation / Agency for Healthcare Research and Quality / Harvard School of Public Health National Survey on Consumers’ Experiences with Patient Safety and Quality Information, November 2004 (Conducted July 7 – September 5, 2005). * Gallup Poll conducted September 11-13, 2000 with 1,008 U.S. adults.

AHRQ’s Mission To improve the safety and quality, efficiency and effectiveness of health care for all Americans

Answers must be: Timely Timely Valid Valid Convincing Convincing Practical Practical

Publication Bibliographic databases Submission Reviews, guidelines, textbook Negative results variable 0.3 year years 50% 46% 18% 35% 0.6 year 0.5 year 9.3 years It takes 17 years to turn 14 per cent of original research to the benefit of patient care Dickersin, 1987 Koren, 1989 Balas, 1995 Poynard, 1985 Kumar, 1992 Poyer, 1982 Antman, 1992 Negative results Lack of numbers Expert opinion Inconsistent indexing 17:14 Original research Acceptance Implementation Balas, 2002

Treatments Thought to Work but Shown Ineffective Sulphuric acid for scurvy Sulphuric acid for scurvy Leeches for almost anything Leeches for almost anything Insulin for schizophrenia Insulin for schizophrenia Vitamin K for myocardial infarction Vitamin K for myocardial infarction HRT to prevent cardiovascular disease HRT to prevent cardiovascular disease Flecainide for ventricular tachycardia Flecainide for ventricular tachycardia Routine blood tests prior to surgery Routine blood tests prior to surgery ABMT for late stage Breast CA ABMT for late stage Breast CA BMJFebruary ; 324:474-5.

Expanding Research and Evaluation Designs…for QII Carolyn M. Clancy, M.D. Director, AHRQ September 13, 2005 DRAFT – Denise D and David Atkins – 9/7/05

“Health services research has not yet been sufficiently helpful in meeting the challenge of improving care in part because it has over-constrained both its methods and its favorite topics….HSR should become more effectively part of the solution. To do that will require that we enrich our portfolio of methods and broaden our area of inquiry.” --Berwick, HSR, 40:2 (April 2005)

Challenges to the RCT Paradigm The intervention targets are not patients directly, The intervention targets are not patients directly, Interventions are complex Interventions are complex Interventions change, and Interventions change, and Where the setting is an essential component of the question and intervention. Where the setting is an essential component of the question and intervention.

AHRQ Research Study: Computerized Physician Order Entry Systems Major Finding: While computerized physician order entry (CPOE) is expected to significantly reduce medication errors, systems must be implemented thoughtfully to avoid facilitating certain types of errors Major Finding: While computerized physician order entry (CPOE) is expected to significantly reduce medication errors, systems must be implemented thoughtfully to avoid facilitating certain types of errors Study looked at clinicians’ experience in using one CPOE system at a major urban teaching hospital Study looked at clinicians’ experience in using one CPOE system at a major urban teaching hospital Implementation problems can be minimized through testing before products are marketed and through adaptation to meet the needs of individual clinical settings Implementation problems can be minimized through testing before products are marketed and through adaptation to meet the needs of individual clinical settings R. Koppel, J. Metlay, A. Cohen, et al., Role of computerized physician order entry systems in facilitating medication errors,Journal of the American Medical Association, March 9, 2005 R. Koppel, J. Metlay, A. Cohen, et al., Role of computerized physician order entry systems in facilitating medication errors, Journal of the American Medical Association, March 9, 2005

Current QI evaluation questions AHRQ and others are asking Can a regional health information organization improve interoperability of health information technology systems and improve patient safety and quality of care? Can a regional health information organization improve interoperability of health information technology systems and improve patient safety and quality of care? Can pay for performance improve quality? Can pay for performance improve quality? Do changes in hospital culture reduce medical errors? Do changes in hospital culture reduce medical errors? What QI strategies work for reducing disparities? What QI strategies work for reducing disparities? – Different strategies for different populations and settings? – Same strategies?

Current QI evaluation questions AHRQ and others are asking - 2 When should coverage be linked with development of better evidence (‘decision-based evidence making’)? When should coverage be linked with development of better evidence (‘decision-based evidence making’)? [In an action-oriented world how to identify a control group?] [In an action-oriented world how to identify a control group?] Can evidence for the effectiveness of team training in obstetrics be developed as training is provided? Can evidence for the effectiveness of team training in obstetrics be developed as training is provided? For which questions is a registry useful? What methods will help us take advantage of new sources of electronic data? For which questions is a registry useful? What methods will help us take advantage of new sources of electronic data?

Galaxies of Evaluation Designs That May Inform QII Evaluation Designs and Methods [1][1] For purposes of this chart, intervention evaluation research is defined as studies designed to Answer the Questions: Does the Intervention Work? For Whom or What? Under What Conditions? RCT HSR: 2° data Analysis Medical/clinical/ HSR Galaxy Behavioral/Social Science Galaxy Action Research— K Lewin PDSA/SPC Deming/ Shewhart RCT Quasi-exp Case reports Case studies Qualita- tive methods Systems Science Case- Control ITS/MBS QII Evaluation Research Cohort study RD GRT

Questions?