1. Medical Education Outcomes Research Frederick Chen, MD, MPH Center for Primary Care Research Agency for Healthcare Research and Quality June 26, 2003

2. Setting the Context What is the goal of medical education? What is the goal of medical education? To produce physicians who deliver high quality care To produce physicians who deliver high quality care The quality of care that the public receives is determined to some extent by the quality of medical education students and residents receive. – Commonwealth Report 2002 The quality of care that the public receives is determined to some extent by the quality of medical education students and residents receive. – Commonwealth Report 2002

3. The Questions How do we demonstrate that medical education is producing physicians who deliver high quality care? How do we demonstrate that medical education is producing physicians who deliver high quality care? Is the delivery of high quality care related in any way to medical education? Is the delivery of high quality care related in any way to medical education? How do we introduce the measurement of patient-centered clinical outcomes to medical education research? How do we introduce the measurement of patient-centered clinical outcomes to medical education research?

4. Why we should study medical education outcomes Accountability in medical education Accountability in medical education American public, policymakers, and private health care managers have a pressing stake in the health of our nations medical education enterprise. American public, policymakers, and private health care managers have a pressing stake in the health of our nations medical education enterprise. Government already involved in licensing, financing Government already involved in licensing, financing

5. Why we should study medical education outcomes Medicare is largest supporter of graduate medical education Medicare is largest supporter of graduate medical education – $7.8 billion in 2000 – >100,000 medical residents in training Over 75% of medical schools receive public subsidies Over 75% of medical schools receive public subsidies – At least $2 billion Better available methodology and data Better available methodology and data

6. Commonwealth Report 2002 Principle: Academic health centers should be held accountable for their performance in educating the nations physicians. Principle: Academic health centers should be held accountable for their performance in educating the nations physicians. Finding: The available data are insufficient to judge the performance of AHCs in discharging their education responsibilities beyond establishing a minimum level of competency. Finding: The available data are insufficient to judge the performance of AHCs in discharging their education responsibilities beyond establishing a minimum level of competency.

7. Commonwealth Report 2002 Recommendation: Government should support research to produce valid and reliable measures of the costs and quality of medical education. Recommendation: Government should support research to produce valid and reliable measures of the costs and quality of medical education. $25M per year to AHRQ to develop and implement improved measures of performance $25M per year to AHRQ to develop and implement improved measures of performance Convene AAMC, ACGME, NBME, ABMS, HRSA to provide direction… Convene AAMC, ACGME, NBME, ABMS, HRSA to provide direction…

8. IOM - Health Professions Education Quality Chasm Recommendation #12 Quality Chasm Recommendation #12 Health care professionals need to be better prepared in order to provide the highest quality & safest care possible, and to function at optimum levels in an ever evolving and increasingly complex 21 st century health system Health care professionals need to be better prepared in order to provide the highest quality & safest care possible, and to function at optimum levels in an ever evolving and increasingly complex 21 st century health system

9. Research & Information AHRQ and foundations should support ongoing research projects focused on the core competencies and links with individual/ population health and links with educational approaches AHRQ and foundations should support ongoing research projects focused on the core competencies and links with individual/ population health and links with educational approaches Develop measures reflecting the core set of competencies Develop measures reflecting the core set of competencies AHRQ should issue report cards focused on clinical educational institutions AHRQ should issue report cards focused on clinical educational institutions

10. Medical Education Research What has been the state of medical education research? What has been the state of medical education research? Authority for medical education historically resides in profession Authority for medical education historically resides in profession Intrinsic capacity to self-regulate Intrinsic capacity to self-regulate

11. Medical Education Research Past 30 years have focused on: Past 30 years have focused on: – Basic research on reasoning – Use of knowledge – Problem based learning – Performance assessment (OSCE, std patients) – Provision of continuing education Norman, G. BMJ 2002;324 Norman, G. BMJ 2002;324

12. Lack of patient outcomes in medical education research Review of 600 research articles published in medical education journals (1996-98) Review of 600 research articles published in medical education journals (1996-98) Only 4 measured clinical outcomes of patients Only 4 measured clinical outcomes of patients Half measured trainee performance Half measured trainee performance 34% measured trainee satisfaction 34% measured trainee satisfaction – Prystowsky and Bordage. Med Ed 2001; 35.

13. Why havent we studied patient outcomes? Focus on undergraduate education Focus on undergraduate education Students able to overcome educational interventions Students able to overcome educational interventions Not able to conduct randomized, blinded trials Not able to conduct randomized, blinded trials No reliable data on outcomes No reliable data on outcomes No money for research No money for research Sound familiar? Sound familiar?

14. Current efforts Best Evidence Medical Education Best Evidence Medical Education International effort to apply evidence-based medicine criteria to medical education research International effort to apply evidence-based medicine criteria to medical education research Campbell Collaboration Campbell Collaboration

15. Current efforts ACGME Outcome Project – Competencies: ACGME Outcome Project – Competencies: – Patient care – Medical knowledge – Interpersonal and communication skills – Professionalism – Practice-based learning and improvement – Systems-based practice

16. Expert Meeting AHRQ - HRSA co-sponsored Expert meeting AHRQ - HRSA co-sponsored Expert meeting July 2001 July 2001 AHRQ – lead agency for healthcare quality AHRQ – lead agency for healthcare quality HRSA – supports training of healthcare providers HRSA – supports training of healthcare providers Investigate the link between medical education and health care improvement Investigate the link between medical education and health care improvement

17. Role of quality improvement How do physicians learn about CQI? How do physicians learn about CQI? What is effect of medical education upon What is effect of medical education upon – Ability to change and adapt? – Ability to improve practice? Do physicians trained in one system bring those skills to another system? Do physicians trained in one system bring those skills to another system?

18. Multi-method approaches Convergence of medical education researchers and health services researchers Convergence of medical education researchers and health services researchers Multi-institutional Multi-institutional New statistical methods New statistical methods Quantitative and Qualitative assessments of quality and performance Quantitative and Qualitative assessments of quality and performance

19. Overarching Goals Making the link between Quality and Medical Education Making the link between Quality and Medical Education Current focus on systems-change but what is influence of individual performance and roles within systems? Current focus on systems-change but what is influence of individual performance and roles within systems? Evaluation drives Curriculum Evaluation drives Curriculum

20. Next Steps Need to catalyze research that examines linkage between quality health care and medical education Need to catalyze research that examines linkage between quality health care and medical education Commitment to patient outcomes Commitment to patient outcomes Need to weave medical education back into fabric systems improvement Need to weave medical education back into fabric systems improvement

21. Role Of AACOM Interest in developing research among faculty and students Interest in developing research among faculty and students Active partnership in public efforts to improve medical education – IOM task force Active partnership in public efforts to improve medical education – IOM task force Encourage active engagement with AHRQ, HRSA in medical education research Encourage active engagement with AHRQ, HRSA in medical education research

23. Past efforts 1993 BHPr-AAMC agenda setting conference 1993 BHPr-AAMC agenda setting conference Research in medical education: Research in medical education: – Policies for the Future Created Centers for medical education research (mostly workforce and specialty choice) Created Centers for medical education research (mostly workforce and specialty choice)