1. Measuring and Reporting Patients’ Experiences with Their Doctors Process, Politics and Public Reports in Massachusetts Melinda Karp MHQP Director of Programs June 26, 2006

2. Today’s Objectives Provide brief background on MHQP as important context for measurement and reporting efforts Describe evolution of MHQP agenda for measuring and reporting patient experiences— key methods questions in moving from research to large scale implementation Describe stakeholder perspectives and decision points around key reporting issues

3. Stakeholders at the MHQP “Table” Provider Organizations –MA Hospital Association –MA Medical Society –2 MHQP Physician Council representatives Government Agencies –MA EOHHS –CMS Region 1 Employers –Analogue Devices Health Plans –Blue Cross Blue Shield of Massachusetts –Fallon Community Health Plan –Harvard Pilgrim Health Care –Health New England –Tufts Health Plan Consumers –Exec. Director Health Care for All –Exec. Director NE Serve Academic –Harris Berman, MD, Board Chair

4. 2002-2003  Demonstration project in partnership with The Health Institute (Funded by Commonwealth and RWJF) 2004-2005  Development of viable business model for implementing statewide patient experience survey 2005-2006  Fielding and reporting of statewide survey The Evolution of MHQP’s Patient Experience Measurement Agenda

5. “1 st Generation” Questions: Moving MD-Level Measurement into Practice Is there enough performance variability to justify measurement? What sample size is needed for highly reliable estimate of patients’ experiences with a physician? How much of the measurement variance is accounted for by physicians as opposed to other elements of the system (practice site, network organization, plan)? What is the risk of misclassification under varying reporting frameworks?

6. Sample Size Requirements for Varying Physician-Level Reliability Thresholds

7. Allocation of Explainable Variance: Doctor-Patient Interactions Communication Whole- person orientation Health promotion Interpersonal treatment Patient trust Source: Safran et al. JGIM 2006.

8. Allocation of Explainable Variance: Organizational/Structural Features of Care Source: Safran et al. JGIM 2006.

9. Source: Safran et al. JGIM 2006; 21:13-21. Risk of Misclassification

10. Source: Safran et al. JGIM 2006; 21:13-21. Risk of Misclassification

11. MHQP 2005 Statewide Survey Physician-level survey format Site-level sampling to support site-level reporting Estimated samples required to achieve > 0.70 site-level reliability

12. Site Reliability Chart: Integration Sample sizes needed to achieve site reliability for integration domain.

13. Setting the Stage for Public Reporting: Key Issues for Physicians What measures get reported How measures get reported

15. Percent of Sites with A-Level Reliability by Measure and Survey-Type Adult PCP % Pediatric % MD – Patient Interactions Communication9897 Knowledge of patient9186 Health Promotion4697 Integration of care7961 Organizational/Structural Features of Care Access99100 Visit-based continuity100 Office Staff9599 Clinical Team3786 Willingness To Recommend6259

16. ½ ½ 15 th ptile50 th ptile85 th ptile 78.780.5 84.3 86.1 87.9 89.7 79.685.288. 8 ½ Framework for Public Reporting Integration of Care

21. Summary and Implications With sufficient sample sizes, data obtained using C/G CAHPS approach yields data with MD- and site-level reliability >0.70 For site-level reliability, number of MDs per site influences required sample sizes Risk of misclassification can be held to <5% with by –Limiting number of performance categories –Creating buffer (“zone of uncertainty”) around performance cutpoints Trade-offs are likely around data quality standards (e.g., acceptable “risk”) vs. data completeness

22. The Continuing Evolution… 2006-2007 Engagement around QI activities –Participation in Commonwealth Fund grant to study highest performing practices –Grant proposal to Physician Foundation to develop and pilot integrated clinical-patient experience QI curriculum Determining achievable benchmarks Fielding of Specialist Care Survey in 2006/2007 Repeat Primary Care Survey in 2007

23. For more information … Melinda Karp, Director of Programs mkarp@mhqp.org 617-972-9056 www.mhqp.org