1. Methodological Issues in Physician-Level Measurement of Clinical Quality Elizabeth A. McGlynn, Ph.D. June 26, 2006

2. McGlynnAcadHealth-2 06/26/06 Information About Individual Physicians Performance is Increasingly Sought Health plans believe they can save money through differential payments to physicians (pay for performance) Employers believe they can save money through increasing consumer cost-sharing (consumer directed health plans) Medical groups believe they can negotiate higher rates or market share by demonstrating better performance (tiered networks, rate increases) Consumers are likely to demand information on performance as the share they pay for health care increases (public release)

3. McGlynnAcadHealth-3 06/26/06 What Is Being Measured?

4. McGlynnAcadHealth-4 06/26/06 Data Sources for Measuring Quality Available sources include: –Administrative (claims) data –Manual abstraction of medical records –Surveys of patients –Inspection of office practice –Extraction of data from electronic medical records –Board certification/Maintenance of certification Each of these sources has strengths and weaknesses No single source is adequate to address all questions

5. McGlynnAcadHealth-5 06/26/06 Most Existing Approaches to Measuring Physician Performance Use Claims Data Data are readily available and impose less burden on providers But they have some significant problems –Generally available one payer at a time –Information availability driven by the benefit package and the ways coding systems are used –Some confounding of physician practice patterns with patient behavior Pressure to deliver answers driving widespread use of these methods

6. McGlynnAcadHealth-6 06/26/06 Current Approaches to Quality Measurement Leading indicators –One measure at a time Condition-specific aggregates/composites –Multiple measures on the same population with the same health problem Comprehensive cross-condition measures –Patient as the unit of analysis

7. McGlynnAcadHealth-7 06/26/06 Examples of Where These Approaches Are Currently Used ApproachUse Leading indicatorsPay for performance Public reporting Tiered networks Disease compositesRecognition programs Maintenance of certification Comprehensive aggregatesNot in widespread use

8. McGlynnAcadHealth-8 06/26/06 What You Measure May Affect the Conclusions You Draw

9. McGlynnAcadHealth-9 06/26/06 Some Challenges in Measuring Physician Performance

10. McGlynnAcadHealth-10 06/26/06 Physicians See Multiple [Different] Patients MD1 PT3 PT2 PT1 So, representing the variety of practice matters: Case Mix Adjustment MD2 PT5 PT4 PT3

11. McGlynnAcadHealth-11 06/26/06 A Market Basket of Indicators May Be Necessary to Reflect the Variety of Practice

12. McGlynnAcadHealth-12 06/26/06 Patients See Multiple Providers PT1 MD3 MD2 MD1 PT2 PT3 PT4 PT5 PT6 PT9 PT8 PT7 So, determining who is responsible matters Attribution Hosp A Hosp B

13. McGlynnAcadHealth-13 06/26/06 Information Rarely Available to Link Patients to Physicians a Priori As gatekeeper models decline, no clear assignment of patients to a physician exists Algorithms are used to assign patients to physicians –Done most frequently in economic profiling –Basis is majority of dollars or visits We are experimenting with other rules: –First eligible provider seen in study period –Provider triggering eligibility for indicator Critical to reality test assignments

14. McGlynnAcadHealth-14 06/26/06 Physicians Have Multiple Contracts Medicare MD3 MD2 MD1 PacifiCare Humana Wellpoint United Aetna Medicare Anthem So, putting the pieces together matters: Aggregation

15. McGlynnAcadHealth-15 06/26/06 Few Physicians Can Be Evaluated Using Single Indicators from One Payer

16. McGlynnAcadHealth-16 06/26/06 Physicians Practice in Different Systems So, understanding the organizational context matters: Fair comparisons

17. McGlynnAcadHealth-17 06/26/06 Little Routine Information Available on Physician Practice Setting Taking organizational context into account is challenging because of data limitations Using location may be misleading –Shared space vs. shared practice Rationale for constructing scores at group level: –Increase sample size –Demonstrate value of integrated medical groups –Avoid scores at the physician level Relatively little known about within vs. between group variation

18. McGlynnAcadHealth-18 06/26/06 Categorizing Physician-Level Results Many applications of physician-level scoring require using results to categorize physicians –In/out of network –In/out of performance bonus –Tiering We prefer statistical testing to straight cut-points Applied this to the three different approaches to MD-level scoring –Test performance compared to the mean –Use 95% confidence interval around each providers score –Those with scores significantly below average were assigned to the low performance category

19. McGlynnAcadHealth-19 06/26/06 Different Methods Will Result in Different Category Assignments

20. McGlynnAcadHealth-20 06/26/06 Different Results Under Different Systems Likely To Produce Challenges from MDs

21. McGlynnAcadHealth-21 06/26/06 Summary A number of methodological issues arise in creating quality scores at the physician level We need to better understand the implications of these methodological choices Because the data on which the scores are based were not intended for this purpose, feedback loops and data quality improvement are essential But, the world isnt going to wait for us to get the methods perfect…

22. McGlynnAcadHealth-22 06/26/06 This Train Is Headed Your Way!