Methodological Issues in Physician-Level Measurement of Clinical Quality Elizabeth A. McGlynn, Ph.D. June 26, 2006

McGlynnAcadHealth-2 06/26/06 Information About Individual Physicians Performance is Increasingly Sought Health plans believe they can save money through differential payments to physicians (pay for performance) Employers believe they can save money through increasing consumer cost-sharing (consumer directed health plans) Medical groups believe they can negotiate higher rates or market share by demonstrating better performance (tiered networks, rate increases) Consumers are likely to demand information on performance as the share they pay for health care increases (public release)

McGlynnAcadHealth-3 06/26/06 What Is Being Measured?

McGlynnAcadHealth-4 06/26/06 Data Sources for Measuring Quality Available sources include: –Administrative (claims) data –Manual abstraction of medical records –Surveys of patients –Inspection of office practice –Extraction of data from electronic medical records –Board certification/Maintenance of certification Each of these sources has strengths and weaknesses No single source is adequate to address all questions

McGlynnAcadHealth-5 06/26/06 Most Existing Approaches to Measuring Physician Performance Use Claims Data Data are readily available and impose less burden on providers But they have some significant problems –Generally available one payer at a time –Information availability driven by the benefit package and the ways coding systems are used –Some confounding of physician practice patterns with patient behavior Pressure to deliver answers driving widespread use of these methods

McGlynnAcadHealth-6 06/26/06 Current Approaches to Quality Measurement Leading indicators –One measure at a time Condition-specific aggregates/composites –Multiple measures on the same population with the same health problem Comprehensive cross-condition measures –Patient as the unit of analysis

McGlynnAcadHealth-7 06/26/06 Examples of Where These Approaches Are Currently Used ApproachUse Leading indicatorsPay for performance Public reporting Tiered networks Disease compositesRecognition programs Maintenance of certification Comprehensive aggregatesNot in widespread use

McGlynnAcadHealth-8 06/26/06 What You Measure May Affect the Conclusions You Draw

McGlynnAcadHealth-9 06/26/06 Some Challenges in Measuring Physician Performance

McGlynnAcadHealth-10 06/26/06 Physicians See Multiple [Different] Patients MD1 PT3 PT2 PT1 So, representing the variety of practice matters: Case Mix Adjustment MD2 PT5 PT4 PT3

McGlynnAcadHealth-11 06/26/06 A Market Basket of Indicators May Be Necessary to Reflect the Variety of Practice

McGlynnAcadHealth-12 06/26/06 Patients See Multiple Providers PT1 MD3 MD2 MD1 PT2 PT3 PT4 PT5 PT6 PT9 PT8 PT7 So, determining who is responsible matters Attribution Hosp A Hosp B

McGlynnAcadHealth-13 06/26/06 Information Rarely Available to Link Patients to Physicians a Priori As gatekeeper models decline, no clear assignment of patients to a physician exists Algorithms are used to assign patients to physicians –Done most frequently in economic profiling –Basis is majority of dollars or visits We are experimenting with other rules: –First eligible provider seen in study period –Provider triggering eligibility for indicator Critical to reality test assignments

McGlynnAcadHealth-14 06/26/06 Physicians Have Multiple Contracts Medicare MD3 MD2 MD1 PacifiCare Humana Wellpoint United Aetna Medicare Anthem So, putting the pieces together matters: Aggregation

McGlynnAcadHealth-15 06/26/06 Few Physicians Can Be Evaluated Using Single Indicators from One Payer

McGlynnAcadHealth-16 06/26/06 Physicians Practice in Different Systems So, understanding the organizational context matters: Fair comparisons

McGlynnAcadHealth-17 06/26/06 Little Routine Information Available on Physician Practice Setting Taking organizational context into account is challenging because of data limitations Using location may be misleading –Shared space vs. shared practice Rationale for constructing scores at group level: –Increase sample size –Demonstrate value of integrated medical groups –Avoid scores at the physician level Relatively little known about within vs. between group variation

McGlynnAcadHealth-18 06/26/06 Categorizing Physician-Level Results Many applications of physician-level scoring require using results to categorize physicians –In/out of network –In/out of performance bonus –Tiering We prefer statistical testing to straight cut-points Applied this to the three different approaches to MD-level scoring –Test performance compared to the mean –Use 95% confidence interval around each providers score –Those with scores significantly below average were assigned to the low performance category

McGlynnAcadHealth-19 06/26/06 Different Methods Will Result in Different Category Assignments

McGlynnAcadHealth-20 06/26/06 Different Results Under Different Systems Likely To Produce Challenges from MDs

McGlynnAcadHealth-21 06/26/06 Summary A number of methodological issues arise in creating quality scores at the physician level We need to better understand the implications of these methodological choices Because the data on which the scores are based were not intended for this purpose, feedback loops and data quality improvement are essential But, the world isnt going to wait for us to get the methods perfect…

McGlynnAcadHealth-22 06/26/06 This Train Is Headed Your Way!