1. Measurement theory and provider profiling Timothy P. Hofer MD Dept. Medicine, University of Michigan VA Ann Arbor HSR&D Center of Excellence

2. The measurement problem Quality  i(jk) construct indicator

3. Site (clinic, hospital) Provider (physician) Patient Levels of care Indicators iii

4. Implications of the measurement model The indicator is a fallible measure of the construct Some indicators are less precise than others Quality indicators are very imprecise for a variety of reasons You need to account for the measurement error The location of the construct variability can suggest different causes, interventions and measurement procedures

5. Intra-class correlation(=reliability) Ability to distinguish between physicians (or sites) single observation under a specified set of conditions of measurement.

6. Vol. 281 No. 22, pp. 2065-2160, June 9, 1999

7. MD laboratory utilization profiles

9. VA Network 11 Diabetes Care Project Health Services Research Volume 37 Issue 5 Page 1159 - October 2002 doi:10.1111/1475-6773.01102 Whom Should We Profile? Examining Diabetes Care Practice Variation among Primary Care Providers, Provider Groups, and Health Care Facilities Sarah L. Krein, *Timothy P. Hofer, Eve A. Kerr, and Rodney A. Hayward

10. Resources available VA Diabetes Registry Project (1998-2001) Automated Clinical Databases Data warehouse (VA Healthcare and analysis group) Database Components Encounter records (OPC/PTF ) Outpatient Pharmacy Lab primary care provider database (PCMM () Vitals Cohort identification procedure Data quality and measure validation Kerr EA, et al. Journal on Quality Improvement 2002; 28(10):555-65.

11. Selected Measures: Resource Use Cost of hypoglycemic medications Cost of home glucose monitoring for patients not on insulin Cost of calcium channel blockers Quality Processes OutcomesIntermediate Outcomes

12. Selected Measures : Intermediate Outcomes Last A1c value A1c  9.5% Last LDL value LDL  3.6 mmol/L (140mg/dl) Quality Processes OutcomesIntermediate Outcomes

13. Selected Measures: Process Measures Hemoglobin A1c obtained LDL-C obtained Lipid profile obtained Quality Processes OutcomesIntermediate Outcomes

14. Selected Measures: Mixed or Linked Measure LDL  3.6 mmol/L (140mg/dl) or on a statin

15. Are there differences between physicians? What are the sources of variation? Noise Unmeasured differences Physician effects Clinic or group effects Health System/payor effects

16. Outcomes

17. Intermediate outcomes

18. Process measures

19. Physician effect size

20. Panel size Variance attributable to level of care.02.04.08.10 0 50 100 150 200 Median PCP Panel size in study sample Last LDL-C Value (1%) Cost of home glucose monitoring for patients not on Insulin Last LDL-C value <3.6 mmol/L or on a statin (5%) Hemoglobin A1c obtained (8%) PCP Effect Negligible SmallModerate

21. Implicit chart review – site level Trained physician reviewers 621 records 26 clinical sites

22. Conclusions Measurement models are fundamentally important to measuring and profiling quality. There is often little reason or capability to profile at the physician level. Profiles that ignore measurement error Misrepresent the variability in quality Are difficult (or impossible) to validate

24. Example – the imprecise thermometer Budget cuts inspire innovation in the clinic

25. Observed temperature

26. Observed vs. true temperature

27. Strength in numbers 85 90 95 100 105 Body temperature(F) trueobservedaverage

28. Scale transformation

29. Reliability “A person with one watch knows what time it is” “A person with two watches is never quite sure”

30. Effect of gaming