1. Quality Measures for Rehabilitation: Policy, Provider and Patient Perspectives Measuring Clinical Change: Quality Indicators ACRM-ASNR Pre-Conference Institute Kenneth J. Ottenbacher University of Texas Medical Branch

2. Disclosures National Institutes of Health Grants: R24 HD065702; R01 AG017638; K12 HD055929; T32 HD007539 PI: Kenneth J. Ottenbacher National Center for Medical Rehabilitation Research (NICHD), National Institute on Neurological Disorders and Stroke, and National Institute on Aging Department of Education Grants: H133G080163; H133P110012 PI: Kenneth J. Ottenbacher National Institute on Disability and Rehabilitation Research

3. Learning Objectives Describe three approaches to measuring clinical change Discuss the difference between clinical change at the individual level and facility level Identify challenges in measuring clinical change

4. How NOT to measure clinically important change: Measuring clinically important change is complex. Multiple methods are available with various advantages and disadvantages. p<.05

5. Statistical Significance versus Clinical Importance Design sensitivity: statistical power for experimental research Mark W. Lipsey The Cult of Statistical Significance Stephen T. Ziliak & Deirdre N. McCloskey

6. Investigators and clinicians have a long history of interpreting statistically significant findings as also being clinically significant. Example: Smith reports a rehabilitation outcome study with statistically significant results. Jones conducts a replication study with (null) non-significant results.

7. Assume that both trials are well-designed and their internal validity is good. Results: Smith, t = 2.21 (df = 78, p < 0.05) Jones, t = 1.06 (df = 18, p = ns)

8. The interpretation is that Smith found a statistically significant difference (change) in the outcome measure and Jones did not find a statistically significant difference (change). And… that the treatment produced a significant change in the Smith study, but not the Jones study.

9. An alternative way to look at the results of the Smith and Jones studies is to examine the degree to which the null hypothesis is false in both investigations. Cohen (1988) refers to this as the “effect size.”

10. If we compute an effect size for each study, we will find the following: Formula: d = 2(t)/  df Smith d = 0.50 Jones d = 0.50

11. The effect size is the same for the two trials - the degree to which the null hypothesis is false is identical for both the Smith and Jones study. The amount of change associated with the outcome is the same in both studies. Why are the findings in one study statistically significant and the other study non-significant?

12. Things are not always what they appear to be. Power for the Jones trial = 0.18 The chance of a Type 2 error is 0.82 (82%) From: Day, R.H. (1972). Science, Vol. 175, p.1335.

13. Reading made Don Quixote a gentleman, but believing what he read made him insane. G. B. Shaw

14. Who is the Target: 1.Assessment of individual change 2.Assessment of facility change 3.Assessment of population change Measuring Clinical Change

15. Approaches to Assessing (individual) Change Comparison of change in raw scores or ratings Effect size (standardized mean difference) Reliable change index (Minimal detectable change) Minimal clinically important difference Standardized regression-based change scores

16. There are two approaches to measuring (individual) change. Distribution-based approaches that rely on expressing change scores in terms of an underlying sampling distribution. Anchor-based approaches that require an external, independent standard to ‘anchor’ the meaning of clinical importance.

17. Examples: Reliability Change Index (RCI) Need the following information: a) baseline score, b) follow-up score, and c) the standard error of measurement (SEM) for the test. (SEM = S 1  1-r) where S 1 = standard deviation, r = reliability coefficient. RCI = (X 2 – X 1 ) / SEM The RCI provides an estimate of whether a change is greater than would be expected by chance (measurement error).

18. Examples: Minimal Clinically Important Difference (MCID) Requires identifying a clinically important external standard. For example, 5 motor FIM points. 5 FIM points represents ~ 5% of total scale range on the motor domain. This standard can be based on clinical experience, research literature, patient goals, or some combination. Use of UDS burden of care values to determine external anchor.

19. Mean Decrease in Minutes of Help Per Day per FIM TM Point Change Spinal Cord Injury Study, 1990-91 Hamilton et. al. 13 - 2626 - 3939 - 5252 - 6565 - 7878 - 91 0 1 2 3 4 5 6 Minutes of Help Minutes per FIM Point Motor FIM Score Ranges UDS MR

20. Converting individual measures of clinical change to facility or population measures of clinical change (quality indicators). Examples: Percent of persons in a Case Mix Group achieving Minimal Detectable Change or Reliability Change Index criteria. Proportion of persons in a Case Mix Group achieving a Minimal Clinically Important Difference.

21. Examples (continued): Percent of persons in a Case Mix Group achieving above benchmark expectations. Polar Graph Admission Discharge Benchmark Ea Gr Ba DU DL To Bl Bo Ch T ToT TuT WC St Co Ex Si Ps M e UDS MR

22. Challenges: 1. How to risk Adjust? 2. How to make the process patient centered? 3. What are the unintended consequences?

23. The End Doing outcomes research is a lot like raising children… you always think you are going to do a better job next time. http://rehabsciences.utmb.edu/R24/ACRM.asp