Reliability and responsiveness of outcome measures in a stroke clinical trial Beverley Turtle 1, Dr Alison Porter-Armstrong 1 and Dr May Stinson 1 1 Centre for Health and Rehabilitation Technologies, Institute of Nursing and Health, School of Health Sciences, Ulster University (NI)

Contents Background Aim and objectives Reproducibility Method Results Discussion Responsiveness Implications for Research and Practice

Background Stroke is a major cause of complex disability.1 Upper limb impairment especially prevalent.2 Impairment of upper limb function linked to: Independence in activities of daily living.3 Quality of life.1 Lack of evidence to indicate most effective upper limb rehabilitation tool.4

Reproducibility – reliability and agreement Rationale Pilot randomised controlled trial (RCT), examining effects of mirror therapy on upper limb function in sub-acute stroke patients. Primary outcome measure: Functional Independence Measure (FIM). Secondary outcome measure: graded Wolf Motor Function Test (gWMFT). Essential measurement properties: Validity Reproducibility – reliability and agreement Responsiveness

Aim and Objectives Aim To inform development of a main RCT, through examination of the psychometric properties of functional outcome measures used in a pilot RCT. Objectives To examine reliability and agreement of the gWMFT. To examine responsiveness of the FIM and gWMFT

What is Reproducibility? Reliability Variability between participant scores.5 Agreement Degree to which rater scores are identical. 5 Inter-rater: Intra-rater: Patient Patient Rater 1 Rater 2 Rater 3 Rater 1

Method Reliability and agreement of gWMFT Participants Pilot study form sample subset (n=30). Inclusion criteria: Adults aged 18 years plus admitted to an inpatient rehabilitation ward. Stroke diagnosis within three months. Upper limb motor loss. Able to understand/follow two-part verbal and written commands. Exclusion criteria: Previous stroke or gross cognitive impairment. Ethical Approval: Office for Research and Ethics Committees, Northern Ireland (Ref:14/NI/1149). Research governance granted by Health and Social Care Trust. Written informed consent obtained from participants. Additional informed consent for recording video footage.

Method Reliability and agreement of gWMFT Outcome Measure- gWMFT: Assessment of upper limb function, moderate to severe impairment.6 13 graded items (level A, level B). Two scores: Functional ability scale (FAS) and performance time. Participants scored by videotape. Limited psychometric evidence available. High intra-rater reliability.7

Method Reliability and agreement of gWMFT Outcome Measure – gWMFT

Method Reliability and agreement of gWMFT Rater 1 Rater 2 Rater 2 Procedure: Inter-rater analyses – two weeks Intra-rater analyses – three months Patient Patient Rater 1 Rater 2 Rater 2 One month later Internal consistency- based on scoring by rater 1. Data analysis: Reliability analyses: Intraclass correlation coefficients. Agreement analyses: Standard error of measurement (SEM). Internal consistency: Cronbach’s alpha

Results Reliability and agreement of gWMFT Characteristic Inter-rater Intra-rater (n = 21) Sex Male, n Female, n   18 10 16 5 Age in years, Mean (SD) 71.3 (9.6) 70.5 (8.7) Side of hemiplegia Left, n Right, n 15 6

Inter-rater reliability Intra-rater reliability Results Reliability and agreement of gWMFT   Inter-rater reliability ICC2,1 (95% CI) Intra-rater reliability ICC3,1 (95% CI) SEM Inter-rater Intra-rater Internal Consistency Two weeks Three months Functional ability 0.979 (0.955-0.990) 0.993 (0.983-0.997) 0.33 0.19 0.988 0.990 Performance time 0.986 (0.970-0.993) 0.996 (0.990-0.998) 6.49 3.64 0.979 0.985 ICC, intraclass correlation coefficient; 95%CI, 95% confidence interval

Discussion Reliability and agreement of gWMFT Good inter- and intra-rater reliability for performance time and FAS - indicating similar ranking of participants. Adequate agreement for FAS. Greater measurement error for performance time. Good internal consistency.

What is responsiveness? Ability to detect meaningful change Internal Responsiveness 8 : External Responsiveness 8 : Discharge

“…occupational therapy…of…benefit…to the return of movement…?” Method Responsiveness of FIM and gWMFT Participants: Full pilot study sample (n=40) Internal responsiveness- difference in scores across time intervals; baseline to discharge, discharge to three months. External responsiveness- participant perceived change as external anchor. “…occupational therapy…of…benefit…to the return of movement…?”

Method Responsiveness of FIM and gWMFT Outcome measure-FIM: Assessment of level of independence completing activities of daily living 18 items (13 motor, 5 cognitive). Ordinal scale score, 1=full assistance to 7=full independence. Good level of reliability9, validity10 and responsiveness.11 Ceiling effects found for cognitive sub-scale. 11 Outcome Measure- gWMFT: As previously discussed.

Method Responsiveness of FIM and gWMFT Data analysis Floor/ceiling effects: ≥15% participants min./max. scores.12 Internal responsiveness: Standardised response mean (SRM). External responsiveness: Receiver operating characteristic (ROC) curve. Area under curve (AUC). Value of 1=perfect ability to discriminate meaningful change. Values of ≤0.5=no ability to discriminate.

Results Responsiveness of FIM and gWMFT Baseline characteristics for sample group Characteristics Total sample (n=40) Age in years, mean (SD) 71.25 (9.23) Gender Male, n (%) Female, n (%)   27 (67.5%) 13 (32.5%) Side of hemiplegia Left, n (%) Right, n (%) 24 (60%) 16 (40%)

Results Baseline (n=40) Discharge (n=35) Three Months FIM (n=32)   Baseline  (n=40) Discharge (n=35) Three Months FIM (n=32) gWMFT (n=30) FIM total Mean (SD) 64.35 (9.28) 78.74 (11.6) 97.88 (18.97) FIM motor 31.4 (7.12) 44.89 (11.42) 64.13 (18.46) FIM cognitive Ceiling effect, % 32.95 (3.73) 25% 33.86 (0.97) 22.86% 33.75 (1.3) 18.75% gWMFT FAS Floor effect, % 2.73 (2.27) 40% 0% 4.01 (2.48) 14.29% 2.86% 4.78 (2.44) 10% 20% gWMFT time 58.92 (56.00) 42.5% 43.81 (54.7) 27.5% 33.93 (50.77) 23.33%

Three month - discharge Results Responsiveness of FIM and gWMFT Mean differences and SRM values for each time interval Measure Discharge - baseline Three month - discharge Mean (SD) SRM FIM total 13.60 (8.74) 1.56 18 (14.83) 1.2 FIM motor 13.46 (8.65) 17.97 (14.65) 1.23 FIM cognitive 0.14 (0.88) 0.16 -0.16 (0.78) -0.21 gWMFT FAS 1.31 (1.27) 1.03 0.62 (1.11) 0.56 gWMFT time -16.02 (34.25) -0.47 -6.81 (24.16) -0.28

Results Responsiveness of FIM and gWMFT External responsiveness Outcome measure AUC 95% CI FIM total 0.581 0.344-0.817 FIM motor 0.566 0.328-0.804 FIM cognitive 0.616 0.405-0.827 gWMFT performance time 0.583 0.368-0.799 gWMFT FAS 0.740 0.541-0.938 AUC, area under the curve; 95%CI. 95% confidence interval

Discussion Responsiveness of FIM and gWMFT Internal Responsiveness High levels of responsiveness for FIM motor, FIM total and gWMFT FAS. Moderate level of responsiveness for gWMFT performance time. Poor responsiveness demonstrated for FIM cognitive scores. External responsiveness gWMFT FAS is sensitive to patient-perceived meaningful change.

Implications for Research and Practice Highlighted issues with use of FIM cognitive sub-scale in stroke rehabilitation. Include cognitive screening tool. gWMFT is a reliable, responsive outcome measure for use in an in-patient stroke rehabilitation setting. Standardised training with refresher courses. gWMFT FAS sensitive to patient-perceptions of upper limb improvement. Variety of external anchors.13

Thank you for listening. Any questions? Email: turtle-b2@ulster.ac.uk

References   Mendis, S. (2013) Stroke disability and rehabilitation of stroke: World Health Organization perspective. International Journal of Stroke, 8(1), 3-4. Lawrence, E.S., Coshall, C., Dundas, R., Stewart, J., Rudd, A.G., Howard, R. and Wolfe, C.D. (2001) Estimates of the prevalence of acute stroke impairments and disability in a multiethnic population. Stroke, 32(6), 1279-1284. Lang, C.E., Bland, M.D., Bailey, R.R., Schaefer, S.Y. and Birkenmeier, R.L. (2013) Assessment of upper extremity impairment, function, and activity after stroke: foundations for clinical decision making. Journal of Hand Therapy, 26(2), 104-115. Pollock, A., Farmer Sybil E, Brady Marian C, Langhorne, P., Mead Gillian E, Mehrholz, J. and van Wijck, F. (2014) Interventions for improving upper limb function after stroke. Cochrane Database of Systematic Reviews, Issue 11. Art. No.: CD010820. doi: 10.1002/14651858.CD010820.pub2. Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B.J., Hróbjartsson, A., Roberts, C., Shoukri, M. and Streiner, D.L. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64(1), 96-106. Constraint-Induced Movement Therapy Research Group. (2002) Manual: Graded Wolf Motor Function Test. Birmingham: University of Alabama and Birmingham’s Veteran’s Administration Centre. Bonifer, N.M., Anderson, K.M. and Arciniegas, D.B. (2005) Constraint-induced movement therapy after stroke: efficacy for patients with minimal upper-extremity motor ability. Archives of Physical Medicine and Rehabilitation, 86(9), 1867-1873. Husted, J.A., Cook, R.J., Farewell, V.T. and Gladman, D.D. (2000) Methods for assessing responsiveness: a critical review and recommendations. Journal of Clinical Epidemiology, 53(5), 459-468. Ottenbacher, K.J., Hsu, Y., Granger, C.V. and Fiedler, R.C. (1996) The reliability of the functional independence measure: a quantitative review. Archives of Physical Medicine and Rehabilitation, 77(12), 1226-1232. Hsueh, I. P., Lin, J. H., Jeng, J. S., & Hsieh, C. L. (2002). Comparison of the psychometric characteristics of the functional independence measure, 5 item Barthel index, and 10 item Barthel index in patients with stroke. Journal of Neurology, Neurosurgery & Psychiatry, 73(2), 188-190. Schepers, V. P. M., Ketelaar, M., Visser-Meily, J. M. A., Dekker, J., & Lindeman, E. (2006). Responsiveness of functional health status measures frequently used in stroke research. Disability and rehabilitation, 28(17), 1035-1040. McHorney, C. A., and Tarlov, A. R. (1995) Individual-patient monitoring in clinical practice: are available health status surveys adequate? Quality of Life Research, 4(4), 293-307. Revicki, D., Hays, R. D., Cella, D., and Sloan, J. (2008) Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61(2), 102-109. .