1. Reliability and responsiveness of outcome measures in a stroke clinical trial
Beverley Turtle 1, Dr Alison Porter-Armstrong 1 and Dr May Stinson 1
1 Centre for Health and Rehabilitation Technologies, Institute of Nursing and Health, School of Health Sciences, Ulster University (NI)

2. Contents Background Aim and objectives Reproducibility Method Results
Discussion
Responsiveness
Implications for Research and Practice

3. Background Stroke is a major cause of complex disability.1
Upper limb impairment especially prevalent.2
Impairment of upper limb function linked to:
Independence in activities of daily living.3
Quality of life.1
Lack of evidence to indicate most effective upper limb rehabilitation tool.4

4. Reproducibility – reliability and agreement
Rationale
Pilot randomised controlled trial (RCT), examining effects of mirror therapy on upper limb function in sub-acute stroke patients.
Primary outcome measure: Functional Independence Measure (FIM).
Secondary outcome measure: graded Wolf Motor Function Test (gWMFT).
Essential measurement properties:
Validity
Reproducibility – reliability and agreement
Responsiveness

5. Aim and Objectives
Aim
To inform development of a main RCT, through examination of the psychometric properties of functional outcome measures used in a pilot RCT.
Objectives
To examine reliability and agreement of the gWMFT.
To examine responsiveness of the FIM and gWMFT

6. What is Reproducibility?
Reliability
Variability between participant scores.5
Agreement
Degree to which rater scores are identical. 5
Inter-rater: Intra-rater:
Patient
Patient
Rater 1
Rater 2
Rater 3
Rater 1

7. Method Reliability and agreement of gWMFT Participants
Pilot study form sample subset (n=30).
Inclusion criteria:
Adults aged 18 years plus admitted to an inpatient rehabilitation ward.
Stroke diagnosis within three months.
Upper limb motor loss.
Able to understand/follow two-part verbal and written commands.
Exclusion criteria:
Previous stroke or gross cognitive impairment.
Ethical Approval:
Office for Research and Ethics Committees, Northern Ireland (Ref:14/NI/1149).
Research governance granted by Health and Social Care Trust.
Written informed consent obtained from participants.
Additional informed consent for recording video footage.

8. Method Reliability and agreement of gWMFT Outcome Measure- gWMFT:
Assessment of upper limb function, moderate to severe impairment.6
13 graded items (level A, level B).
Two scores: Functional ability scale (FAS) and performance time.
Participants scored by videotape.
Limited psychometric evidence available. High intra-rater reliability.7

9. Method
Reliability and agreement of gWMFT
Outcome Measure – gWMFT

10. Method Reliability and agreement of gWMFT Rater 1 Rater 2 Rater 2
Procedure:
Inter-rater analyses – two weeks
Intra-rater analyses – three months
Patient
Patient
Rater 1
Rater 2
Rater 2
One month later
Internal consistency- based on scoring by rater 1.
Data analysis:
Reliability analyses: Intraclass correlation coefficients.
Agreement analyses: Standard error of measurement (SEM).
Internal consistency: Cronbach’s alpha

11. Results Reliability and agreement of gWMFT Characteristic Inter-rater
Intra-rater
(n = 21)
Sex
Male, n
Female, n 18 10 16 5
Age in years, Mean (SD)
71.3 (9.6)
70.5 (8.7)
Side of hemiplegia
Left, n
Right, n 15 6

12. Inter-rater reliability Intra-rater reliability
Results
Reliability and agreement of gWMFT
Inter-rater reliability
ICC2,1 (95% CI)
Intra-rater reliability
ICC3,1 (95% CI)
SEM
Inter-rater Intra-rater
Internal Consistency
Two weeks Three months
Functional ability
0.979 ( )
0.993 ( )
0.33
0.19
0.988
0.990
Performance time
0.986 ( )
0.996 ( )
6.49
3.64
0.979
0.985
ICC, intraclass correlation coefficient; 95%CI, 95% confidence interval

13. Discussion
Reliability and agreement of gWMFT
Good inter- and intra-rater reliability for performance time and FAS - indicating similar ranking of participants.
Adequate agreement for FAS.
Greater measurement error for performance time.
Good internal consistency.

14. What is responsiveness?
Ability to detect meaningful change
Internal Responsiveness 8 : External Responsiveness 8 :
Discharge

15. “…occupational therapy…of…benefit…to the return of movement…?”
Method
Responsiveness of FIM and gWMFT
Participants:
Full pilot study sample (n=40)
Internal responsiveness- difference in scores across time intervals; baseline to discharge, discharge to three months.
External responsiveness- participant perceived change as external anchor.
“…occupational therapy…of…benefit…to the return of movement…?”

16. Method Responsiveness of FIM and gWMFT Outcome measure-FIM:
Assessment of level of independence completing activities of daily living
18 items (13 motor, 5 cognitive).
Ordinal scale score, 1=full assistance to 7=full independence.
Good level of reliability9, validity10 and responsiveness.11 Ceiling effects found for cognitive sub-scale. 11
Outcome Measure- gWMFT:
As previously discussed.

17. Method Responsiveness of FIM and gWMFT Data analysis
Floor/ceiling effects: ≥15% participants min./max. scores.12
Internal responsiveness: Standardised response mean (SRM).
External responsiveness: Receiver operating characteristic (ROC) curve. Area under curve (AUC).
Value of 1=perfect ability to discriminate meaningful change.
Values of ≤0.5=no ability to discriminate.

18. Results Responsiveness of FIM and gWMFT
Baseline characteristics for sample group
Characteristics
Total sample (n=40)
Age in years, mean (SD)
71.25 (9.23)
Gender
Male, n (%)
Female, n (%)
27 (67.5%)
13 (32.5%)
Side of hemiplegia
Left, n (%)
Right, n (%)
24 (60%)
16 (40%)

19. Results Baseline (n=40) Discharge (n=35) Three Months FIM (n=32)
Baseline
 (n=40)
Discharge
(n=35)
Three Months
FIM (n=32)
gWMFT (n=30)
FIM total
Mean (SD)
64.35 (9.28)
78.74 (11.6)
97.88 (18.97)
FIM motor
31.4 (7.12)
44.89 (11.42)
64.13 (18.46)
FIM cognitive
Ceiling effect, %
32.95 (3.73)
25%
33.86 (0.97)
22.86%
33.75 (1.3)
18.75%
gWMFT FAS
Floor effect, %
2.73 (2.27)
40% 0%
4.01 (2.48)
14.29%
2.86%
4.78 (2.44)
10%
20%
gWMFT time
58.92 (56.00)
42.5%
43.81 (54.7)
27.5%
33.93 (50.77)
23.33%

20. Three month - discharge
Results
Responsiveness of FIM and gWMFT
Mean differences and SRM values for each time interval
Measure
Discharge - baseline
Three month - discharge
Mean (SD)
SRM
FIM total
13.60 (8.74)
1.56
18 (14.83)
1.2
FIM motor
13.46 (8.65)
17.97 (14.65)
1.23
FIM cognitive
0.14 (0.88)
0.16
-0.16 (0.78)
-0.21
gWMFT FAS
1.31 (1.27)
1.03
0.62 (1.11)
0.56
gWMFT time
(34.25)
-0.47
-6.81 (24.16)
-0.28

21. Results Responsiveness of FIM and gWMFT External responsiveness
Outcome measure
AUC
95% CI
FIM total
0.581
FIM motor
0.566
FIM cognitive
0.616
gWMFT performance time
0.583
gWMFT FAS
0.740
AUC, area under the curve; 95%CI. 95% confidence interval

22. Discussion Responsiveness of FIM and gWMFT Internal Responsiveness
High levels of responsiveness for FIM motor, FIM total and gWMFT FAS.
Moderate level of responsiveness for gWMFT performance time.
Poor responsiveness demonstrated for FIM cognitive scores.
External responsiveness
gWMFT FAS is sensitive to patient-perceived meaningful change.

23. Implications for Research and Practice
Highlighted issues with use of FIM cognitive sub-scale in stroke rehabilitation.
Include cognitive screening tool.
gWMFT is a reliable, responsive outcome measure for use in an in-patient stroke rehabilitation setting.
Standardised training with refresher courses.
gWMFT FAS sensitive to patient-perceptions of upper limb improvement.
Variety of external anchors.13

24. Thank you for listening. Any questions?

25. References
Mendis, S. (2013) Stroke disability and rehabilitation of stroke: World Health Organization perspective. International Journal of Stroke, 8(1), 3-4.
Lawrence, E.S., Coshall, C., Dundas, R., Stewart, J., Rudd, A.G., Howard, R. and Wolfe, C.D. (2001) Estimates of the prevalence of acute stroke impairments and disability in a multiethnic population. Stroke, 32(6),
Lang, C.E., Bland, M.D., Bailey, R.R., Schaefer, S.Y. and Birkenmeier, R.L. (2013) Assessment of upper extremity impairment, function, and activity after stroke: foundations for clinical decision making. Journal of Hand Therapy, 26(2),
Pollock, A., Farmer Sybil E, Brady Marian C, Langhorne, P., Mead Gillian E, Mehrholz, J. and van Wijck, F. (2014) Interventions for improving upper limb function after stroke. Cochrane Database of Systematic Reviews, Issue 11. Art. No.: CD doi: / CD pub2.
Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B.J., Hróbjartsson, A., Roberts, C., Shoukri, M. and Streiner, D.L. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64(1),
Constraint-Induced Movement Therapy Research Group. (2002) Manual: Graded Wolf Motor Function Test. Birmingham: University of Alabama and Birmingham’s Veteran’s Administration Centre.
Bonifer, N.M., Anderson, K.M. and Arciniegas, D.B. (2005) Constraint-induced movement therapy after stroke: efficacy for patients with minimal upper-extremity motor ability. Archives of Physical Medicine and Rehabilitation, 86(9),
Husted, J.A., Cook, R.J., Farewell, V.T. and Gladman, D.D. (2000) Methods for assessing responsiveness: a critical review and recommendations. Journal of Clinical Epidemiology, 53(5),
Ottenbacher, K.J., Hsu, Y., Granger, C.V. and Fiedler, R.C. (1996) The reliability of the functional independence measure: a quantitative review. Archives of Physical Medicine and Rehabilitation, 77(12),
Hsueh, I. P., Lin, J. H., Jeng, J. S., & Hsieh, C. L. (2002). Comparison of the psychometric characteristics of the functional independence measure, 5 item Barthel index, and 10 item Barthel index in patients with stroke. Journal of Neurology, Neurosurgery & Psychiatry, 73(2),
Schepers, V. P. M., Ketelaar, M., Visser-Meily, J. M. A., Dekker, J., & Lindeman, E. (2006). Responsiveness of functional health status measures frequently used in stroke research. Disability and rehabilitation, 28(17),
McHorney, C. A., and Tarlov, A. R. (1995) Individual-patient monitoring in clinical practice: are available health status surveys adequate? Quality of Life Research, 4(4),
Revicki, D., Hays, R. D., Cella, D., and Sloan, J. (2008) Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61(2), .