1 The influence of the questionnaire design on the magnitude of change scores Sandra Nolte 1, Gerald Elsworth 2, Richard Osborne 2 1 Association of Dermatological Prevention Hamburg, GERMANY 2 Deakin University Melbourne, AUSTRALIA

2 The measurement of program outcomes … it is the basis for continuous quality assurance / improvement … it delivers crucial information for a wide range of stakeholders … it can / should deliver information on what works and what doesnt … is important because …

3 Bias in outcomes assessment However … while program evaluations are crucial, there are continuous concerns about: biases that may threaten the validity of outcomes data one such bias that is a common concern in pre-test / post-test data is: Response Shift (Howard 1979)

4 Response Shift Change in common metric because of redefinition, reprioritisation and/or recalibration of the target construct (Schwartz & Sprangers, 1999) Common remedy to circumvent Response Shift: collection of retrospective pre-test data [actual pre-test - retrospective pre-test] = magnitude and direction of Response Shift [post-test - retrospective pre-test] = true program outcome (Visser et al., 2005)

5 The retrospective pre-test Collected after an intervention, generally in close proximity to post-tests How good (i.e. valid, reliable) are retrospective pre-test data? Past research generally focused on comparison of retrospective pre-test with actual pre-test; however, only few tested influence of scores on each other none tested the psychometric performance of retrospective pre-tests

6 Study aim 1) To explore influence of posing retrospective pre-test questions on ratings of post-tests 2) To explore whether other types of questions influenced post-tests (i.e. transition questions)

7 Research design Setting: chronic disease self-management courses Randomised design: three versions of the Health Education Impact Questionnaire (heiQ) were distributed at post-test (randomised within courses)

8 Research design Randomised design – Version I 1) post-test ONLY (n=331) (6-point Likert scale: strongly disagree to strongly agree)

9 Group I: post-test ONLY

10 Research design Randomised design – Version II 1) post-test ONLY (n=331) (6-point Likert scale: strongly disagree to strongly agree) 2) post-test + transition questions (n=304) (transition Qs: 5-point response scale: much worse to much better)

11 Group II: post-test + transition question

12 Research design Randomised design – Version III 1) post-test ONLY (n=331) (6-point Likert scale: strongly disagree to strongly agree) 2) post-test + transition questions (n=304) (transition Qs: 5-point response scale: much worse to much better) 3) post-test + retrospective pre-test (n=314) (both 6-point Likert scale: strongly disagree to strongly agree)

13 Group III: post-test + retro pre-test

14 Results Across the three randomised groups: no significant differences in: demographic characteristics pre-test scores (= scores collected before intervention) The randomisation worked

15 Results (cont.) Posing transition questions in addition to post-test questions had hardly any influence on post-test levels (Group II) In contrast, posing retrospective pre-test questions after an intervention had significant influence on ratings of post-tests in six of the eight heiQ subscales: Post-test ONLY (Group I) mean post-test: 4.76 Post-test + retrospective pre-test (Group III) mean post-test: 4.96 (on 6-pt Likert scale)

16 Group I Mean (SD) Group II Mean (SD) Group III Mean (SD)

17 Conclusions Asking retrospective pre-test questions at post-test has a significant influence on the ratings of post-test levels The influence was so substantial that it leads to different conclusions about program effectiveness It remains uncertain whether the application of retrospective pre-tests provides a more or less accurate reflection of the impact of chronic disease self-management programs

18 Conclusions It remains uncertain whether the application of retrospective pre-tests provides a more or less accurate reflection of the impact of chronic disease self-management programs However, psychometric properties of retrospective pre-test data seem to be substantially weaker than classic pre-test Classic pre-test / post-test design may be the more valid approach to evaluate self-management programs

19 Discussion Possible explanations: 1.Cognitive task may have triggered distorted responses consistent with theories: Effort justification (Hill & Betz, 2005) Implicit theory of change (Ross, 1989) Social desirability (Crowne & Marlowe, 1964) 2.The task of remembering pre-test levels might have been too complex for some respondents making these data less reliable

20 Discussion (cont.) 3.It remains to be shown what people think while responding to questionnaires qualitative research into response processes is essential to help understand & interpret self-report data

21 Thank you