1. Measurement invariance of the Beck Depression Inventory in assessing and comparing treatment effects Marjolein Fokkema, MSc Niels Smits, PhD Henk Kelderman, PhD Pim Cuijpers, PhD Faculty of Psychology and Education Vrije Universiteit Amsterdam, the Netherlands

2. Introduction Increasing use of routine outcome monitoring in clinical practice No big or consistent differences found between therapies in depression research Real change or response shift? Measurement invariance across timepoints

3. Data National Institute of Mental Health Treatment of Depression Collaborative Research Program First major attempt at comparing efficacy of treatments for depression: 2 psychotherapy conditions (PSY) 2 medication conditions (MED) 239 subjects entered treatment (T0), 155 subjects completed treatment (T1).

4. Beck Depression Inventory (BDI) BDI (Beck et al, 1961) assesses depressive symptoms 21 items with 4 response categories, ranging from neutral to maximum severity

5. Example: item 8 Self Accusations (0) I don’t feel I am any worse than anybody else (1) I am critical of myself for my weakness or mistakes (2) I blame myself all the time for my faults (3) I blame myself for everything bad that happens

6. Example: item 15 Work Inhibition (0) I can work about as well as before (1) It takes an extra effort to get started at doing something (2) I have to push myself very hard to do anything (3) I can’t do any work at all

7. Beck Depression Inventory (BDI) BDI (Beck et al, 1961) assesses depressive symptoms 21 items with 4 response categories, ranging from neutral to maximum severity Total score is widely used as outcome measure

9. Methods SEM approach (LISREL) –Measurement invariance between T0 and T1 : Configural invariance Metric invariance Scalar invariance Invariant uniqueness

10. Configural invariance Modeldfχ2χ2 RMSEANNFICFIAIC 1D3781177.400.1000.900.911522.87 2D3761025.490.0840.920.931259.39 3D372922.710.0770.930.941162.00

12. Invariance constraints Modeldfχ2χ2 RMSEANNFICFIAIC Configural372922.710.0770.930.941162.00 Metric390981.000.0780.93 1182.26 Partial Metric *386945.400.0780.940.931151.34 Scalar4041084.100.0830.92 1269.14 Partial scalar **397980.850.0760.93 1161.82 * Freed loadings for items 18, 19, 5 & 3 ** Freed intercepts for items 14, 7, 8, 3, 5, 21 & 9

13. Methods SEM approach (LISREL) –Measurement invariance between T0 and T1 IRT approach (MULTILOG) –Comparisons between groups (PSY and MED) and time points (T0 and T1)

14. IRT analysis (MULTILOG) Samejima’s (1969) Graded Response Model

15. Item Characteristic Curves Item 4: Lack of satisfaction

16. PSY T0 MED T0MED T1 PSY T1

17. Item Characteristic Curves Item 8: Self Accusations

18. PSY T0 MED T0MED T1 PSY T1

19. Item Characteristic Curves Item 16: Sleep Disturbance

20. PSY T0 MED T0MED T1 PSY T1

21. Item Characteristic Curves Item 19: Weight Loss

22. PSY T0 MED T0MED T1 PSY T1

23. IRT analysis (MULTILOG) Samejima’s (1969) Graded Response Model Freely estimated model, compared with model with a-parameters constrained to be equal –across time points –across groups

24. Invariance between T0 and T1 -2 times log likelihood PSYMED Free5145.85264.0 Constr. a’s5204.55313.7 Difference*58.749.7 p-value< 0.001 * ~ χ 2 (21)

25. Invariance between PSY and MED -2 times log likelihood T0T1 Free8566.31665.0 Constr. a’s8587.61665.0 Difference*21.327.9 p-value0.4410.143 * ~ χ 2 (21)

26. Discussion Response shift occured –Therapies are very effective at improving ICC’s! –Response shift seems different in different groups Future research: –Proper control group –Larger sample –Different measures for depression

27. Questions, remarks, suggestions? Thank you for your attention! m.fokkema@psy.vu.nl