1. Disability Management and the Effects of Workplace Interventions: A Meta-Analysis IFDM GLADNET-IDMRN Symposium John Lui Norm Hursh David Rosenthal

2. meta-analysis2 Our underlying interests… DM is acknowledged as a proven approach that effectively decreases the economic costs and human impact of disability Earliest DM practices targeted RTW outcomes as a cornerstone of DM BUT, no meta-analyses examining an aggregation of empirical studies has been conducted

3. meta-analysis3 Our underlying interests… DM is now a global movement, with developing standards, and accepted practices Early RTW practices have expanded, DM has evolved, yet benefit systems across countries vary widely We wanted to look at the state-of-the- art outcome studies conducted internationally

4. meta-analysis4 Our underlying interests… Our thinking was…lets look at the most likely DM practice that most countries adopt How strong is the research around what we may find to be most consistently defined practice What are the drivers of evidence-based practice…of developing best-practice approaches to our work

5. meta-analysis5 Meta-analysis Gold standard of best practice in medicine –In medicine, with its positivist scientific methods tradition, the gold standard for scientific evidence is still randomized clinical trials and the method of choice for determining the cumulative evidence of the effectiveness of a treatment is meta-analysis. Randomized clinical trials Meta-analysis

6. meta-analysis6 Study Authors David A. Rosenthal University of Wisconsin- Madison John Lui University of Wisconsin- Stout Norm Hursh Boston University Li Jialiang National University -Singapore James A. Curcio Hewitt Associates

7. meta-analysis7 This study applied meta-analytic processes to investigate the aggregate effect size of the studies that met criteria for inclusion in the research. Initial aggregation processes are within studies (when necessary) and subsequently, across studies, both providing ES estimates.

8. meta-analysis8 Criteria for inclusion of studies consisted of six areas: 1) population of interest 2) nature of intervention or strategy 3) provider of intervention 4) receiver of intervention 5) outcomes and 6) study design – quantitative

9. meta-analysis9 Populations of interest include a) workers who are off work due to MSK conditions; pain related conditions that were episodic or non-episodic or associated with a degenerative or nondegenerative condition; chronic pain; or b) the workers compensation claimant population.

10. meta-analysis10 The category of nature of intervention or strategy is specifically aimed at improving RTW outcomes, including disability management; case management; education to workplace staff, insurance case managers or workers; changes in general organizational factors, but specifically aimed at improving RTW outcomes and decreasing durations of disability and associated direct labor costs..

11. meta-analysis11 Providers of intervention come from the workplace, or from an insurance company (private or governmental) or third-party claims administrator (TPA) which could be provided from the workplace. They could also come from healthcare providers in very close collaboration with the workplace.

12. meta-analysis12 Receivers of intervention include workers, workplace staff, and insurance company/TPA case managers.

13. meta-analysis13 Outcomes include work disability durations, described as self-reported time to return to work, time on wage replacement benefits (Workers Comp or Short/Long-Term Disability), total duration of lost time (in calendar or business days), total labor cost of direct lost time, recurrences; and point- prevalence of status (e.g., back at work versus not back at work).

14. meta-analysis14 Other outcomes include the associated costs of healthcare, indirect costs associated with wage replacement and intervention (training, administrative, etc.); as well as the quality of life, including mental health, functional status, general physical health during and/or after work interruption, quality of work life, medication taking during and/or after work interruption.

15. meta-analysis15 Search EBSCOhost was used as a primary searching reference system. Four databases were searched for relevant studies including CINAHL Plus with Full Text, MEDLINE, PsycARTICLES, and PsycINFO between January 1996 and October 2009. In addition, relevant research institutes, including Institute for Work and Health (IWH), and National Institute for Occupational Safety and Health (NIOSH) were also explored.

16. meta-analysis16 Terms used in searching included workers compensation, workplace intervention, return to work, return to work and intervention, disability management, short-term disability, long-term disability,.standardized metrics, benefit program performance, and benefit program evaluation.. In total, 2765 papers were identified by the search.

17. meta-analysis17 Study designs must be quantitative and include randomized controlled trial (RCT), nonrandomized trial, cross- sectional, pre-postdesign, time series, case control, and cohorts (retrospective and prospective).

18. meta-analysis18 Our Criteria for exclusion of studies consisted of: 1) population of interest Populations of interest include mental health conditions as a primary condition, phantom limb pain, short duration of self-limiting pain, and pain associated with a malignant condition.

19. meta-analysis19 Exclusion 2) nature of intervention or strategy –Natures of intervention or strategy include policies, primary prevention, workplace ergonomic interventions, and clinical interventions provided outside the workplace.

20. meta-analysis20 Exclusion 3) provider of intervention –Providers of intervention come from healthcare providers with no or minimal integration with the workplace.

21. meta-analysis21 Exclusion 4) outcomes –Outcomes include absenteeism unrelated to MSK or other pain-related conditions.

22. meta-analysis22 Exclusion 5) study design – not quantitative or quantitative consisting of non- comparative studies, such as case series and case studies.

23. meta-analysis23 Studies included (n=15): US – 4 Norway - 1 Netherlands – 3 Sweden – 2 Finland – 1 North America – 1 Canada – 3

24. meta-analysis24 Results we obtained an aggregate effect size d+ = 0.372 What does this mean?

25. meta-analysis25 More specifically, computation of an aggregate effect size for each study was accomplished on a Microsoft Excel worksheet. For the 15 studies retrieved for the Disability Management interventions in business and industry that met the criteria for workplace interventions to address managing disability in the workplace and measurable outcome criteria such as cost savings or reduced lost days at work.

26. meta-analysis26 we obtained an aggregate effect size d+ = 0.372, indicating that at the conclusion of treatment, workers in the intervention groups scored, on average, nearly half of a standard deviation higher than workers not receiving such interventions.

27. meta-analysis27 Meta-analysis Process Issues Comprehensive review Inclusion criteria –Finding studies – data bases –Publication bias –Significance bias

28. meta-analysis28 The Tyranny of P Values Cohen (1994) argued that the results of statistical significance tests (i.e., the test statistic and associated p value) are not valid indicators of study findings, primarily because they are largely determined by the sample size of the study. Meta-analysis goes beyond this illusion of conflicting findings (Hunter & Schmidt, 1996, p. 326) by ignoring p values in favor of an index of the strength of association between variables, referred to as the observed effect size for each study.

29. meta-analysis29 Sample size Effect sizes derived from small N studies tend to have large sampling variances, and correspondingly large standard errors of estimate (reflecting relatively low precision). Effect sizes derived from large N studies have relatively small sampling variances, correspondingly small standard errors, reflecting their relatively greater precision. For this reason, the inverse of the variance for each effect size serves as a weight in meta-analysis. This procedure gives greater weight to more precise effect estimates and lesser weight to less precise effect estimates in computing the aggregate d+. (See VCV matrix in Excel example)

30. meta-analysis30 Effect sizes (Cohen) Standardized Mean Difference Effect Size –large effect:d =.80 –medium effect:d =.50 –small effect:d =.20

31. meta-analysis31 Fixed Versus Random Effects Used Random effects versus fixed (two sources of error - model for generalizability). In a random-effects meta-analysis, the variance of study effect sizes has two components: –variance due to sampling error (estimated in the effect size variances computed earlier) and – –random effects variance component reflecting differences in effect sizes attributable to systematic differences between studies (Raudenbush, 1994).

32. meta-analysis32 Q Statistic – Test of Homogeneity Examines whether the variability among study effect sizes is greater than would be expected due to chance (i.e., to sampling error) alone. The Q statistic represents a ratio of between-studies variance to within-study variance, analogous to the F ratio in ANOVA. The Q statistic has (k – 1) degrees of freedom (where k is the number of studies included in the meta-analysis) to determine whether the null hypothesis (i.e., variability among observed effect sizes is attributable to sampling error alone) can be rejected at a desired probability level (usually a =.05) (Shadish & Haddock, 1994).

33. meta-analysis33 - Q Statistic – Test of Homogeneity For this study data, Q(k-1=14) = 16.82, p >.05, so the null hypothesis could not be rejected. That is, the variance among study effect sizes was not greater than expected based on sampling error, so that there is no warrant for suspecting that systematic differences among studies contributed to differences in their obtained effect sizes.

34. meta-analysis34 Additional analyses Also will conduct moderator Analyses - (a) type of observation - behavioral performance, or observer judgment) and (b) hours/amount of intervwention provided. Moderator analyses using both continuous (hours of treatment) and categorical (type of observation) variables.

35. meta-analysis35 Implications for ongoing international research Lets discuss: –What are outcome research of stakeholders from different countries, different systems –How valued is research by different stakeholders –How can we develop a common language related to DM outcomes to promote research

36. meta-analysis36 Implications for ongoing international research Defining DM – what does it mean to different countries when disability benefit systems and healthcare systems are so different Is DM passe? How does it Line up for employers who promote wellnes With these differences, how do we align collaborative international research to talk the same language…what are common metrics and metrics of WHAT

37. meta-analysis37 Other questions?