1. © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde RTI International

2. Steps for Conducting Effective Evaluations 2 Centers for Disease Control

3. Types of Evaluations Adapted from: Norland, E. (2004, Sept). From education theory.. to conservation practice Presented at the Annual Meeting of the International Association for Fish & Wildlife Agencies, Atlantic City, New Jersey.

4. Theoretical/Logic Model ■ A diagram of the theory of how a program is supposed to work— a graphic depiction of relationships between activities and results (conceptual/theoretical model) ■ A logical chain of connections showing what a program intends to accomplish ■ Increases intentionality and purpose ■ Guides prioritization and allocation of resources ■ Helps to identify important variables to measure; use evaluation resources wisely ■ Supports replication

6. WFHN Evaluation Design ■ Single protocol ■ Work redesign, supervisor training and self-monitoring ■ Multiple industries and worksites ■ Healthcare (blue collar) ■ Telecommunications (white collar) ■ Group randomized field experiment ■ Intervention and comparison group assignment at the worksite or work group level ■ Adaptive randomization to balance covariates

7. Outcome Evaluation Design ■ Nested cohort design ■ Worksite partners are randomized within industry to intervention or control group ■ Outcomes evaluated at multiple levels, employees, workgroups and worksites ■ Allows for multiple levels of clustering (e.g., workgroups within worksites), and a variety of outcomes (e.g., discrete, continuous, count)

8. Formative(Process) Evaluation Framework 8 Implementation Outcomes Acceptability Adoption Appropriateness Cost Feasibility Fidelity Penetration Sustainability Implementation Outcomes Acceptability Adoption Appropriateness Cost Feasibility Fidelity Penetration Sustainability - *IOM Standards of Care What? QIs ESTs How? Implementation Strategies Implementation Research Methods Service Outcomes* Efficiency Safety Effectiveness Equity Patient- centeredness Timeliness Individual Outcomes Satisfaction Function Health status/ symptoms Proctor’s Model of Implementation Outcomes

9. Process Evaluation Data ■ Document Reviews – provide information to build an “a priori” understanding of the program content, operations, context, and program stakeholders ■ Review reports, instruments, protocols, promotional materials, patient materials, resource lists, organizational documents (org charts, flow charts, operation manuals), web sites. ■ Observational Studies – provide empirical evidence to assess program fidelity, and generate service flow and timing data to inform outcome and cost analysis ■ Key Informant Interviews – provide contextual information related to program utility, contextual factors that influence program implementation, service delivery, dispersion, and sustainability ■ Practioner and Consumer Surveys -- provide systematic data related to service delivery experiences and program related perceptions

10. Qualitative Analysis Methods ■ Recursive abstraction (Document Summaries) ■ Iterative process that generates summaries, classifications, lists, rates, or groupings ■ Deductive and Inductive Analysis (data coding) ■ Deductive “a priori” framework ■ Inductive “grounded theory” analysis ■ Results in taxonomies, themes, categories, orders ■ Comparative analysis ■ Documented vs observed processes, behaviors or outputs ■ Document changes over time (e.g., model migration) ■ Variation between processes or outputs ■ Mixed Methods (triangulation and convergence) 10

11. Process Evaluation Results  Comprehensive description of program components  Performance indicators and proficiency scores for program delivery  Common barriers and facilitators across worksites  Descriptive taxonomy of program settings  Construction of moderator variables for use in outcome and economic analyses  Program delivery protocols that are used at each worksite site

12. Outcome Evaluation Questions ■ Program Outputs ■ To what extent is the program actually performed as measured by (e.g. the number of health risk appraisals completed, percentage of employees participating in workshops, number of follow-up services delivered) ■ Proximal and Distal Outcomes ■ What is the effectiveness of the program on outcomes of interest? (changes in sleep quality, changes in bio- measures)? ■ What stakeholder and employee-level characteristics moderate the willingness and ability of performance sites and practitioners to adopt the program

13. WFHN Study Data

14. Outcome Evaluation Methods ■ Descriptive statistics generating classifications or groupings of the participating worksites, provider characteristics, employee characteristics ■ Multivariate regression examining the relationship between groups ■ Statistical modeling measuring changes on outcomes over time

15. Outcome Evaluation Results ■ Rates and frequencies of program participation ■ Performance site descriptive characteristics (including staff characteristics) that are associated with outcomes of interest ■ Within group pre to post changes in health and wellness outcomes ■ Between group comparative changes in health and wellness outcomes at multiple time points

17. Intervention Effect on Work Environment FSWE Hypotheses ■ H1. There are changes in work environment for the ■ H1a. TOMO intervention group ■ H1b. TOMO control group ■ H1c. LEEF intervention group ■ H1d. LEEF control group ■ H2. There are baseline differences in environment between intervention and control groups for the ■ H2a. TOMO industry ■ H2b. LEEF industry ■ H3. There are differences in the rate of change in climate between intervention and control groups for the ■ H3a. TOMO industry ■ H3b. LEEF industry

19. FSWE Factor Analysis

20. Growth Model Results by Industry HypothesisIndustryGroupRandom Effect EstimateStandard Errorp H1aTOMOInterventionSlope0.0420.009<0.0001 H1bTOMOControlSlope0.0140.008 0.081 H1cLEEFInterventionSlope0.0030.007 0.657 H1dLEEFControlSlope0.0010.006 0.905 H2aTOMO Intervention Vs. Control Intercept0.0330.038 0.393 H2bLEEF Intervention Vs. Control Intercept0.0130.027 0.619 H3aTOMO Intervention Vs. Control Slope0.0280.012 0.016 H3bLEEFIntervention Vs. Control Slope0.0020.009 0.804

21. Economic Evaluation Questions ■ What is the program cost to worksites and to other stakeholders? ■ What is the cost-effectiveness of the program? ■ What characteristics moderate the cost-effectiveness of the program?

22. Translational Study  An effective intervention is only useful if it is communicated to and adopted by workplaces  Complementary methods:  Use participant feedback to inform post-study messaging  Analyze process data to study employee perception for developing portrayals of the intervention  Assess potential dissemination channels  Stimulate market demand for effective intervention

23. Thank You For more information on the Work, Family, and Health Network Study http://projects.iq.harvard.edu/wfhn/people