© 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde RTI International

Steps for Conducting Effective Evaluations 2 Centers for Disease Control

Types of Evaluations Adapted from: Norland, E. (2004, Sept). From education theory.. to conservation practice Presented at the Annual Meeting of the International Association for Fish & Wildlife Agencies, Atlantic City, New Jersey.

Theoretical/Logic Model ■ A diagram of the theory of how a program is supposed to work— a graphic depiction of relationships between activities and results (conceptual/theoretical model) ■ A logical chain of connections showing what a program intends to accomplish ■ Increases intentionality and purpose ■ Guides prioritization and allocation of resources ■ Helps to identify important variables to measure; use evaluation resources wisely ■ Supports replication

WFHN Evaluation Design ■ Single protocol ■ Work redesign, supervisor training and self-monitoring ■ Multiple industries and worksites ■ Healthcare (blue collar) ■ Telecommunications (white collar) ■ Group randomized field experiment ■ Intervention and comparison group assignment at the worksite or work group level ■ Adaptive randomization to balance covariates

Outcome Evaluation Design ■ Nested cohort design ■ Worksite partners are randomized within industry to intervention or control group ■ Outcomes evaluated at multiple levels, employees, workgroups and worksites ■ Allows for multiple levels of clustering (e.g., workgroups within worksites), and a variety of outcomes (e.g., discrete, continuous, count)

Formative(Process) Evaluation Framework 8 Implementation Outcomes Acceptability Adoption Appropriateness Cost Feasibility Fidelity Penetration Sustainability Implementation Outcomes Acceptability Adoption Appropriateness Cost Feasibility Fidelity Penetration Sustainability - *IOM Standards of Care What? QIs ESTs How? Implementation Strategies Implementation Research Methods Service Outcomes* Efficiency Safety Effectiveness Equity Patient- centeredness Timeliness Individual Outcomes Satisfaction Function Health status/ symptoms Proctor’s Model of Implementation Outcomes

Process Evaluation Data ■ Document Reviews – provide information to build an “a priori” understanding of the program content, operations, context, and program stakeholders ■ Review reports, instruments, protocols, promotional materials, patient materials, resource lists, organizational documents (org charts, flow charts, operation manuals), web sites. ■ Observational Studies – provide empirical evidence to assess program fidelity, and generate service flow and timing data to inform outcome and cost analysis ■ Key Informant Interviews – provide contextual information related to program utility, contextual factors that influence program implementation, service delivery, dispersion, and sustainability ■ Practioner and Consumer Surveys -- provide systematic data related to service delivery experiences and program related perceptions

Qualitative Analysis Methods ■ Recursive abstraction (Document Summaries) ■ Iterative process that generates summaries, classifications, lists, rates, or groupings ■ Deductive and Inductive Analysis (data coding) ■ Deductive “a priori” framework ■ Inductive “grounded theory” analysis ■ Results in taxonomies, themes, categories, orders ■ Comparative analysis ■ Documented vs observed processes, behaviors or outputs ■ Document changes over time (e.g., model migration) ■ Variation between processes or outputs ■ Mixed Methods (triangulation and convergence) 10

Process Evaluation Results  Comprehensive description of program components  Performance indicators and proficiency scores for program delivery  Common barriers and facilitators across worksites  Descriptive taxonomy of program settings  Construction of moderator variables for use in outcome and economic analyses  Program delivery protocols that are used at each worksite site

Outcome Evaluation Questions ■ Program Outputs ■ To what extent is the program actually performed as measured by (e.g. the number of health risk appraisals completed, percentage of employees participating in workshops, number of follow-up services delivered) ■ Proximal and Distal Outcomes ■ What is the effectiveness of the program on outcomes of interest? (changes in sleep quality, changes in bio- measures)? ■ What stakeholder and employee-level characteristics moderate the willingness and ability of performance sites and practitioners to adopt the program

WFHN Study Data

Outcome Evaluation Methods ■ Descriptive statistics generating classifications or groupings of the participating worksites, provider characteristics, employee characteristics ■ Multivariate regression examining the relationship between groups ■ Statistical modeling measuring changes on outcomes over time

Outcome Evaluation Results ■ Rates and frequencies of program participation ■ Performance site descriptive characteristics (including staff characteristics) that are associated with outcomes of interest ■ Within group pre to post changes in health and wellness outcomes ■ Between group comparative changes in health and wellness outcomes at multiple time points

Intervention Effect on Work Environment FSWE Hypotheses ■ H1. There are changes in work environment for the ■ H1a. TOMO intervention group ■ H1b. TOMO control group ■ H1c. LEEF intervention group ■ H1d. LEEF control group ■ H2. There are baseline differences in environment between intervention and control groups for the ■ H2a. TOMO industry ■ H2b. LEEF industry ■ H3. There are differences in the rate of change in climate between intervention and control groups for the ■ H3a. TOMO industry ■ H3b. LEEF industry

FSWE Factor Analysis

Growth Model Results by Industry HypothesisIndustryGroupRandom Effect EstimateStandard Errorp H1aTOMOInterventionSlope < H1bTOMOControlSlope H1cLEEFInterventionSlope H1dLEEFControlSlope H2aTOMO Intervention Vs. Control Intercept H2bLEEF Intervention Vs. Control Intercept H3aTOMO Intervention Vs. Control Slope H3bLEEFIntervention Vs. Control Slope

Economic Evaluation Questions ■ What is the program cost to worksites and to other stakeholders? ■ What is the cost-effectiveness of the program? ■ What characteristics moderate the cost-effectiveness of the program?

Translational Study  An effective intervention is only useful if it is communicated to and adopted by workplaces  Complementary methods:  Use participant feedback to inform post-study messaging  Analyze process data to study employee perception for developing portrayals of the intervention  Assess potential dissemination channels  Stimulate market demand for effective intervention

Thank You For more information on the Work, Family, and Health Network Study