Evaluating new OH initiatives - lessons from WHC Claire Tyers Institute for Employment Studies

Main evaluation elements PROCESS IMPACT COST-BENEFIT Management information Provider visits 2 wave employer survey with control group 2 wave employer case studies Cost Benefit Analysis Costs to business survey OH purchasing survey

User profile and reactions Feb 06 to April 07: ■ over 5,000 adviceline calls ■ 2,590 visited by advisers Reaching small companies ‘Improvers’ versus ‘starters’ Free, but credible, high satisfaction OH issues covered, but little user recognition

Early outcomes Recommendations useful Majority changed or plan to Greater knowledge and confidence Initial change to procedures Full impact assessment will take time Awareness/ knowledge transfer Policies/procedures Real behavioural change/health outcomes

Evaluation challenges Complex service Effect size of intervention Nature of effects Latency of effects Data deficiencies in relation to effect indicators Disentangling effects

The evaluation quandary ‘Ideal’ intervention‘Real’ interventions Substantial intervention Limited effect size Identical service for all Variations in delivery Clearly definable and measurable outcomes Indistinct or changing outcomes Large numbers of users Constrained by resource and take up Runs for long period Time limited

Evaluation will provide… Learning about operational model Improved understanding: chain of impact Objective assessment of change Perceptions of change Full understanding of intermediate outputs Assessment of final outcomes – with caveats

… thank you, any questions?