Guidance for Analyses of Pilot Interventions European Workplace and Alcohol Berlin, 9 November 2012 Jon Dawson

Broad approach Quantitative analysis of all pilots (baselines and follow-up) More in-depth and independent analyses of selection of pilot interventions (semi-structured interviews with key informants)

Aims To see whether the pilots worked as planned and whether they were delivered in the most effective way To assess the extent that the pilots achieve specific outputs (eg no. of businesses and employees engaged To assess the impacts of the pilot interventions and the changes that they have brought about To appraise effectiveness, highlight good practice and identify “success factors” or, indeed “failure factors”

Scope of evaluation Following the completion of the baseline and post- intervention follow-up (employee survey and organisational audit), a series of analyses will be carried out by the SARD team, primarily to assess changes, if any, in awareness, attitudes and behaviour (employees and organisations) At least 6 pilot partners will also be selected by the SARD team to be the focus of a detailed qualitative analysis Selection criteria: type of intervention; geographic spread; employee survey evidence; organisational audit evidence.

Qualitative Analysis: Aims Overall aim: explore and identify lessons about the process of implementing pilots and their effectiveness Specific aims: 1)qualitative interpretation of quantitative data 2)in-depth understanding of how the project was delivered 3)identify success factors and/or barriers to success.

Qualitative Analysis: Methodology Semi-structured interviews: with project leads, project workers, other stakeholders with detailed knowledge of intervention Focus of interviews:  How project was delivered  Examining the process: what worked well, “success factors” or “barriers” to success  Assess achievements (objectives reached, impact, generating change)  Good practice and policy lessons  Transferability

OUTPUT Overall pilot analysis reports that: (1) Set out findings from analysis of pilot (2) Informs toolkit (3) Informs policy recommendations