Evaluation – Principles and methods Matt Barnard Head of Evaluation NSPCC

Purpose of evaluation Definition “Examine how a policy or intervention was designed and carried out and with what results.” (Magenta Book) Asks objective questions –What were the impacts? –How was it delivered? –What were barriers and facilitators? –Did it deliver value for money? Aims to provide –‘Scientific’ basis for policy making

Evaluation Design Process Evaluation findings Intervention Design Logic model Intervention implementation planning Evaluation implementation Intervention implementation Evaluation Design

Logic models Characteristics –Mechanisms not processes –Key steps not every step –Explanatory not descriptive –Reflects theoretical assumptions Benefits –Sense check –Identifies realistic outcomes –Facilitates evaluation design

Strength of design Weak design (Poor/ no counterfactual) Strong design (Realistic counterfactual) Low power (Small numbers/ effect size) Unlikely to detect difference/ Low confidence in attribution Unlikely to detect difference/ High confidence in attribution High power (Large numbers/ effect size) Likely to detect difference/ Low confidence in attribution Likely to detect difference/ High confidence in attribution Strength of design matrix

Strength – Evaluation design Randomized controlled trial Quasi-experimental design Before and after measures

Types of design RCT –Individual randomization –Cluster randomization/ roll out –BAU/waiting list/alternative services Quasi-experimental designs –Matched area/ groups –Matched individual –Interrupted time series –Regression discontinuity

Factors influencing methodology Intervention stage of development –Early exploration –Defined and established but not proven –Transferability Potential Costs and benefits –Resources –Timescales

Key principles Clarity about key question –Avoid ‘default’ questions Methods matched to question –Ensure methods match desired questions Claims match evidence –Avoid over-claiming Have a coherent story to tell