Contribution Analysis: An introduction Anita Morrison Scottish Government

Evaluation in Scottish Government A need to revisit our practice in context of performance management and outcomes-based approach – shift balance from small scale, process evaluation Building a more ‘strategic’ approach to evaluation planning and delivery - multiple policies / interventions to deliver outcomes Ensure the right kind of evidence is available – answering questions of effectiveness, contribution, for Spending Review 09 and beyond

Key analytical questions What progress are we making towards the Purpose, Purpose targets and National Outcomes? What contribution have our policies and programmes made? How cost-effective are our policies and programmes? What are the lessons for future policy intervention /changes in resource allocation? Contribution Analysis

Contribution analysis Holy grail of attribution…“Builds a reasonably credible case about the difference the program is making.” (John Mayne) Good contribution analysis is, in effect, theory-based evaluation Use of logic models and results chains strengthens programme planning and design Clearly sets out assumptions and external factors Also strengthens approach to research / analytical planning Developing in a couple of ways in Scotland: contribution of organisations; contribution of policies and programmes

Alcohol – Partner Contributions to Outcomes Inputs Activities Outputs Reach Short-term outcomes Intermediate outcomes High level outcomes NHS Brief advice Preventive services Hazardous and harmful drinkers Service uptake & engagement Police Enforcement of drink driving laws Random breath testing Drivers Increased detection rate Local authorities Enforcement of planning controls & licensing laws Enforcement actions Licensed trade Compliance with laws Behaviour Reduced alcohol consumption levels Less drunkenness; less drink-driving Improved mental wellbeing Reduced inequalities in healthy life expectancy Reduced inequalities in alcohol-related deaths and hospital admissions Environments Physical: Reduced exposure to alcohol-related hazards Economic: Reduced availability/affordability of alcohol Social: Drunkenness less attractive; sensible drinking the norm SG, UK govts, EU Industry regulation Taxation,displays, promotions, advertising Alcohol industry Increased price Reduced incentives Scottish Govt Media campaigns Sensible drinking messages General public - targeted Understanding risks, attitudes to drinking Vol orgs Detox, Intensive support Addiction services Adults with alcohol problems Increased sobriety & stability

A Basic Results Chain 9 7. End results 7. What is our impact on ‘ends’? 6. Practice and behavior change 6. Do we influence [behavioural] change? 5. Knowledge, attitude, skill and / or aspirations changes 5. What do people learn? Do we address their needs? 4. Reactions 4.Are clients satisfied? How do people learn about us? 3. Engagement / involvement 3. Who do we reach? Who uses / participates? 2. Activities and outputs 2. What do we offer? How do we deliver? 1. Inputs 1. How much does our program cost? ($, HR etc) Program (Results) Chain of Events (Theory of Action) Key Questions Source:Adapted from Claude Bennett Taken from Michael Quinn Patton, Utilization-Focused Evaluation: The New Century Text, Thousand Oaks, California, 1997, p 235. Indirect Influence Direct Influence Control WHY? WHAT? WHO? HOW?

Assembling and critiquing the story Multiple sources of evidence to accommodate; can include: Population based statistics Survey data Needs assessments Process and outcome evaluations Economic analysis and evaluation Systematic reviews and other evidence synthesis International policy and analytical work Baseline and forward look – data /evidence gaps

How do you actually do this in practice? To date – few ‘live’ examples of contribution analysis applied in the UK – Health Scotland led on this Scottish Government early work: outcome 15, GES population target, interest on outcome 8, outcome 12 Sessions to develop and discuss logic model (mix of analysts and policy leads) Baseline ‘narrative’ and performance story so far Analytical planning – what data and analysis will we need in the future?

Some tentative conclusions Outcomes focus is an opportunity to undertake longer-term, more strategic analytical work Needs input of policy, practice and analytical professionals Contribution analysis not a panacea – but a tool to help think and discuss policy and service development differently Does help you to plan any research and analysis required in assessing progress

Over to you…. Are you using this, or components of this, already? Can this be applied to outcomes / indicators in Communities? Or to larger programmes you are evaluating?