1. Assessing the Contribution of Policies to Outcomes – Logic Modelling and Contribution Analysis Jackie Horne, Alison Stout Strategic Research Strategy and Performance Division The Scottish Government Heather Doherty Research Manager Museums Galleries Scotland

2. Learning Outcomes Distinguish between inputs, outputs and outcomes Understand the theory and practice of logic modelling and contribution analysis Practice developing a results chain and a logic model to help assess how and whether policies are delivering on outcomes Practice developing indicators to measure progress against outcomes

4. Outcomes approach Integral to: Programme for Government Spending Review 2011 Government Economic Strategy Renewing Scotland's Public Services - Priorities for reform in response to the Christie Commission

5. The outcomes approach It is widespread…… Countries that have implemented an outcomes based approach include Canada, New Zealand, Unites Stated (in various forms across different states), England, Singapore and Malaysia, almost all other OECD member countries, many developing countries and a number of international agencies and NGOs. But the language differs……… Outcome governance, outcomes management, accountability for outcomes, results based management, but, overall, the reform agendas are similar.

6. The outcomes approach There are commonalities: they generally involve greater devolution of power and decision-making and commonly adopt some form of performance management and reporting system (e.g. Virginia Performs, New Zealand governments’ Statements of Intent, Canadian Government’s whole of government framework). It takes time is a common message…. Evidence suggests at least 4-5 years of consistent effort is required to embed an outcomes approach —and many organisations have been at it much longer.

7. Exercise Distinguishing between inputs, activities, outputs and outcomes

8. Understanding the relationship between policies and outcomes – single intervention

9. A simple results chain Over 65s, especially those not doing enough exercise Staff, equipment, leisure centres, etc. Invest in exercise facilities and classes for over 65s More accessible exercise classes and facilities for over 65s. Increased levels of physical activity in over 65s Lower blood pressure, improved mobility among over 65s Increase in healthy life expectancy

10. National outcomeWe live our lives safe from crime, disorder and danger (NO 9) Our young people are successful learners, confident individuals, effective contributors and responsible citizens (NO 4) Intermediate outcomes Less antisocial behaviour. Less time spent bored and on streets. Constructive use of leisure time. Aspirations and confidence increase. Young people recognise that they have more opportunities. Young people learn a new skill. Reach16 – 21 year old in areas hardest hit by crime. OutputCoaching, games, leagues, role models, education during time out. ActivitiesTwilight basket ball (funded through Cashback)

11. Exercise Results chain

12. Multiple policies - logic models and contribution analysis Move beyond results chains to consider inter- linkages between policies and outcomes Consider strengths (and weaknesses) of the contribution of different policies to outcomes Assess and measure progress towards purpose targets and national outcomes Inform cost-effectiveness and resource allocation – decisions on priorities Highlight strengths and weaknesses in evidence base

13. Steps of contribution analysis Identify outcome Logic modelling Evidence gathering Performance story

14. Uses of logic modelling and contribution analysis Evidence effectiveness of a defined policy area (Population Purpose Target) Work out public sector partner organisations’ contribution to policy area and high level outcomes (Alcohol – Health Scotland) Identify cross office and agency contributions to National Outcomes (National Outcome 12) Develop indicators (CYP) Business planning and indicators (Justice)

16. Alcohol – Partner Contributions to Outcomes Inputs Activities Outputs Reach Short-term outcomes Intermediate outcomes High level outcomes NHS Brief advice Preventive services Hazardous and harmful drinkers Service uptake & engagement Police Enforcement of drink driving laws Random breath testing Drivers Increased detection rate Local authorities Enforcement of planning controls & licensing laws Enforcement actions Licensed trade Compliance with laws Behaviour Reduced alcohol consumption levels Less drunkenness; less drink-driving Improved mental wellbeing Reduced inequalities in healthy life expectancy Reduced inequalities in alcohol-related deaths and hospital admissions Environments Physical: Reduced exposure to alcohol-related hazards Economic: Reduced availability/affordability of alcohol Social: Drunkenness less attractive; sensible drinking the norm SG, UK govts, EU Industry regulation Taxation,displays, promotions, advertising Alcohol industry Increased price Reduced incentives Scottish Govt Media campaigns Sensible drinking messages General public - targeted Understanding risks, attitudes to drinking Vol orgs Detox, Intensive support Addiction services Adults with alcohol problems Increased sobriety & stability

17. NO 12 - We value and enjoy our built and natural environment and protect and enhance it for future generations

19. National Outcomes- Justice 9 – We live our lives safe from crime, disorder and danger 11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others 15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs

20. Unpacking the National Outcomes – Justice National OutcomeIntermediate Outcomes 9 – We live our lives safe from crime, disorder and danger A – We experience low levels of crime B – We experience low levels of fear, alarm and distress C – We are at a low risk of unintentional harm 11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others D – Our social and cultural values promote pro-social behaviours E – We have high levels of public confidence in justice institutions and processes F – Our public services are fair and accessible 15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs G – Our institutions and processes are effective and efficient H – Our public services respect the rights and voice of users

21. National OutcomeIntermediate Outcomes Key low level outcomes 9 – We live our lives safe from crime, disorder and danger A – We experience low levels of crime tackling the underlying causes of crime, deterring offending, active prevention of offending, promoting desistance for those who have already offended B – We experience low levels of fear, alarm and distress people valuing family and communities, good social and dispute resolution skills, communities that are integrated, good enforcement of the law, good public understanding of risk C – We are at a low risk of unintentional harm active management of physical risks, effective emergency planning, effective management of risk by justice organisations 11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others D – Our social and cultural values promote pro-social behaviours public understanding of rights and responsibilities, people valuing family and communities, active citizenship and strong community leadership, low levels of social inequality, public intolerance and social stigma of criminal behaviour, strong and consistent political messages E – We have high levels of public confidence in justice institutions and processes equality of access to justice, efficient processes and joined up information systems, a swift and visible criminal justice system, positive experiences of the justice system F – Our public services are fair and accessible strong community engagement and involvement, equality of access to justice, protection of the rights of all member of the public 15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs G – Our institutions and processes are effective and efficient good leadership and management, continuously improving services, efficient processes and joined up information systems, good governance and understanding of organisational roles and responsibilities, efficient and effective resource allocation H – Our public services respect the rights and voice of users effective user feedback mechanisms, protection of the rights of all member of the public, a “consumer-led” service ethos

22. KEY TO RECOMMENDATION: RED – No further action Amber – Evidence unclear or lacking Green – Pursue further

23. Piloting contribution analysis in Museums Galleries Scotland

24. What did we pilot it on? Hands On - An existing project that had come to an end in its current form

25. Where have we used it since? Festival of Museums – A new programme that was about to start Choices for Change toolkit – a decision- making toolkit that was developed collaboratively Collaborating to Compete conference

26. What did we learn? 1.It was helpful to name our first attempt as a pilot

27. 2. It’s additional to any existing evaluation activity – rather than a replacement

28. 3. It was helpful to apply the tool retrospectively to familiarise ourselves with the logic model part of the proces

29. 4. BUT, more value in applying from the beginning of a project

30. 5. It gives an overview of a piece of work and its component parts in a way that we don’t get with any other method

31. 6. It can be applied to a small programme as well as a large one

32. 7. The logic model can be customised to suit our needs –To incorporate our evaluation framework –To incorporate other frameworks, for example Curriculum for Excellence –From project to project

33. 8. Using the logic model part of the process is valuable in itself, even if you don’t go on to further analyse each contribution

34. 9. It’s helpful to populate the logic model as a group (even a group of 2 people)

35. 10. Once you understand the logic model headings and what they mean in a contribution analysis context, populating the logic model becomes easier and quicker

36. 11. The logic model headings are very useful for revisiting the reason for your project or programme

37. 12. The type of work that fits well with this approach quickly becomes apparent

38. 13. Introducing contribution analysis to people works best with conversation and coaching, being given the documents alone to read is not enough

39. Exercise Logic modelling

40. Indicators ‘’Imagine a car dashboard: an indicator is a warning light flashing on the dashboard. It is fed by one of many streams of data – maybe oil level, temperature, etc...It flashes when all is not well, suggesting we stop the car. The indicator “alerts us to something worthy of further investigation.” The Good Indicator Guide (NHS Institute for Innovation and Improvement)

41. Early Years Outcomes and Indicators Framework Core set of indicators to monitor progress at a national and local level www.scotland.gov.uk/Topics/People/Young-People/Early-Years-and- Family/Early-Years-Framework/Implementation/Measuring-Practice www.scotland.gov.uk/Topics/People/Young-People/Early-Years-and- Family/Early-Years-Framework/Implementation/Measuring-Practice Developed using logic modelling Criteria –easily available at sub national level (local authority or NHS Board) –collected regularly –strong evidence base –collected by a credible source –tell us more than the face value of the individual indicator –taken together, cover the whole pre-birth to 8 age range.

42. Early Years – Core Indicators 1.Teenage pregnancy rate (pregnancies among under 16 year olds (3 year average per 1000 relevant population)) 2.Appropriate birth weight for gestational age (low weight live births) 3.Breastfeeding at 6-8 weeks (% of newborn children exclusively breastfed at 6-8 weeks) 4.% parents who rate their neighbourhood as a good place to live (% of adults stating their neighbourhood is a 'very good' place to live) 5.Children in benefit dependent households (% of children living in households that are dependent on out of work benefits or Child Tax Credit) 6.% of obese children in P1 (estimated % of obese children in P1) 7.% of children with dental decay in P1 (% of children with dental decay in P1) 8.Children (pre birth to 8) referred to reporter on care and protection grounds 9.Number of children pre-birth to 8 looked after a) at home b) accommodated 10.% of young people in sustainable post school destinations (% of school leavers in positive and sustained destinations)

43. Exercise - Developing indicators 1.How will the indicator be defined and measured? 2.Where will the data come from? 3.Will it measure absolute numbers or proportions? 4.How frequently will the data be updated? 5.Will you set an associated target, to drive forward change? 6.What is a reasonable length of time to achieve the desired outcome? 7.Is it clear what type of change in the data represents an improvement in performance? 8.Are there any caveats/warnings/problems? e.g. missing data, perverse incentives

44. The outcomes approach: general findings Key organisational challenges: ensuring visible and clear political and senior level leadership unrealistic expectations about the timescale for change ensuring co-ordination across policy areas and tiers of government linking performance information and decision-making locally and nationally most jurisdictions have struggled to link outcomes with budgets accepting that outcomes are only one factor in decision making

45. The outcomes approach: general findings Key technical challenges defining outcomes in a measureable way can be difficult avoiding performance indicator overload attribution of outcomes to government action per se, or to a given policy, programme or project is not straightforward outcomes generally have a long-term focus, which can result in reporting time-lags outcome measurement, data quality and reporting problems

46. Contact details Jackie Horne (jackie.horne@scotland.gsi.gov.uk)jackie.horne@scotland.gsi.gov.uk Alison Stout (alison.stout@scotland.gsi.gov.uk) HeatherDoherty (heatherd@museumsgalleriesscotland.org.uk)