Innovation or evidence in children’s services? Provocation presentation to the 2012 Leadership Foundation Conference ‘Unlocking Learning’ Sonia Sodha Head of Policy & Strategy Dartington Social Research Unit

Who we are Small research-based charity of around 20 staff, based in Dartington, Devon and London Our mission: 1.Understanding what works in improving children and young people’s outcomes. 2.Helping people – central government, local commissioners, trusts and foundations, service providers – apply that evidence in the real world. We seek to fill the gap between evidence as its produced and something that people can use in real-world decisions. Much of the policy debate on children is focused on the national level. We are also interested in how local decisions get made.

Evidence matters Better application of evidence about what works and analysis of what is most cost effective has the power to transform children and young people’s outcomes in a time of austerity. EVIDENCECOST SAVINGSBETTER OUTCOMES Family Functional Therapy Families and Schools Together (FAST) Reading Recovery Scared Straight Reduces offending with effect size 0.59 Improves attainment with effect size 0.10 Improves attainment with effect size 0.48 Increases offending with effect size 0.11 £1 invested saves £2.31 for taxpayers £1 invested saves £1.36 for taxpayers £1 invested saves £3.72 for taxpayers £1 invested costs taxpayers £11.54

Decision-makers need 3 types of data to commission evidence-based services 1.Needs data

Total 9-18 population of Renfrewshire: n = 15,409 Problem counts across Renfrewshire 9-18 yrs 6% of children with many problems (n = 925) 53% of children with no problems (n = 8,166) 41% of children with a few problems (n = 6,318)

Reach of Systems: Combined service contact 9-18 yrs 53% of children with no problems (n = 8,166) Total 9-18 population of Renfrewshire: n = 15,409 6% of children with many problems (n = 925) 41% of children with a few problems (n = 6,318) 7% of children involved in at least one system (n = 1,079) 38% of system-involved have no problems (5% of total ‘no problem’ pop.) 50% of system-involved have have a few (9% of total ‘few problems’ pop.) 12% of system-involved have havemany (14% of total ‘many problems’ pop.) 38% 50% 12% 86% of those with many problems have no service contact

Decision-makers need 3 types of data to commission evidence-based services 1.Needs data 2. Evidence of what works

‘What works’ standards 1.Evaluation 2.Impact 3.Specificity 4.Public-service ‘readiness’

The ‘what works’ evidence pipeline INNOVATIONIMPACT Not an either/or between innovation and evidence. Every stage of the pipeline matters in terms of generating evidence of what works. theory of change pre/post outcomes monitoring experimental evaluation

Decision-makers need 3 types of data to commission evidence-based services 1.Needs data 2. Evidence of what works 3.Value for money: cost-effectiveness or cost- benefit analysis

We are applying a cost-benefit model developed by Washington State Institute for Public Policy that determines costs and benefits (to the individual, taxpayer and citizen) of competing options for investment in children’s services. It is cautious and robust in estimation of impact, accurate in estimation of cost data and consistent between different policies and programmes. Service areas:youth offending, early years and education, child protection, child and adolescent mental health, drugs and alcohol services.

SUMMARY OF COSTS AND BENEFITS Functional Family Therapy Cost£2,501 Benefits to Taxpayers£5,775 Benefits to Participants£2,177 Benefits to Others£17,650 Total Benefits£25,603 Benefits - Costs (NPV)£23,102 Benefit-Cost Ratio£10.24 Rate of Return on Investment109% Risks costs < benefit1%

Source of benefitsTo participants Totaxpaye rs ToothersTotal benefits Crime £0£4,708£17,650£22,358 Earnings via High School Graduation(A-Levels) £2,177£1,068£0£3,245 TOTAL £2,177£5,776£17,650£25,603

Significant barriers exist to the application of evidence Supply sideDemand side NOT ENOUGH HIGH-QUALITY EVIDENCE OF WHAT WORKS evaluation market functions poorly INSTITUTIONAL BARRIERS often not in commissioners’ interests to commission based on evidence and CBA NOT ENOUGH COST-BENEFIT ANALYSIS CBA market functions even more poorly! POOR QUALITY DATA INFORMING COMMISSIONING service data doesn’t allow intelligent commissioning LACK OF UNDERSTANDING little understanding amongst commissioners about what evidence of impact looks like HEARTS AND MINDS commissioners tend to commission on basis of intuition rather than hard evidence

Many different stakeholders influence the use of evidence EVIDENCE central government local government service providers consumers private investors philanthropists

Replication and scale Three challenges to replicating and scaling what works: 1.Does what works in one context work in another? 2.Supply side: how do you replicate an evidence-based programme with fidelity to what makes it effective? 3.Demand side: no point trying to scale something that commissioners and users don’t want to use! Answer: you need to figure out what’s non-negotiable (egsmartphone hardware) and what’s personalisable (egsmartphone apps)

Helping people apply and generate evidence INNOVATION TO PROVEN IMPACT supporting service providers to move from innovation to proof of impact COST EFFECTIVE RCTS making experimental evaluation affordable REDESIGNING SYSTEMS to help commissioners reinvest savings eg children in care; youth offending. SUPPORTING EVIDENCE- BASED INVESMENT bespoke strategy development with LAs& trusts/foundations BLUEPRINTS FOR SUCCESS An online, interactive portal for commissioners that brings together information about what works with rigorous cost-benefit analysis CHILDRENCOUNT an online data portal and survey builder generating high-quality needs data HOW DO YOU REPLICATE & SCALE? work on how to make what works replicable and scaleable SUPPLYDISSEMINATIONDEMAND

Innovation or evidence in children’s services? Provocation presentation to the 2012 Leadership Foundation Conference ‘Unlocking Learning’ Sonia Sodha Head of Policy & Strategy Dartington Social Research Unit