Policymaking and evidence: how can we improve the fit? Sonia Sodha Head of Policy & Strategy The Social Research Unit at Dartington Using evidence to improve young lives

Who we are Small research-based charity of around 20 staff, based in Dartington, Devon and London Our mission: 1.Understanding what works in improving children and young people’s outcomes. 2.Helping people – central government, local commissioners, trusts and foundations, service providers – apply that evidence in the real world. We seek to fill the gap between evidence as its produced and something that people can use in real-world decisions.

Many different stakeholders influence how and whether evidence gets used EVIDENCE central government local government service providers consumers of public services private investors trusts & foundations practitioners

Who are the ‘policymakers’? Central government – decisions about systems Commissioners – deciding how money gets spent on what services Service providers – deciding how to structure services Practitioners – interacting with service users

What is evidence? Evidence of impact- whether we know something works Evidence about needs – what’s the issue services are trying to address? Evidence about what affects educational outcomes Evidence about how people feel about a service

What are the barriers to greater use of evidence in decision-making? 1.Political and institutional incentives

What are the barriers to greater use of evidence in decision-making? 1.Political and institutional incentives 2. The ‘what works’ test is not a simple one

1. How do we tell if something has impact? Our ‘what works’ standards of evidence have four dimensions: 1.Intervention ‘specificity’ 2.Public service ‘system readiness’ 3.Evaluation quality 4.Impact But these are standards that are designed to apply to programmes rather than other features of public service design, like system reform policy, practice and processes What’s being evaluated – design and implementation Evaluation of impact

2. It’s not easy to prove impact – so the evidence base is limited Not an either/or between innovation and evidence. INNOVATIONIMPACT theory of change pre/post outcomes monitoring experimental evaluation

Total 9-18 population of Renfrewshire: n = 15,409 Problem counts across Renfrewshire 9-18 yrs 6% of children with many problems (n = 925) 53% of children with no problems (n = 8,166) 41% of children with a few problems (n = 6,318) 3. Evidence of impact is not enough

Reach of Systems: Combined service contact 9-18 yrs 53% of children with no problems (n = 8,166) Total 9-18 population of Renfrewshire: n = 15,409 6% of children with many problems (n = 925) 41% of children with a few problems (n = 6,318) 7% of children involved in at least one system (n = 1,079) 38% of system-involved have no problems (5% of total ‘no problem’ pop.) 50% of system-involved have have a few (9% of total ‘few problems’ pop.) 12% of system-involved have have many (14% of total ‘many problems’ pop.) 38% 50% 12% 86% of those with many problems have no service contact

4. How do we replicate and scale? Three challenges to replicating and scaling what works: 1.Does what works in one context work in another? 2.How do you replicate what makes a programme or practice effective again and again? Evidence-based programmes are one vehicle but they will have limited reach. How would you scale an evidence-based practice?

Evidence rarely gives the black and white answers policy-makers expect

What are the barriers to greater use of evidence in decision-making? 1.Political and institutional barriers 2. ‘What works’ is not a simple test 3. The link between policy and practice is distal

What can we do to break down some of the barriers? 1.Depoliticisation? Are there ways of depoliticising some decisions and lengthening time horizons? Can politicians and policymakers build in safeguards? Eg the Educational Endowment Fund

What can we do to break down some of the barriers? 1.Depoliticisation? 2. System reform

Can systems be reformed to better promote use of evidence in policy? Important levers: Accountability Funding: subsidy, matched funding for local budgets Funding of knowledge generation and dissemination – better tying the two together

What can we do to break down some of the barriers? 1.Depoliticisation? 2. System reform 3. Embedding understanding of evidence into professional development

What can we do to break down some of the barriers? 1.Depoliticisation? 2. System reform 3. Embedding understanding of evidence into professional development 4. Commission better evaluation that contribute to learning

Policymaking and evidence: how can we improve the fit? Sonia Sodha Head of Policy & Strategy The Social Research Unit at Dartington Using evidence to improve young lives