Evidence and policy Jill Rutter

What I am going to cover The theory of government and evidence Practical barriers to use of evidence by government Progress on improving the use of evidence by government Assessment of the use of evidence by government

The theory ………… of policy making

The theory Of evidence use…

What policy making looks like in practice Based on asking senior civil servants…

IFG, together with NIESR, NESTA and with help from the national audit office held a series of events to look at use of evidence in practice.

We found some significant supply barriers Problems Comments Research relevance and academic engagement “ as a policy maker I was always looking for the evidence to support what we should do and being very disappointed that the academic community hadn’t provided the answer for me”, Dr Rachel Glennerster, director of MIT Poverty Lab There are some good academics out there, who know their stuff, can think rigorously and understand the policy process…if you are lucky enough to have some of those academics out there, make the most of them” Chief Analyst.

There were some significant supply barriers Problems Comments Untested and untestable policies Absence of good, useable and unbiased data “ of every ten pilots one was [able to be evaluated]… but they could have all been done better by designing in evaluation from the beginning” Departmental Chief Analyst “there is still an instinctive belief you can do it quickly… [there is] an imperative to solve problems now. Education policy is littered with reforms of good intentions without evidence”. Dr Kevan Collins, Chief Executive of the Education Endowment Foundation “ as soon as the computers discs went missing, our ability (at IFS) to get any data from DWP without being in an armed compound to read it became very difficult” Robert Chote, former Director IFS, head OBR

But we concluded the bigger problems were with demand Comments Timeliness and helpfulness of evidence to ministers “I sat down with a research body the other day and they set out what they were planning to do. I said that “do you realise that by the time your reach your conclusions, it will be far too late to be of use to anybody. It will be great history, but it won’t help anyone make policy….. [by the time you produce your results] I’ll be fisheries minister by then” Minister “whenever you mention the work evaluation to ministers, they feel we are trying to slow them down and stop what they are trying to do”. Senior official

Demand barriers Problems Comments The politics (and ethics) of evidence “ sometimes politicians don’t want evidence because it’s a values thing” Geoff Mulgan, former head of PM Strategy Unit “problems came when evaluations recommended policies you thought would mean you lose your job” Former Minister “ no one likes doing trials of social policy on people for perfectly understandable reasons” Kevan Collins

Demand barriers Problems Comments Skills, behaviour and culture of civil servants “I don’t think I did enough to explain to the permanent secretary how ..good evaluation could help him manage the risks sitting in front of the PAC” Former chief analyst A lot of the policy making community and especially the British civil service is far too self-confident about what they know and what they don’t know” Owen Barder, Center for Global Development The more permanent civil service may be more impermanent Senior Treasury official

Demand barriers Problems Comments Incentives and the role of the Treasury No sense that “the better your evidence, the more money we will give you.” Jonathan Portes, Director of NIESR “if government really understood the outputs it was trying to achieve, then it should be routinely working out the best value way of achieving them…some Whitehall departments don’t do that in as serious a way as they should” Former Minister

The government has taken some steps to increase demand and improve supply Making “open policy making the default” Source: Cabinet Office open policy making blog

Steps to improve demand and supply The Behavioural Insights Team

Steps to improve demand and supply Professionalising policy making – and introducing policy tests

Steps to improve demand and supply The What Works Initiative

So is all this making a difference? The truth is – its very hard to tell….. Report: Evaluation in Government – December 2013

Benchmarking government use of evidence IfG, Alliance for Useful Evidence and Sense about Science framework

Evidence transparency framework

Evidence transparency framework