1. Evidence and policy
Jill Rutter

2. What I am going to cover The theory of government and evidence
Practical barriers to use of evidence by government
Progress on improving the use of evidence by government
Assessment of the use of evidence by government

3. The theory …………
of policy making

4. The theory
Of evidence use…

5. What policy making looks like in practice
Based on asking senior civil servants…

6. IFG, together with NIESR, NESTA and with help from the national audit office held a series of events to look at use of evidence in practice.

7. We found some significant supply barriers
Problems
Comments
Research relevance and academic engagement
“ as a policy maker I was always looking for the evidence to support what we should do and being very disappointed that the academic community hadn’t provided the answer for me”, Dr Rachel Glennerster, director of MIT Poverty Lab There are some good academics out there, who know their stuff, can think rigorously and understand the policy process…if you are lucky enough to have some of those academics out there, make the most of them” Chief Analyst.

8. There were some significant supply barriers
Problems
Comments
Untested and untestable policies
Absence of good, useable and unbiased data
“ of every ten pilots one was [able to be evaluated]… but they could have all been done better by designing in evaluation from the beginning” Departmental Chief Analyst “there is still an instinctive belief you can do it quickly… [there is] an imperative to solve problems now. Education policy is littered with reforms of good intentions without evidence”. Dr Kevan Collins, Chief Executive of the Education Endowment Foundation “ as soon as the computers discs went missing, our ability (at IFS) to get any data from DWP without being in an armed compound to read it became very difficult” Robert Chote, former Director IFS, head OBR

9. But we concluded the bigger problems were with demand
Comments
Timeliness and helpfulness of evidence to ministers
“I sat down with a research body the other day and they set out what they were planning to do. I said that “do you realise that by the time your reach your conclusions, it will be far too late to be of use to anybody. It will be great history, but it won’t help anyone make policy….. [by the time you produce your results] I’ll be fisheries minister by then” Minister “whenever you mention the work evaluation to ministers, they feel we are trying to slow them down and stop what they are trying to do”. Senior official

10. Demand barriers Problems Comments
The politics (and ethics) of evidence
“ sometimes politicians don’t want evidence because it’s a values thing” Geoff Mulgan, former head of PM Strategy Unit “problems came when evaluations recommended policies you thought would mean you lose your job” Former Minister “ no one likes doing trials of social policy on people for perfectly understandable reasons” Kevan Collins

11. Demand barriers Problems Comments
Skills, behaviour and culture of civil servants
“I don’t think I did enough to explain to the permanent secretary how ..good evaluation could help him manage the risks sitting in front of the PAC” Former chief analyst A lot of the policy making community and especially the British civil service is far too self-confident about what they know and what they don’t know” Owen Barder, Center for Global Development The more permanent civil service may be more impermanent Senior Treasury official

12. Demand barriers Problems Comments
Incentives and the role of the Treasury
No sense that “the better your evidence, the more money we will give you.” Jonathan Portes, Director of NIESR “if government really understood the outputs it was trying to achieve, then it should be routinely working out the best value way of achieving them…some Whitehall departments don’t do that in as serious a way as they should” Former Minister

13. The government has taken some steps to increase demand and improve supply
Making “open policy making the default”
Source: Cabinet Office open policy making blog

14. Steps to improve demand and supply
The Behavioural Insights Team

15. Steps to improve demand and supply
Professionalising policy making – and introducing policy tests

16. Steps to improve demand and supply
The What Works Initiative

17. So is all this making a difference?
The truth is – its very hard to tell…..
Report: Evaluation in Government – December 2013

18. Benchmarking government use of evidence
IfG, Alliance for Useful Evidence and Sense about Science framework

19. Evidence transparency framework

20. Evidence transparency framework