Government Social Research Unit Evidenced Based Policy: The Role of Randomised Controlled Trials Ricky Taylor Government Social Research Unit HM Treasury London SW1A 2HQ

There is nothing a government hates more than to be well-informed; it makes the process of arriving at decisions much more complicated and difficult John Maynard Keynes

Social science should be at the heart of policy making. We need a revolution in the relationship between government and the social research community - we need social scientists to help determine what works and why, and what type of policy initiatives are likely to be most effective … (David Blunkett, 2000)

Government policy must be: Evidence-based Properly evaluated Based on best practice The Context of More Experimental Designs in Government

Policy making must be soundly based on evidence of what works Improve departments capacity to make best use of evidence Improve the accessibility of the evidence available to policy makers The Context of More Experimental Designs in Government

Better Policy Making November 2001 UK policy making is now more informed by evidence: The Context of More Experimental Designs in Government

…a society that would use social science methods and evaluation techniques to vigorously try out possible solutions to recurrent problems and would make hard- headed, multidimensional evaluations of outcomes, and when the evaluation of one reform showed it to have been ineffective or harmful, would move on and try other alternatives The Experimenting Society (Donald T. Campbell)

A social experiment benefits society by providing better information on which to base public policy. Such information can improve policy in one of two ways: it can lead policymakers to adopt a program or policy that is found to have net social benefits, or it can lead to the termination of an existing program that is found to have a net social cost (Orr, L., 1999). Why Experiment?

Why Experiment? …Studies using non-random assignment may produce acceptable approximations to results from randomised experiments under some circumstances, but that reliance on results from randomised experiments as the gold standard is well founded ( Shadish and Ragsdale, 1996 ).

Why Experiment? Randomised Controlled Trials: Provide most precise estimates of the likely effects of policy interventions. Against a strong counterfactual – ie, net effect/impact Reduces/accounts for various biases. Establishes cause and effect of policy interventions and outcomes. Raises quality of evidence to support policy making.

Number of published RCTs Source: Campbell Collaboration

For 30 years social experiments have been the primary method of evaluating new policies in the US. Lyndon Johnsons War on Poverty saw the evolution of randomised trials. Late 1960s the Head Start programme, and the Perry pre- school programme were evaluated through randomised trials. 1968, the Mathematica Foundation conducted the New Jersey-Pennsylvania Income Maintenance Experiment The U.S Experience

It is probably correct to report that - in contrast to either Britain or the EU - RCTs are effectively the default option in the USA...It is mainly when such trials turn out to be impractical for one reason or another that other methods, such as matched area-based trials or before-and-after studies, come into their own. (Trying It Out: The Role of Pilots in Policy-Making, Cabinet Office 2003) The U.S Experience

Too easy to blame policy makers A longer tradition of experimentation US academics are more interested in policy development Much greater capacity to do trials Fewer social researchers in UK Government, or in academia, have the skills to conduct large scale trials. Capacity problems in UK social science But things are changing… Why Are RCTs More Common in the U.S?

Employment Demonstration Project (ERA) Job Retraining and Rehabilitation Project (JRRP) Restart evaluation BA Visiting Office Pilot Evaluation of Employment Zones New Deal 25+ Pilots Evaluation of the New Deal for Young People Intensive Gateway Evaluation of the Restorative Justice programme Examples of Recent Government RCTs

It works but how does it work?! Black Box issues Budgets (not necessarily more expensive) Practicalities Ethical Issues Some Challenges for Experimental Designs

Research of any kind is seldom the determining factor in shaping public policy. Experimentation is no exception to this rule (Orr, L., 1999) Ideology and Beliefs Manifesto Commitments Special Advisers Lobbyists and Pressure Groups Electoral/Parliamentary timetables Policy cycle Timeliness Other Factors Influencing Policy

Recognition/acceptance of the value of experimental designs in policy making /evaluation Capacity building is required in UK social science Need for training and professional development Need to promote experimentation in social research Next Steps

Ricky Taylor Government Social Research Unit HM Treasury London SW1A 2HQ Tel: +44 (0) Contact