March 2004 University of Bristol Leverhulme Centre for Market and Public Organisation Comments on Trying it Out: The Role of Pilots in Policy Making Carol Propper Research Methods Festival July 2004

March 2004 Report very welcome –Facilitate greater understanding of the benefits of trials and the constraints that policy makers and researchers work within –Recommendations sensible and timely But … –Trials and pilots are not an automatic fix –RCTs may not be the goldstandard (or more properly, will not necessarily provide easy to find answers)

March 2004 Problem in interpreting trials Governments have large set of policy innovations running at same time –For area initiative difficult to find areas in which to undertake pilots - report section if present trends continue, the supply of suitable untouched localities may soon be exhausted –Same issue arises in pilots implemented in organisations aimed at individuals (e.g. Makinson scheme in C+E) –Participation in trials may be because extra monies are forthcoming

March 2004 Problem in implemeting RCTs RCTs difficult to administer in social settings –MTO scheme: treatment = living in (moving to) a better neighbourhood –Some of the treated did not participate; some controls moved because of other policies –Some of the treated moved back very quickly to neighbourhoods more similar to their own (treatment not for long) –Some maintained close ties with former neighbourhood (what exactly is the treatment)

March 2004 Problem in interpreting RCTs RCT approach may not be appropriate when there are externalities –RCT designed to provide partial equilibrium estimates of the treatment for a well designed population –If practice by one person/organisation affects outcomes for others effectiveness of the treatment depends on where the trial is conducted –If doing high-tech surgery creates externalities which has positive effects on other patients at same hospital/in same area, surgical interventions will perform well in areas which already do lots, but perform poorly in areas where other treatments are conducted –RCTs do not pick up general equilibrium effects