1. ‘Trying it out’ Review of effectiveness of government pilots
Roger Jowell
Centre for Comparative Social Surveys
City University, London
with Annette King, Government Chief Social Researcher’s Office, Cabinet Office
1 July 2004

2. Panel of Enquiry
Roger Jowell (Chair), City U Waqar Ahmad, DETR Sue Duncan, PM’s Strategy Unit John Fox, DoH Edward Page, LSE Michael Richardson, DWP Judy Sebba, DfES Ann Taggart, HM Treasury Robert Walker, Nottingham U Paul Wiles, Home Office
CABINET OFFICE TEAM:
Phil Davies Annette King Rebecca Stanley Tess Ridge Lucy Woodward

3. Components of the Review
Expert Workshop
Literature Review
Postal survey of 11 Departments
Interviews with senior civil servants
Interviews with selected ministers
Case studies

4. Advantages of policy pilots
Evidence-informed policy and delivery
Prior assessment of risks and benefits
Identifying fault-lines and potentially expensive failures
Promoting innovation

5. Forms of evaluation Summative and formative Systematic Reviews
Before-and-after studies
Quasi-experimental designs
Randomised controlled trials – of individuals or areas

6. Obstacles Issues of ‘clinical equipoise’
Election cycles and legislative time frames
Lack of resources
Hazards of false positives/negatives

7. Consequences of present practice
Pilots that aren’t pilots
Insufficient planning and training
Overlooking what is already known
Inadequate rigour

8. Conclusions 27 recommendations, adding up to case for …
greater use of ‘genuine’ pilots across government
early discussion of ethical and practical difficulties
pilots as part of continuous policy evaluation
pilots must run their course
distinction between negative results & failures
more rigour, more RCTs, more independence, more time, more resources
“an indispensable tool of modern government”