Institute for Policy Research Evidence for Agile Policy Makers The Contribution of Transformative Realism Graham Room 14 February 2014 Victoria University Wellington

Institute for Policy Research Evidence for Policy Assess impacts of different interventions by systematic review of evidence ‘What Works Centres’: National Institute for Health and Clinical Excellence.National Institute for Health and Clinical Excellence Education Endowment FoundationEducation Endowment Foundation Centre for Local Economic GrowthCentre for Local Economic Growth

Institute for Policy Research X Y

The Hierarchy of Evidence and Rigour 1aEvidence from a systematic review of randomized controlled trials 1bEvidence from at least one randomized controlled trial II aEvidence from at least one study without randomization II bEvidence from at least one other type of quasi-experimental study IIIEvidence from non-experimental descriptive studies (e.g., case control studies, correlation studies) IVExpert consensus

Institute for Policy Research Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective ‘Impact’ (medical) versus ‘engagement’ (social policy) Interventions unfold in institutionally complex environments Rarely is it the ‘same’ intervention that is delivered across all contexts Pawson’s Realist Critique

Institute for Policy Research Pawson’s contingencies of intervention AA1A11A12A2A21A22

Institute for Policy Research Darwin’s Tree of Life

Institute for Policy Research ‘ Transformative realism’ A A1A2 A3 B B1 B3 B2

Institute for Policy Research EBPM: Evidence of impact of a given intervention: universal and unconditional Pawson: Impact of a given intervention under a range of contingencies Transformative realism: Dynamic synergies among interventions Three Paradigms of Evidence-Based Policy-Making

Institute for Policy Research The Choice of Paradigms

Institute for Policy Research Paradigm 1: Well-evidenced generalisations about impact (universal truths?) Paradigm 2: Well-based theory as to out- turn under specific local contingencies Paradigm 3: Need real-time watching and learning-by-doing Types of Knowledge

Institute for Policy Research How to tune social developments in real time? Where to probe and where to avoid? What signs of tipping points to avoid or approach? What data in real time? Big Data? Ian Stewart (1997), Does God Play Dice?) Tuning a complex system (engineering and medicine) Can this approach be applied to social systems? Real-Time Watching and Learning-by-Doing

Institute for Policy Research How to ‘read’ the ‘sea conditions’? Networks of ‘navigators’ – tracking how policy ecosystem is developing and deciding what ‘tuning’ is needed Elizabeth Eppel, Amanda Wolf, David Taylor: Complex Policy Implementation: The Role of Experimentation and Learning ‘Ordinary’ situations - rule of thumb Anomalous (extraordinary) situations – examine carefully and devise a specific strategy What thresholds to take as triggers for concern? Real-Time Watching and Learning-by-Doing

Institute for Policy Research Danger of de-politicising policy What works for whom? Corporate and other interests try to shape our policy futures: is there any shared national interest? Good research on impact is necessary but not sufficient for good policy Concluding Reflections

Institute for Policy Research Behavioural Insights Team (2012), Test, Adapt, Learn, UK Cabinet Office Eppel, E., Turner, D., & Wolf A. (2011). ‘Complex policy implementation: The role of experimentation and learning’. In B. Ryan & D. Gill (Eds.), Future State: Directions for public management in New Zealand (pp ). Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective Room, G. (2011), Complexity, Institutions and Public Policy: Agile Decision- Making in a Turbulent World Room, G. (2013), 'Evidence for Agile Policy Makers :The Contribution of Transformative Realism', Evidence and Policy, 9(2): Stewart, I (1997), Does God Play Dice? References