Real world approaches to assessing the impact of research on policy Annette Boaz King’s College London 23rd September 2010 My interest in the use of research knowledge – interest in critical appraisal of qualitative research – searching for a study - experience at DH with research at bottom of a filing cabinet. Background in politics big believer in conspiracy – not now. Politics has a significant role to play, but lots we can do to improve impact

Outline Models of the relationship between research policy and practice Rationales for assessing impact What can we learn from the literature? 10 research impact assessment case studies Consider the implications for SSH research Wanted to come at translational research from a slightly different angle

Models Relationship – two communities Linear – relay race Engineering Diffusion Broadly speaking three different types of model of the relationship between research, policy and practice: The first, a linear model (dissemination, diffusion of innovations etc.) The second, one in which there are two communities that aim to come together around, for e.g. research advisory groups The third, where researchers work with potential users to produce research together Systems – co-production

Activities designed to promote research impact Research briefings Advisory groups Knowledge brokers Co-locating researchers and policy makers Participation in Networks Joint research projects We tend to be engaged in a wide range of activities which are designed to promote use – here are the most common ones Seminars/ presentations

Models Relationship – two communities Linear – relay race Advisory groups Models Research briefings Relationship – two communities Linear – relay race Joint research projects Systems – co-production These activities can be mapped onto the models. For e.g.

Effective knowledge transfer depends on the nature of the research and its utility the personal characteristics of both researchers and users the links between researchers and users the context for the application of the research See Nutley, Walter and Davies, 2007, pp 81-83, See also Overseas Development Institute

Why assess the impact of our research? Moderating between stakeholders Learning Accountability Why assess the impact of our research? In its quest for excellence, the European Commission must and will encourage, promote and support every effort to understand and monitor the quality of research at universities. 1 Promoting organizational achievements Performance 1Commissioner Janez Potočnik, Assessing Europe’s University-Based Research, 2010

Review question: How do you (best) evaluate the impact of research on policy? Explored the websites of 30 organizations Hand search of selected journals and reference lists Search of SPRU and IDS library catalogues Search of 10 research databases 1987-2007 UK and international sources An example is the work done by the Health Economics Research Group based at Brunel University Innovating in terms of methods for analysis and theoretical frameworks Famous study of a prominent Diabetes researcher George Alberti – tracking his papers from 1981 and interviewing members of the team. Found much from going beyond bibliometrics. e.g. The great things his students and other co-investigators went on to achieve.

Initiatives to increase research use Focusing the review on policy use Research Users Research Single study Policy makers Research programmes Systematic reviews Initiatives to increase research use

Other includes: Brazil, Denmark, Spain and Norway

Methods for measuring research impact Recently conducted a review for Defra of methods for assessing research impact on policy – drawing on 156 papers. A few points to note: Main methods: semi structured interviews, case studies, documentary analysis To note: people talk about bibliometric analysis a lot more than they do it Panel reviews: popular in Europe for their consensus building qualities Lots of health research in the area – most famous Knowledge into action, but no systematic reviews and meta-analyses, randomised controlled trials, Cohort studies, Case-control studies, cross sectional surveys

How do people evaluate the impact of research programmes? Wide range of methods and approaches (often in combination): · Qualitative methods: semi-structured interviews, documentary analysis, field visits and observations · Quantitative methods: surveys, bibliometrics and patent/new technology tracking · Panels and peer review · Workshops and focus groups · Process tracking

Ten impact assessment case studies The Foundation for Research, Science and Technology (New Zealand) French Ministry of Ecology, Energy, Sustainable Development and the Sea Land and Water Australia CGIAR Stockholm Environment Institute Ten impact assessment case studies The Irish Environmental Protection Agency Defra Environment Agency (England and Wales) Finnish Environment Institute Swedish Environmental Protection Agency

The Irish Environmental Protection Agency An example is the work done by the Health Economics Research Group based at Brunel University Innovating in terms of methods for analysis and theoretical frameworks Famous study of a prominent Diabetes researcher George Alberti – tracking his papers from 1981 and interviewing members of the team. Found much from going beyond bibliometrics. e.g. The great things his students and other co-investigators went on to achieve.

Finnish Environmental Institute Embedded approach: Ten year independent evaluation, four year strategic evaluations, internal annual evaluations, annual researcher self evaluation Goal: how to increase impact Methods: multi-method

Key themes: Diverse of objectives, scope and scale Types of approach Generic challenges Challenges specific to environmental research

What are the implications for SSH research in Latin America and Europe?

8 key questions (when designing an impact evaluation) 1. What is your conceptual framework? . What are the outcomes of interest? . What methods will best explore the outcomes of interest? . How do you address attribution? . What is the direction of travel for the evaluation? . Is this a mixed method approach, providing scope for triangulation? . Will the methods selected capture context and the complexity? . When might be the best time to conduct the evaluation? These need to be considered in the light of the time, skills and resources available for the evaluation. See Boaz et al 2009

My contact details: Annette.boaz@kcl.ac.uk Some useful references and websites: Boaz, A, Fitzpatrick, S and Shaw, B (2009) Assessing the impact of research on policy: a literature review. Science and Public Policy, 36(4): 255-270 Nutley S, Walter I, and Davies H (2007) Using evidence: How research can inform public services. Bristol: The Policy Press. O’Leary, G, Boyle, R, Donlon, B and Sheils, L (2008) Providing a framework for accountability and learning in environmental research. Administration. 5 (4): 159-170 -The Overseas Development Institute RAPID programme: http://www.odi.org.uk/programmes/rapid/ - Wooding S, Nason E, Klautzer L, Rubin J, Hanney S and Grant J (2007) Policy and practice impacts of research funded by the ESRC. A case study of the Future of Work programme, approach and analysis. Cambridge: RAND Europe. http://www.rand.org/pubs/technical_reports/2007/RAND_TR435.pdf My contact details: Annette.boaz@kcl.ac.uk

Managing Editors: Annette Boaz, Division of Health and Social Care Research, King's College London, UK and David Gough, Institute of Education, University of London, UK Volume 6, 2010, 4 issues: January, May, August and November http://www.policypress.co.uk/journals_eap.asp