The Payback Framework: & developments in assessing policy and economic impacts of health research & application to development research Stephen Hanney Health Economics Research Group, Brunel University UKCDS/DFID/IDRC workshop: Evaluation of Research Impact: Options and Challenges Wellcome Trust: 18 October 2012

Overview of presentation HERG Payback Framework to trace impacts from specific bodies of health research - 2 elements; applications; strengths/weaknesses. Payback Framework stream developed in several ways including: - research impact on policy - economic benefits of medical research More speculatively: would it be possible to combine approaches for development research? - use the Payback Framework to assess impacts as ICDDR,B doing - address the inevitable controversies around the economic benefits, and explore the possibilities of studies on this

HERG’s Payback Framework Payback Framework has 2 main elements (Buxton & Hanney, 1996) Multidimensional categorisation of benefits or paybacks: - knowledge production - targeting future research, capacity building, & absorption - informing policies & product development - health and health sector benefits, eg better health, health equity etc - broader economic benefits (overlaps and changing definitions) Logic model of how to assess the benefits: - 7 stages - 2 interfaces

The payback framework: logic model for assessing benefits Adapted from: Hanney S, Gonzalez-Block M, Buxton M and Kogan M, 2003.

Payback framework to assess the impacts research: corticosteroids for prevention of RDS Used payback framework to organise data to show benefits from work of Liggins on ante-natal use of corticosteroids for prevention of Respiratory Distress Syndrome Researcher moved from animals to humans: randomised the first patients to receive the treatment (unusual, important in attribution) Considerable impact in most categories (Hanney et al, 2005) - knowledge production: major, widely cited publications - informed research of many others (these studies inspired the Cochrane logo) - influenced clinical policies in many countries - thousands of babies survived, plus reduced morbidity & possible cost savings - broader economic benefits in terms of human capital more complex to show Raises nationality/attribution issues in impact assessment

Cochrane logo: Liggins’ RCT and subsequent trials

Payback framework to assess research benefits: corticosteroids for prevention of RDS - evidence The evidence to attribute the impacts to Liggins research gathered in various ways including: Archival analysis – access to some documents from funder Bibliometric analysis shows many later researchers cited this work, and in many later trials this was the key reference Review of policy statements such as guidelines, consensus statements shows this work is cited as a key paper Review of series of textbooks shows how early this work cited and leading clinicians were influenced by it Interviews with some key players in this story (in this instance did not use the surveys that often used in other studies)

Payback Framework: limitations, strengths Limitations: - often difficult to identify impacts in later categories eg health gain - the full impacts might not have occurred by time of assessment - tends to be resource intensive to apply - limitations of working forwards from individual projects which can downplay role of other contributions Strengths: - includes range of benefit categories relevant to funders (Bacon, 1627) - builds on major UK analysis of the collaborative approach - provides a conceptual framework for organising application of various methods (surveys, interviews) in a consistent way across a programme - often successfully engages researchers in data collection - facilitates presentation of data in a consistent way - provides a framework for working forwards, addressing many attribution issues, & maximising opportunities to identify impacts that have occurred - facilitates analysis of factors associated with impacts

Payback Framework & policy impacts Alternative opening of Jane Austen’s Pride and Prejudice: ‘It is a truth universally acknowledged that a single researcher in possession of a good finding must be in want of a policymaker’ (Hanney, 2005) Rothschild experiment in government departments in 1970s: formative evaluation in health department by Kogan & Henkel (’83;’06) - collaborative approach developed & analysed: importance but difficulty of getting policymakers & researchers to work together on agendas etc; - knowledge brokerage role in getting research used by policy receptors; - importance of research evaluation criteria recognising policy relevance. Ideas informed Framework on ways to assess research impact on policy: - collaboration might help to illustrate attribution/contribution of policy impact; - Framework can help analysis of research’s policy impact in a variety of systems But key issue for evaluation: is research impact from collaboration and pushing own findings always good – what about SR? - the role of mechanisms/receptors to absorb SRs & other research.

Development of framework: economic returns Reviewed literature for WHO, very little in LMIC (Buxton et al., 2004) Then study for MRC/Wellcome Trust/Academy of Medical Sciences: Economic returns from health research to the UK consist of two main elements :  The UK health gain attributable to the research  UK GDP gains [This assessed by OHE] Cardiovascular (CVD) and mental health (MH) research used as exemplars [Just CVD shown here] 2 main approaches possible: - top-down macro approaches that take health gain of nation and value it; - estimate health gain by detailed ‘bottom-up’ approach (Buxton et al., 2008 )

Development of framework: economic returns Estimate public and charitable expenditure on CVD research Estimate net value of CVD health gains attributable to research in each year from 1986 – 2005: - started with 45 specific health interventions in CVD (eg use of statins for secondary prevention of CHD post MI), calculated health gain of each. - used the opportunity cost of achieving health gain value of a QALY (Quality Adjusted Life Year) consistent with NICE and allowed for costs of treatment Estimate % of health change attributable to UK research and time- lag from research to health benefit Health gains to UK residents deliver a rate of return for CDV of 9%; Non-health gains (GDP spill over gains) best estimate 30% Best estimate of total gain equivalent to IRR of 39% per year

Are these approaches relevant for development research? The UK economic returns study with its data requirements etc not replicable in LMIC? But is it acceptable/feasible at all to consider the value of the health gain from health research and build on specific examples? Our report and the 39% IRR figure was the first reference in the Royal Society’s submission to the DFID’s 2009 consultation on eliminating world poverty. This might give some legitimacy to the general approach. Meltzer D (2006) chapter on ‘Economic approaches to valuing global health research.’ in: World Bank report by Jamison et al., eds, Disease Control Priorities in Developing Countries: - describes studies showing value of a statistical life increases with income; - claims the value of recent health gains in countries such India is larger than for the USA, even if a local statistical value of a life used; - to move from the finding that gains in health are highly valued to the finding that health research is highly valued, gains in health must be connected to health research. [So, back to the start]

Combining approaches for development research? Can we use the Payback Framework to show examples of research impact (eg ICDDR,B) elements of the Buxton (2008) approach for development research? Key point from our study is that we do not rely on valuing a statistical life. Instead we rely on identifying the amount a system is willing to spend to generate health gains. Factors to consider: - might the time lags might be shorter if much development research is applied research? - how far is local research or the global stock of knowledge contributing to achieving health gains? Could case studies using Payback Framework help address some of these issues of impacts in relation to specific examples?

References Buxton M, Hanney S (1996) How can payback from health services research be assessed? J Health Serv Res & Policy, 1: Buxton M, Hanney S, Jones T (2004) Estimating the economic value to societies of the impact of health research: a critical review. Bull WHO, 82: Buxton M, Hanney S, Morris S, et al (2008) Medical Research - What's it worth? Estimating the economic benefits from medical research in the UK. London: UK Evaluation Forum (Academy of Medical Sciences, MRC, Wellcome Trust). Hanney S, Gonzalez-Block M, Buxton M, Kogan M (2003) The utilisation of health research in policy-making: Concepts, examples and methods of assessment. Health Res Policy Syst, 1:2 Hanney S, Mugford M, Grant J, Buxton M (2005) Assessing the benefits of health research: Lessons from research into the use of antenatal corticosteroids for the prevention of neonatal respiratory distress syndrome. Soc Sci Med, 60: Hanney S (2005) Personal interaction with researchers or detached synthesis of the evidence: modelling the health policy paradox. Evaluation and Research in Education, 18: Hanney S, Buxton M, Green C, Coulson D, Raftery J (2007) An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess, 11(53). Kogan M, Henkel M, Hanney S (2006) Government and Research; Thirty Years of Evolution. (Second edition). Dordrecht: Springer.