Quantifying the Impact of Social Science Development Research: Is It Possible? Kunal Sen IDPM and BWPI, University of Manchester Based on paper: Literature Review on Rates of Return to Research, available on DFID R4D website.

Quantifying the impact of research: the rate of return to research Similar to any other investment by the public sector, research is expected to yield benefits that are in excess of the costs of funding research. The rate of return to research is one important way that net benefits to funding research can be measured. To calculate the rate of return to research, the present value of the current and future benefits of the research is compared to the total costs of the research, and an internal rate of return is calculated to equalise the revenue stream with the cost outlays. This internal rate of return is the rate of return to research. The higher the rate of return to research, the higher is the expected net payoffs from research, and the stronger case for investing in research as compared to other types of public investment. Or for investing in one type of research versus another.

TWO QUESTIONS WHAT DO WE KNOW ABOUT THE RATE OF RETURN TO DIFFERENT TYPES OF SOCIAL SCIENCE DEVELOPMENT RESEARCH? TO WHAT EXTENT IS IT POSSIBLE TO CALCULATE RATES OF RETURN TO DIFFERENT TYPES OF DEVELOPMENT RESEARCH?

The Causal Chain from Research to Impact 1. Did the research influence policy thinking/decisions/processes (the attribution problem) 2. Did the policy intervention/change/reform lead to the observed outcome (the identification problem) 3. Can the benefits of outcome(s) be quantified? (the measurement problem)

The Attribution Problem The attribution problem can be broken down to the following components: a) how well defined is the set of research users? b) the counter-factual: will the policy change have occurred without the research taking place? c) how important are contextual factors and exogenous events in influencing policy, independent of the research?

The Identification Problem Since developmental outcomes may occur due to many reasons, and policy interventions is one possible cause of such outcomes among many others, it is often difficult to precisely identify whether the policy intervention can be causally related to the outcome in question. There are three different aspects to the identification problem: a) selection bias; b) omitted variable bias; c) Reverse causality.

The Measurement Problem An important requirement in the application of the rate of return approach is that all benefits, past, present and future, can be quantified and expressed in the same unit of value. This leads to five problems in the measurement of these benefits: a) valuing multiple outputs; b) valuing intangible outcomes; c) time-scale of measurement; d) the degree of uncertainty on the size of the impact; e) measuring effects, where there are macro-changes or strong spillover effects.

Methodologies to quantify the impact of policy change/intervention Simulation models Regression based methods Case studies Randomised control trials

The Results Chain for Different Types of Research

The Results Chain for Different Types of Research – contd.

What do we know about the rates of return to different types of social science research? Usable rates of return to research (RORs) exist – agriculture and health research Proxy rates of return do not exist, but there are credible ways to calculate RORs – infrastructure research, economic and social policy research Proxy Rates of return do not exist, and there are no credible ways to calculate RORs - governance research, climate change research.

So can we calculate the rates of return to different types of social science research? A non-starter for research which lead to intangible outcomes, where the time-scale of outcomes is very long and where the identification problem is particularly challenging– governance and climate change research. Possible for economic and social policy research – but the informational requirements for doing so are very high. Already exists for agriculture and health research.

How to improve our ability to measure the impact of research In general, there is a need for investing in improved methodologies that tackle the identification problem (but not necessarily a focus on randomised control trials only). Investing in monitoring and evaluation processes at the start of the research programme to address the attribution problem – creating baselines and using case-studies to track the impact of research. Looking at best practice on how to address the attribution problem – e.g. Fred Cardens work in IDRC. A limited use of methodologies such as willingness to pay where there are clear tangible benefits of research to address the measurement problem.