Little pain, big gain: Data gaps and the development of cost-benefit analysis tools Jacqueline Mallender & Rory Tierney Matrix Knowledge

Agenda SectionPage Introduction/motivation3 Reviews & the issue of data gaps4 Decision modelling & dealing with data gaps5 The creation of user-accessible cost-benefit tools Conclusions Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Introduction Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Matrix: Advisory & software services Work across health and social care, justice, education & life sciences Public & private clients In the US, UK and continental Europe Rory Tierney: Senior Economist Focus on life sciences & public health and justice Design & build models Jacque Mallender: Health and welfare economist Founder, Director of Matrix Convenor of Campbell Cochrane Economics Methods Group Economist expert on Campbell Crime and Justice Steering Committee Who are we? Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

When will the cost savings come through (ROI)? What of current programs delivers most value? Which segment of population will benefit most (inequalities)? Where will I get better outcomes and the most value for my investment How do I prioritise where to invest (ranking)? Where in the pathway should I invest in and how much? What are the problems of interest in my local population? The questions: implicit or explicit Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Reviews & the issue of data gaps Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

What works? A policymaker wants to know: What are the most effective programs I can spend my money on? How much will it cost? What is the return to government and society? What should I stop spending on? What kind of impact should I expect, and for whom? e.g. how much of a reduction in morbidity and mortality, in which populations e.g. how much of a reduction in recidivism, in which offenders Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Systematic reviews Viewed as a first time approach to generate evidence base Should review evaluations of intervention ‘trials’ that: Select, critique and synthesise reliable evidence Draw from published and unpublished studies Use explicit, systematic and replicable methods For relevant interventions, applicable to the policy & societal context Quality & quantity important Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

The data gap Not enough robust published evidence Large variations in methodology General weaknesses of systematic review (Anderson & Shemilt, 2010): International variation in methods is likely and justifiable Context matters: resources required & unit costs change across geographies, time, and availability of infrastructure and services Same is true of benefits & their associated values Cannot always rely on a review of existing evidence Anderson, R., & Shemilt, I. (2010). Chapter 3: The role of economic perspectives and evidence in systematic review. In: Shemilt, I., Mugford, M., Vale, L., Marsh, K., Donaldson, C. (eds.) Evidence-based decisions and economics: health care, social welfare, education and criminal justice. Oxford: Wiley- Blackwell Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Decision modelling & dealing with data gaps Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Decision modelling The usage of decision modelling is growing: To compare interventions in context To model ex-ante policy options & assess critical assumptions and required effectiveness To evaluate ex-post policy options and assess what would need to change to improve effectiveness Incremental inputs, outputs and outcomes Relationship between inputs, outputs and outcomes Value of inputs, process and outcomes Multiple data sources Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

NICE: Public health National Institute of Health and Care Excellence Issues guidance on interventions & methods to improve public health (in the UK) Publishes on such topics as smoking cessation, substance misuse, sexually transmitted infections, nutrition etc. Approximately half of NICE public health interventions were evaluated using a decision model (as of 2012) 33% 18% 30% 20% 13% 20% 40 Public Health interventions Static modelling Decision Tree (100%) Not a model (e.g. review) Methodology not clear Dynamic modelling Markov model (75%) Discrete Event Simulation (8.3%) Regression analysis (8.3%) Other mathematical model (8.3%) Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Washington State Institute for Public Policy: Criminal justice Produces best known example of review + economic modelling in criminal justice One key output is a yearly report containing a list of evidence-based interventions with independent assessment of costs and benefits of each Perspective of Washington citizens and taxpayers Models built on rigorous review of primary studies complemented with clear and published meta analytic strategy Produces internally consistent estimates, enabling comparison of interventions and programmes Detailed methodologies published: can see continuous improvement in methods over time Enables assessment of costs and benefits in the long-run and associated risks and uncertainties Four-step research approach. 1. systematically assess evidence on “what works” (and what does not) to improve outcomes. 2. calculate costs and benefits for Washington State and produce a Consumer Reports-like ranking of public policy options 3. measure the riskiness of conclusions by testing how bottom lines vary when estimates and assumptions change. 4. provide a “portfolio” analysis of how a combination of policy options could affect state-wide outcomes of interest. Has contributed to greater improvement in crime measures in the state than national average and $1.3 billion in savings over 2 year budget cycle (Urahn, 2012) Urahn, S.K. (2012). The Cost-Benefit Imperative. Governing.com Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Overcoming the data gap Combining jurisdiction-specific economic decision models with systematic review can help overcome the data gap Data & information from systematic review can be used to populate the model The review can be ‘expanded’ by sourcing more data & using decision-analytic techniques to combine different sources Expert opinion can be used to supplement findings & assess transferability of systems Sensitivity analysis assesses the impact of uncertain variables & assumptions Decision models: Provide a structured way of demonstrating consequences Involves the construction of a mathematical representation of the relationships between inputs and results Provide an explicit way to synthesise disparate evidence available on the outcomes and costs of alternative (mutually exclusive) interventions Is a socio-technical process combining quantitative evidence and deliberative techniques Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Expert opinion There is great value in engaging experts and stakeholders in the design & development of models and identification of data It both involves decision makers/influencers in the analysis being done & supplements existing data (helps ‘mind the data gap’) Can be done through the use of surveys, interviews, workshops etc. Stakeholders can provide ‘inside’ knowledge not available in published studies By engaging them in model design, stakeholders will trust a model to make decisions Transparency is vital Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

The creation of user-accessible cost-benefit tools Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Health: Alcohol misuse & physical activity Two VBA (Excel) based tools, developed by Matrix will be available online through NICE Created to measure the social return on investment of interventions for reducing alcohol use and promoting physical activity Allows a user to see the impact in a local area of a package of policies – can compare two scenarios at once Users can customise population size, interventions, uptake and effectiveness. Models developed by synthesising existing studies with stakeholder input Mathematical/statistical techniques used to allow results to be aggregated in terms of effect sizes and subsequent health outcomes Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Justice: Reducing reoffending in young adults Online tool developed by Matrix to allow organisations to assess the economic impact of an intervention to reduce reoffending Underpinned by economic model with data from a variety of different sources such as longitudinal crime studies, UK cost of crime data and pilot interventions Developed with stakeholder input Provides a ‘business case’ – allows users to set up an intervention and access a report on its outcomes and benefit:cost Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Conclusions Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Little pain: Data gaps can be overcome Systematic reviews can provide excellent data on effectiveness, benefit-cost and comparison of interventions However, data is often limited and may be of low quality Combining systematic reviews with economic decision modelling can help overcome the data gap The review can be ‘expanded’ by sourcing more data & using decision-analytic techniques to combine different sources Expert opinion can also be sourced when published data is unavailable Must be clear and transparent Can add value to a review by being locally tailored Provides a mathematical, structural view of the consequences of interventions or programs of interventions Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

Big gain: Benefit-cost can inform policymakers Two examples of decision modelling being successfully used to inform policymakers and practitioners: NICE public health guidance (UK, health) Washington State Institute for Public Policy (US, criminal justice) Shows that decision models can help policymakers understand the choices they make when investing in one program over another Provides an evidence base and gets policymakers thinking about benefit-cost and return on investment with different combinations of programs and interventions and see the effects Taking this one step further are tools which allow the user to specify inputs and see the impact of an intervention or package of interventions: Alcohol and physical activity return on investment tools (UK, health) Business case application for interventions to reduce reoffending (UK, criminal justice) Tools can be created based on existing reviews & analysis, stakeholder engagement and expert opinion, and provide a way for users to ‘play around’ Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

THANKS! Rory Tierney Senior Economist Jacqueline Mallender Founder, Director Matrix Knowledge Kemp House City Road London EC1V 2NP +44 (0) Little pain, big gain: Data gaps and the development of cost-benefit analysis tools

The information, materials and opinions in this presentation (the "Information") are intended to describe the activities of Matrix knowledge only. It should not be used for any other purpose or in any other context and Matrix accepts no responsibility for its use in either regard. In keeping with our values of integrity and excellence, Matrix has taken reasonable professional care in the preparation of this document. Although Matrix has made reasonable efforts, we cannot guarantee absolute accuracy or completeness of information, data and any other sources used in producing this document. This document has been produced on the basis of the information and explanations made available to Matrix at the time this document was prepared. Accordingly, no representation or warranty, express or implied, is given and no responsibility or liability is accepted by or on behalf of Matrix or by any of its employees or agents or any other person as to the accuracy, completeness or correctness of the information contained in this document or any oral information made available and any such liability is expressly disclaimed. To the fullest extent possibly, Matrix disclaims any liability arising out of the use or non-use of this document and its contents, including any action or decision taken as a result of such use or non-use. Except where permitted under the provisions of confidentiality above, this document may not be made available, reproduced, retained or stored beyond the period of validity, or transmitted in whole or in part to any person without Matrix’s prior written permission. All copyright and other proprietary rights in this document remain the property of Matrix (unless otherwise provided in this document) and all rights are reserved. If you require any clarification or further information about this document, please contact Matrix on Little pain, big gain: Data gaps and the development of cost-benefit analysis tools