nef (new economics foundation) Workshop: Value for Money (VfM) in international development using Social Return on Investment (SROI) Natalie Nicholles nef (new economics foundation) 19th October 2010

Housekeeping Ground rules Fire exits Toilets Workshop materials

Agenda 09:30 Big picture introduction to Value for Money and SROI 10:00 Crash course on SROI: understanding change 10:45 Tea/coffee break 11:00 Crash course on SROI: measuring change 11:30 Crash course on SROI: valuing change 12:30 Lunch 13:15 Crash course on SROI: attribution & partnership working 13:45 Crash course on SROI: forecasting long term benefits 14:45 15:00 Worked-through example & organisational decision making 15:45 Answering key questions 17:00 Close

Aims for the day To present an overview of what Value for Money (VfM) means To introduce ways of capturing Value for Money in international development through the Social Return on Investment approach To share nef’s work on this subject to date To facilitate discussion on how to further develop demonstrating value and impact of international development work

new economics foundation (nef) Founded in 1986 Economics Think Tank working to promote: Innovative solutions that challenge mainstream thinking on economic, environmental and social issues nef consulting is the consultancy arm. It exists to promote and disseminate nef solutions History of working with public sector on implementing value for money solutions

Problems with measurement Financial measurement: limited measure of value 2. We allocate resources only to the things we can measure 3: Stakeholders are left out of decision making

The challenge Measurement across the ‘triple bottom line’ The economy The environment People

Value for Money (VfM) VfM is about making sure that spending achieves as much as possible A way of making decisions about how to use limited resources Sometimes mistaken for lowest unit cost Without a measure of quality or effectiveness, risk false economies E.g. Children in care

Treasury VfM Definition “VfM is defined as the optimum combination of whole-of-life costs and quality (or fitness for purpose) of the good or service to meet the user’s requirement. VfM is not the choice of goods and services based on the lowest cost bid.” HM Treasury (2006) Value for money guidance

Treasury VfM Definition “Wider social and environmental costs and benefits for which there is no market price also need to be brought into any assessment. They will often be more difficult to assess but are often important and should not be ignored simply because they cannot easily be costed.” HM Treasury Green Book Effectiveness, not lowest cost Wider costs and benefits Whole-of-life costs

Value for Money Resources / Investment Often VfM is understood by comparing unit costs Resources / Investment Service & Wider Outcomes Money Economic People Inputs Outputs Social Environment Environmental Real VfM is achieved by comparing outcomes with investment

How does SROI fit in? Framework for telling us how effectively money is spent – make VfM decisions Adjusted form of cost-benefit analysis. SROI = value of positive + negative outcomes investment (or cost) Essentially a measure of the efficiency of achieving outcomes It considers triple-bottom-line benefits and investments – economic, social & environmental

History of SROI Mid 1990s: REDF & Jed Emerson Early 2000s: new economics foundation Mainstreaming: nef consulting & others

SROI today Third Sector: SROI Guide Local Government: commissioning Think tanks/public policy Central Government Department of Health National Audit Office: VfM DFID: VfM Pilot: HIV Aids Alliance International interest Europe, Australia, Canada, Asia

Crash Course on SROI: Understanding change

SROI Principles 1. Involve Stakeholders 2. Understand what changes 3. Value the things that matter 4. Only include what is material 5. Do not over claim 6. Be transparent 7. Verify the result

SROI Process Engage stakeholders to identify outcomes Data collection Deadweight, attribution, displacement Benefit period and drop off Model and calculate Valuation of non-traded outcomes Report

Features that enhance VfM measurement Focus on outcomes Places quality and effectiveness at heart of analysis Measures what matters, not what is easiest Stakeholder-informed Grapples with outcomes measurement Values traded and non-traded outcomes Failing to monetise non-traded outcomes effectively they gives these a value of zero Rigour and transparency Concerned with impact: deadweight, displacement, attribution, drop off

Develop a theory of change Inputs Activities Outputs Outcomes Output: Tells you an activity has taken place and is usually quantitative (e.g. number of people trained) Outcome: The change that occurs as a result of an activity (e.g. improved well-being of training participants)

Case study: African Community Development Foundation (ACDF)  Background ACDF started of as a U.K community based organisation, supporting socially isolated Africans in various ways. In 2009, ACDF sent 28 volunteers as a pilot in Kisumu, Kenya - known as Diaspora Development Associates (DDA). ACDF specialise in team volunteering Kenya has a preference for formal employment rather than self-employment but ormal employment hard to get High level migration from Kisumu to Nairobi to secure jobs Due to preference for formal employment, very limited public interest for business training in Kisumu so organisations do not provide it. The Government of Kenya in 2008 launched an ‘enterprise fund’ to encourage people to start their own business but there was hardly any uptake of these funds in Kisumu. 20

Case study: African Community Development Foundation (ACDF) Activity ACDF delivered Train the Trainer courses to infrastructure organisations and to direct beneficiaries. This comprised of DDAs with skills covering- business set up, business management, book-keeping, organisational development, fundraising etc. 21

Exercise 2: Impact Mapping What changes do the stakeholders experience as a result of the project? I.e. what is the story of change?

X Impact map Stakeholders Inputs / Outputs Outcomes Activities Train the trainer courses on: business set up, business management, book-keeping Kisumu’s unemployed resident X Kisumu’s infrastructure organisations ACDF’s volunteers (DDAs) State 23

Outputs vs. outcomes Outputs do not always lead to the desired outcome Key challenge in determining VfM is measuring outcomes Outcomes measurement rarely takes place Biggest obstacle in our work across sectors/organisations is the lack of outcomes data

Crash Course on SROI: Measuring & valuing change

Measuring outcomes Some outcomes are ‘hard to measure’ subjective, qualitative, diverse, far removed from the activity, long term Systems and tools for capturing complex outcomes are improving distance travelled translating into quantitative format that enables aggregation collecting complex information simply

Evidencing outcomes Distance travelled

Example: National Accounts Model of well-being www.nationalaccountsofwellbeing.org

Evidencing outcomes Select ‘ways of knowing’ that an outcome (a change) has taken place = indicators Stakeholder Stakeholder (A): long term unemployed Kisumu resident Outcome Beneficiary (A) has sustainable, meaningful employment Indicators (A) sustains employment for at least 6 months (A) reports levels of job satisfaction (A) improved financial situation

Ways of measuring Stakeholder Outcome(s) Indicators (Data Collection Method) Stakeholder (B): Volunteers (DDAs) Increased employability (B) increase in salary Increased well-being e.g. confidence, empowerment, aspiration (B) reported improvements in well-being Stakeholder (C): Kisumu State Improved economic situation Increase in tax take Reduction in use of state services (if relevant) Stakeholder (D): UK State Improved economic situation (from volunteering) Increased tax take from salary increase

Valuing SROI analyses value to all material stakeholders, not just the one funding the activity. Key question: “what does this (change) mean to you?” Exercise Imagine your employer gave you a day off. What will you do on your day off? What does this free time mean to you? If you had to put a financial value on this, what would it be? I.e. what is this day off worth to you?

Valuing and Pricing Value means different things to different people. How much is your house/flat worth? Who sets the price of fish? BUT What does it mean to a Kisumu resident to have a job? SROI uses financial proxies to estimate the social value of non-traded goods to different stakeholders. Common currency

Subjectivity of value Value is adjusted...not by any accurate measure, but by the higgling and bargaining of the market, according to that sort of rough equality which, though not exact, is sufficient for carrying on the business of common life.

is sufficient for helping Subjectivity of value Can I put a value on anything? Can I find a value which, though not exact, is sufficient for helping me evaluate change?

Financial proxies Sometimes this is straightforward mainly with outcomes to the State (e.g. value of health) More difficult with non-traded outcomes These do not have a ‘price’ and so require a proxy, or stand-in (e.g. emotional well-being) Standard economic valuation techniques Contingent Valuation Willingness to pay (e.g. new park) or willingness to accept compensation (e.g. noise pollution) Revealed Preference Hedonic pricing (e.g. high risk job); Travel cost method (e.g. local service); Observed spending on related goods (e.g. leisure)

Monetisation Each outcome is then valued Where no direct financial value is available, we use financial proxies to represent the social value created Stakeholder Outcome Possible proxy value Beneficiary (A): long term unemployed Kisumu resident Beneficiary (A) has sustainable, meaningful employment £ Value of increased earning potential OR £ Cost of fees for a skills development training course Stakeholder (B): Volunteers (DDAs) Increased well-being e.g. confidence, empowerment, aspiration Contingent valuation: what would (B) pay for to get similar level of well-being? Revealed preference – what do we observe (B) does to get a similar level of well-being?

Valuation exercise Exercise: In pairs, select 2 outcomes from the ACDF impact map Think about options for financial proxies for the outcome. Remember, we are putting a value on the outcome for that particular stakeholder!

Crash Course on SROI: Attribution & Partnership working

Understanding impact Deadweight: what would have happened anyway? Attribution: how much is down to this project, and how much down to other factors? Displacement: have we just moved an outcome to / from somewhere else? Benefit period: how long does the outcome last, and does the effect ‘drop off’ over time?

Impact: Attribution Attribution – how much credit can your organisation take for the outcomes? Expressed as a percentage Not an exact science Methods: Based on your understanding of the journey of change Involve stakeholders – interviews or surveys Consult with other organisations

Attribution stages No account of attribution levels Effectively attribution = 100% 2. Informed estimate of attribution levels Qualitative research allows SROI practitioner to divide attribution between contributors 3. Quantitative research Quantitative stakeholder research gives average stakeholder’s view on contributors Academic research Academic research can help refine (although not replace) stakeholder research

Quantitative stakeholder research Attribution Time (years)

Partnership working

Projecting into the future Benefit period – the length of time over which outcomes are expected to endure e.g. benefits of an employment training programme may endure for some years after the course Drop off – the rate at which benefits decrease over time e.g. it is likely that benefit for training participants wears off as time goes on Time value of money : discount rate NB. Comparing value in different countries

Inputs Size of investment If just looking at one project within organisation, need to work out its ‘share’ of inputs (Full Cost Recovery) Where multiple funding streams, need to have identified the share of benefits when scoping if looking at return on just one of these investments Include financial and non-financial inputs e.g. volunteer time

Example & Organisational decision-making

Hypothetical Example INGO in Nigeria Outcome: capture the value of the influence on national policy-making i.e. changes for stakeholders that result from the national policy. INGO in Nigeria Part of National Task Group on Sanitation Successfully completed community-led total sanitation pilot (CLTS) Advocated within the Task Group for drafting of National Strategy for Scaling-up Rural Sanitation and Hygiene Key lesson: INGO’s advocacy on the National Strategy is not the outcome – it is a means to an end.

INGO Impact Map Outcomes Value of improved health to the individual Value of increased economic activity to the State

Value to the individual No monitoring in place to track impact of the National Strategy Forecasted using secondary research and stakeholder engagement Outcome Incidence Assumption 1: An additional 2% of the rural population is provided with improved access to sanitation. Disease Number of incidences avoided Diarrhoea 29596 Schistosomiasis 19319 Ascariasis 85601 Hookworm 9111

Value to the individual Impact Deadweight (population reached) -0.25% baseline trend improvements in sanitation for rural communities (UNICEF/WHO) Attribution to National Strategy 75% significant change but concurrent factors included: International Year of Sanitation 2008 which saw increases in funding and other new initiatives Attribution to INGO 5% number of other actors involved in Task Group e.g. UNICEF, DFID. Credit for role in policy formation is 35% but implementation involved other actors including local government and this reduced attribution to 5%.

Value to the individual Proxies – value of health to individual is made up of: Increased income: fewer days lost to illness (working age population) Based on secondary research on average rural wage in Nigeria Direct costs associated with illness (all ages) Based on WHO research into transport & medical expenses Value of increased well-being (all ages) Non-traded outcome. (Limited) stakeholder engagement to find out what investments rural Nigeria might make to improve well-being. Cost of generator divided by 5 for value to individual. All proxies converted in US$ using World Bank PPP

Value to the individual Benefit Period Health improvement dependent on continued access to sanitation WaterAid research suggests 5 years benefit period Policy assumed to be in effect for 3 years Drop-off WaterAid research suggests 16% drop-off over 3 years. Year 1: 2% Year 2: 4% Year 3: 10% Years 4 & 5: 20% per year (nef estimate) Discount rate: 10% (DFID)

Value to the individual Social Value Total value of improved health to individuals as a result of policy $22,561,910.66 INGO share of value $1,128,095.53 Cost incurred by INGO during policy influencing $14,863.00 Return on investment $1:$75.90 Caution: benchmarks needed to understand whether ratio represents VfM

Value to the individual Sensitivity Analysis ($1:$75.90) Assumptions in model systematically varied. Decreasing outcome by 95% = $1:$11.81 Well-being proxy is a sensitivity Doubling proxy = doubling ratio Removing proxy = $1:$1.21. Proxy informed by limited stakeholder engagement therefore worth investigating further Reducing attribution to 2% reduced ratio to $1:$30.36

Decision-making: organisational Model to commission/plan/ design services or projects Recommendation to DFID Adoption by Local Authorities

Organisational Decision-Making

Sustainable Commissioning Commissioner & service user priorities Community strategy & Corporate priorities 1. Activity 2. Output 3. Service level outcomes 4. Camden Community outcomes social economic environment 5. Value Quantitative Qualitative Monetiseable Where value accrues: To Service Camden wide central government National outcome frameworks

Answering key questions As a sector, how do we take this forward as a sector in terms of demonstrating the value and impact of our work? Is this is an approach that can be useful? In what circumstances can it be useful? What are the limitations of this approach? Where are the opportunities to for this to be adopted? Any other reflections/thoughts on the approach?

Summary: SROI as VfM measurement Focus on outcomes Places quality and effectiveness at heart of analysis as only outcomes tell you that change has occurred Values traded and non-traded outcomes Enables consideration of wider costs and benefits Failing to monetise non-traded outcomes effectively they gives these a value of zero Stakeholder informed Ensures that you measure what matters to those experiencing the change – i.e. ‘quality of fit’ for service user Rigour and transparency Concerned with impact: deadweight, displacement, attribution, drop off 59

Summary SROI is a tool for assessing VfM and for achieving continuous improvement Proving and improving Not an ‘all or nothing’ approach, complement existing M&E systems Next steps – from Bond or nef Pilot?