X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title of the document (can be changed under view/header and footer. Add date here if needed.1

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT NPC - title of the document (can be changed under view/header and footer. Add date here if needed.2 AGENDA 1 What is impact measurement 2 NPC’s approach 3 Quantitative approaches 4 Tips and things to remember

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT TRANSFORMING THE CHARITY SECTOR Increasing the impact of charities eg, outcomes frameworks Charity FunderSector We are a think tank and consultancy working at the nexus between charities and funders Increasing the impact of funders eg, effective commissioning Strengthening the sector eg, collaboration towards shared goals

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT WHAT IS IMPACT MEASUREMENT?

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT Impact: The difference an organisation or activity makes to the people or problem it aims to help Impact measurement: Planning, managing, tracking and reviewing how much difference you are making 5

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT NPC’S APPROACH TO MEASURING SOCIAL IMPACT

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT NPC’S APPROACH TO MEASURING SOCIAL IMPACT 7 Strategic vision / goals A well- developed Theory of Change Existing evidenceAppropriate measurement tools

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT QUANTITATIVE RESEARCH 8

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT WHY USE QUANTITATIVE RESEARCH Qualitative: understand something in detail. Rich information and insight. Quantitative: how many people? How large an effect? How much impact and how robust? Use quantitative research when dealing with large numbers of people and when you want to show a consistent effect. E.g does new type of psychotherapy lead to improved mental well- being? Need consistent measure and need large sample. Depends on type of ‘intervention’ and questions you are trying to answer. Know a lot, measure a little, know a little, measure a lot. 9

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT DIFFERENT METHODS IN QUANTITATIVE RESEARCH Ranges from easy to measure to hard to measure Easier: Hard and objective outcomes: move into employment, gain a qualification, reduction in reoffending. Harder: Softer, difficult to observe outcomes: changes in feelings, personal growth, well-being. NPC’s preference is for previously tested scales and questionnaires. If there is a gap, develop own questions. 10

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 11 TYPES OF QUANTITATIVE TOOLS AND THEIR USES 11 Type of tool Identify needs of group Measures progress for individuals Aggregate to show change Robust measure of change Explain why change has happened Admin. data  ???  Case work tool (eg, Recovery Star) ?  Clinical tools  Scales (developed by researchers)  Single questions  Hard outcomes (eg, statutory data)  

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 12 WHEN TO MEASURE Pre-post design: how outcomes change over time by collecting data before (pre) and after (post) the programme. When you think change happens influences your decision about: –When to collect the before measure –When to collect the after measure 12 Before measure After measure Programme / service

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 13 THINK ABOUT ATTRIBUTION Is the programme the only factor influencing any changes? What would have happened anyway? Control groups Compare before and after data for participants and non-participants, ideally matching on their key characteristics Holy grail: Randomised Control Trial, where potential participants randomly assigned to control group 13

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 14 THINK ABOUT SAMPLING Representativeness –Does your sample represent your population? Response rates –Is your response rate high enough to avoid bias? Statistical validity –Are your sample sizes large enough for analysis? Oversampling (segmentation) 14 Biased/small sample => Avoid percentages

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT YOU DON’T NEED TO MEASURE EVERYTHING 15

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF: 16 Has anyone already proved the causal link you want to make? Yes No Is it really important you need data on this outcome? Do you already collect this data? Don’t collect unless you decide it is really important Use research to limit your data collection Yes No Yes Can you access this data? No Develop your own data source Use this data (if it is fit for purpose) Does anyone else collect this data? No Yes

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT PRIORITISE OUTCOMES THAT: you directly influence (rather than indirectly support) are important / material to your mission are not too costly to measure will produce credible data? 17

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT HOW ROBUST DOES EVIDENCE OF YOUR SOCIAL IMPACT NEED TO BE? What do your target stakeholders (investors?) think? What is possible, given your resources etc? 18

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT LEVELS OF EVIDENCE 19 Randomised control trial Anecdotes / quotes Before and after survey Self-reported change Case studies Control groups Credibility BasicAdvanced Nesta level 1 Nesta level 2 Nesta levels 3, 4, 5

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 6 TIPS FOR MEASURING YOUR IMPACT 20 2.Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation. 1.Decide what’s important. Define your research question and be clear what you want to achieve. 3.Know your limits. You need to choose an approach that matches your resources. 4.Consider investing in training. If you are not confident, get training or seek external help. 5.Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs. 6.Use the information that you collect: If you don’t use it, don’t measure.

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT RESOURCES 21 Review of evidence of ICT and learning: of evidence of ICT and learning: A good website to look for measures and indicators: How to communicate your results: Hedley, S et al (2010) Talking about results. New Philanthropy Capital. Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco. The innovation network’s logic model workbook: The Centre for What works ( has an outcome portal where you can browse different outcomes and their corresponding indicators. TRASI ( has a database where you can search different approaches to impact assessment. NPC has a page on shared measurement where it outlines some indicators for different sectors. er.aspx er.aspx

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT RESOURCES 22 Survey question bank: A website where you can research widely used surveys and single questions by theme or using key word searches. CES’s guide to theory of change: vol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan pdfhttp:// vol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan pdf Forum for Youth Investment’s guide to measuring soft skills in youth programmes: A list of the most common psychological scales and questionnaires: Harvard Family Research Project’s tools for out-of-school time programmes: evaluating-out-of-school-time-programs-an-evaluation-resource2 evaluating-out-of-school-time-programs-an-evaluation-resource2 Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco. Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science Students and Researchers. Sage Publications Ltd Measurement and assessment tools for family and parenting interventions:

X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v Eibhlín Ní NPC - title of the document (can be changed under view/header and footer. Add date here if needed.23