Download presentation
Presentation is loading. Please wait.
Published byRandolf Merritt Modified over 9 years ago
1
Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual Metrics Conference June 23-24, 2015
2
Overview What you can expect What we’re going to do What APEP does
3
Starting point: I want to assess my impact, but I can’t run a large-scale RCT
4
When? Random assignment isn’t possible The intervention is changing over time Impacts are not adequately captured by quantitative measures Causal pathways are complex Small-scale projects We are not hypothesis testing
5
Alternatives? Quantitative Surveys Documents/records Qualitative Interviews Focus groups Site visits Open-ended survey questions Document review Participatory methods
6
Can we still say something meaningful about impact?
7
Absolutely!
8
How do I optimize my ability to collect meaningful and useful data on impact?
9
Prioritize Who is your audience?
10
Decisions, decisions, decisions…
11
Am I looking at my impact on SGBs or my impact on the people the SGBs serve? ANDE member does X Impact on SGB Impact on individuals, communities, etc. SGB does Y
12
Example:
13
Root Capital provides loans to Freshco Freshco is able to produce more high- yield, drought- resistant maize seed Farmers have increased yield (more food, increased income, surplus grain for local population) Greater access to high-quality seed for farmers
14
Am I looking at short-term or long-term impacts?
15
Am I in hypothesis testing mode or exploratory mode? Did I have X impact? What impacts did I have? How or why did I have an impact? vs.
16
Make the most of your measures
17
#1. Measure outcomes, not just outputs OutputsOutcomes Your products/services to SGBs. Impact.
18
#2. Not just “whether” but also “how” and “why” Did I have an impact? Whether How Why How did my activities contribute to a positive impact? Why didn’t I have an impact?
19
Vote with your You’ve worked with quantitative methods You’ve worked with qualitative methods
20
#3. Consider qualitative and quantitative Combine if possible! Quantitative Qualitative
21
#4. Sample with purpose
22
How big is big enough? Sample size calculator: http://enterprise- development.org/page/ calculator Sample size calculator: http://enterprise- development.org/page/ calculator
23
#5. Scout external factors
24
#6. Try for two (or more) Time 1Time 2 T1T2 T3 T4 T5
25
#7. Optimize questionnaires Is this clear? If everyone answers “yes,” what will that tell me? Ask good questions Be mindful of burden Beware of branching Too many questions can lead to fatigue and skipping Sub-groups may be too small to analyze
26
and share Measuring outcomes rather than outputs Measuring “how” or “why” Combining qualitative and quantitative Sampling External influences Collecting data at multiple points Questionnaires
27
Ok, I’ve collected my data. Now what?
28
Data analysis: Things to check
29
Your sample What’s my response rate? Anybody under/over- represented? Does the pre match the post? Are there missing data?
30
Analysis Does it make sense to calculate statistical significance? Should you disaggregate? What’s the appropriate way to analyze your qualitative data? Is that calculation meaningful?
31
Assess your limits Attribution vs. Contribution Counterfactuals Generalizability
32
What to do when…
33
Group Exercise
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.