1. Responding to Complexity in Impact Evaluation
INTRODUCING CONTRIBUTION
TRACING
Gavin Stedman-Bryce
20 June 2017

2. Pamoja exists to improve lives.
Since 2008 we have been inspiring change in the global not for profit sector, supporting organisations in over 30 countries, to learn and improve.
daring to find a better way

3. To introduce you to some of the features and benefits of using Contribution Tracing in impact evaluation.

4. Free journal publication
Befani, B., Stedman- Bryce, G. Process Tracing and Bayesian Updating for Impact Evaluation b.com/doi/abs/ /

5. Key Points
What is Contribution Tracing?

6. Key Points What is Contribution Tracing?
Contribution Tracing is an approach with inherent rigor
in the sense that biases and the weakness of evidence can be identified clearly… perhaps not always remedied, but like the approach is falsifiable like the scientific method, which protects it against accusations of low reliability or replicability or validity. and also it is suitable for complex contribution claims which are difficult to articulate and find evidence for. it clarifies what is the best you can do with those, makes you get the best out of the situation

7. Key Points What is Contribution Tracing?
Contribution Tracing is an approach with inherent rigor
Contribution Tracing provides guidance to the evaluator on the ‘right’ data to collect
EFFICIENCY: this is the KEY point.

8. Key Points What is Contribution Tracing?
Contribution Tracing is an approach with inherent rigor
Contribution Tracing provides guidance to the evaluator on the ‘right’ data to collect
Contribution Tracing has participatory elements
ABSOLUTELY it is what makes it strong.

9. What is Contribution Tracing?

10. Why Contribution Tracing?
Contemporary debates in impact evaluation have created space for a range of new approaches.
There is no single best approach to impact evaluation. Only the most appropriate approach based on your evaluation questions, the attributes of your programme and the available options.
There is a need/demand for more rigor in theory-based approaches to impact evaluation.
Mitigates against over- or under-claiming
Improves internal validity (Contribution Trial)
Minimises conservative bias (Use of Bayes Formula)
Minimises confirmation bias (Explores the role of complementary and alternative factors)
Fits well with the growing discourse on what is called ‘plausible contribution’ and how to measure it.

11. Contribution Tracing ¦ Day One
WHAT IS CONTRIBUTION TRACING?
Those commissioning impact evaluations are increasingly concerned with formulating and validating a ‘contribution claim’ about the role played by an intervention (or parts of it) in the realisation of one or more outcomes of interest.
Contribution Tracing is based on the principles of Process Tracing and Bayesian Updating.

12. CT = PT + BU Contribution Tracing Process Tracing is the Foundation
Bayesian updating gives strength
Bayesian updating gives strength
CT = PT + BU
Process Tracing is the Foundation

13. What is Contribution Tracing?
Contribution Tracing ¦ Day One
What is Contribution Tracing?
Contribution Tracing helps to answer the following impact evaluation questions:
How/why did the intervention work?
How/why did the intervention make a difference?
What role did the intervention play in the achievement of the outcome?
What was the contribution of the intervention to the outcome?
How did the intervention contribute to the outcome?
How confident are we that the intervention made a contribution?
We will use Bayesian analysis to understand these questions.

14. What is Contribution Tracing?
Appropriate
Single case, not comparative
To answer how/why questions
Good management commitment and support
Rigor more important than speed
Openness to using a wide variety of data sources
Not Appropriate
Multiple cases, comparative
To answer ‘how much’ questions
Little management commitment or support
Speed more important than rigor
Reliance solely on quantitative data

15. Features and Benefits

16. Provides guidance on what evidence to look for, supporting data gathering strategies.

17. Helps determine the relative strength of evidence (probative value) towards (or against) a contribution claim.

18. Contribution Tracing ¦ Day One
What is Contribution Tracing?
We use data to increase or decrease our confidence in a hypothesis about the existence of a contribution claim.
BUT not all data is the same – some is very weak, while some is very strong.
Our subjective confidence is subject to several systematic biases.

19. Focuses limited resources on gathering the ‘right’ evidence.

20. Some steps can be designed in a participatory way to engage stakeholders

21. Participatory elements
Construction of contribution claims
Development of the hypothesised causal mechanisms
Stakeholder Analysis
Contribution Trial

22. A Contribution Trial is…
A symbolic trial where stakeholders and critical friends have dialogue to assess the probative value of particular pieces of evidence for specific contribution claims

23. Contribution Tracing: built on a solid foundation

24. Likelihood Ratio = Sensitivity / Type I Error
HIGH
Confidence Increases
LOW
Sensitivity
No Confidence Change Line
Confidence Decreases
Type I Error

25. Qualitative Descriptors of Confidence
Practical Certainty
> 0.99
Reasonable Certainty
0.95 – 0.99
High Confidence
0.85 – 0.95
Cautious Confidence
0.70 – 0.85
More Confident than not
0.50 – 0.70
No information
0.50
Rejection of the component of the claim
< 0.5
Increasing confirmation of the component of the claim
Increasing rejection of the component of the claim

26. Pamoja.uk.com for more information

27. Like us on Facebook www.facebook.com/pamojaevaluation
Follow us on
Like us on Facebook