1. Module 4: Understanding the Evaluation Context and Program Theory of Change

2. Introduction Front-End Analysis
Identifying the Main Client and Key Stakeholders
Understanding the Context
Investigating Existing Knowledge
Constructing, Using, and Assessing a Theory of Change
IPDET © 2009

3. Front-End Analysis Client and stakeholders Timing Time management
Nature of resources
Relevance of social science theory
Findings of similar evaluations
Role of theory of change
Existing data
IPDET © 2009

4. Balancing Cost and Benefits: Benefits
Evidence-based decision making that leads to sound decisions about:
what to scale up or replicate
What to continue
what to improve
what to decrease or terminate
Contributions to streams of knowledge
Building of local capacity
IPDET © 2009

5. Balancing Cost and Benefits: Costs
Cost of the evaluation in relation to the cost of the program
Burden of the evaluation on those who must supply information
Reputation and political costs to the organization, the evaluator, and the evaluation community if the evaluation is not done well
IPDET © 2009

6. Dangers or Pitfalls Resistance to modifying the original plan
McDonaldization
Fixed beliefs (“truisms”)
“Group think”
Disproportionate weighting of the views of the powerful
IPDET © 2009

7. Main Client(s) May be those: authorizing or requesting the evaluation
authorizing or funding the program
accountable to the public for the intervention
to whom the evaluators are accountable
IPDET © 2009

8. Stakeholders
People or organizations other than the client with strong interests in the intervention and its evaluation
May stand to gain or lose from the outcomes
May represent the interests of such groups
Important to consider those who would typically not be asked to participate
IPDET © 2009

9. Checklist of Stakeholder Roles
Individuals, groups, or agencies
To make policy
To make operational decisions
To provide input to evaluation
To react
For interest only
Developers of program
Providers of other resources
Heads of implementing agencies
etc.
IPDET © 2009

10. Involving Stakeholders
Identify the main stakeholders
Provide early opportunity for them to raise issues for the evaluation to address
Consider offering periodic updates or briefings
Evaluation report should indicate the nature and extent of stakeholder involvement
IPDET © 2009

11. Stakeholder Analysis
A technique to identify and assess the importance of key people, groups, or institutions
Assists in anticipating the influence stakeholders may have on
mobilizing support or resistance to the evaluation
use of evaluation findings and implementation of recommendations
Evaluator may develop strategies to involve appropriately in the evaluation
IPDET © 2009

12. Program Stages and Evaluation Questions
Examples of evaluation questions
1. Prospective
What can be learned from the experiences of similar programs?
2. Early implementation
Is the program operating as planned? What issues are surfacing?
3. Mid-implementation
To what extent are there cross-site variations in how the program is being implemented?
4. Maturity
To what extent have the outcomes been achieved? Can gains be attributed to the program? Has the program had unanticipated positive or negative effects?
5. Ex-poste
To what extent did the program contribute to the observed impacts?
IPDET © 2009

13. Tapping Existing Knowledge
Own organization’s internal reviews and published evaluations
Other organizations’ websites with evaluation publications
Repositories, syntheses, or collections of evaluations
Evaluation journals
Doctoral dissertations
Related literature: e.g. economics, statistics
IPDET © 2009

14. Using Knowledge for a “Theory of Change”
Identify stream of knowledge that explains or predicts an event (a theory)
example: evidence from evaluative research that indicates students do better when parents are involved in their studies
Use this knowledge to develop a “theory” of how the program will achieve intended change
example: if parents in rural Zambian primary schools are more involved in their children‘s studies, the children will do better in their studies
IPDET © 2009

15. Theory of Change
A knowledge-based diagram of how an intervention intends to achieve results
T of C should:
depict a sequence linking inputs to activities, activities to outputs, and the results--outcomes and impacts– expected from them.
identify critical assumptions underlying the intervention
Indicate other factors in the context of the intervention that might influence its success
IPDET © 2009

16. Potential Influences on Program Results
Political Environment (Governance, etc.)
Policy Context
Inputs  Activities  Outputs
Outcomes  Impacts
Black box
Macro-economic Picture
Public Attitudes
Aid Players
Environment
IPDET © 2009

17. Why Use a Theory of Change
Visually shows how the intervention is supposed to work and flaws in logic
Assumptions and potential influences identified may indicate risks to the achievement of intended results
Provides questions for an evaluation
Helps build a common understanding of the intervention and expectations among stakeholders
IPDET © 2009

18. Review or Create
If the intervention already has a developed theory of change, carefully review it
May need to refine, rework, or reconstruct
If there is no theory of change, create one
IPDET © 2009

19. Constructing a Theory of Change
What knowledge underlies the intervention? [Slide 13]
What is the logic of the intervention?
What are the key assumptions being made?
What are potential influences operating in the context of the intervention?
IPDET © 2009

20. How to Determine the Logic of the Intervention
Chain of “if-then” statements
If X happens, then Y should follow
Ask, is this plausible? Are the activities and outputs likely to lead to the intended results?
Ask, is this feasible? Are the inputs (capabilities and resources) sufficient for the activities?
IPDET © 2009

21. Mapping or Drawing the Logic
Can and are done in many different ways
Usually use forms of flow-charting
Boxes with arrows show links and relationships
IPDET © 2009

22. Results Chain Area of Control Internal to the Organization
Inputs (Resources)
Activities
Outputs
Outcomes (Short-Term)
Outcomes (Intermediate)
Impacts (Long term)
Area of Control Internal to the Organization
Outputs Reach Direct Beneficiaries
Area of Influence External to the Organization
External Factors
IPDET © 2009

23. Results Chain for Training Program
Inputs 
Activities
Outputs
Outcomes
Impacts
Resources
▪ Money
▪ Staff
▪ Volunteers
▪ Supplies
• Eligible participants
Services
▪ Training
▪ Education
▪ Counseling
Products
▪ Total number of classes
▪ Hours of service
▪ Number of participants completing course
Benefits
▪ New knowledge
▪ Increased skills
▪ Changed attitudes
▪ New employment opportunities
Changes
▪ Trainees earn more over five years than those not receiving training
▪ Trainees have higher standard of living than the control group
IPDET © 2009

24. Simple Theory of Change Model
Training
High Quality Evaluations
Useful Information
Better Decisions
We should get better decisions
By training
IPDET © 2009

25. Identifying Key Assumptions
Make explicit the implicit assertions about how the intervention will work in its context to produce the results
IPDET © 2009

26. Theory of Change with Assumptions
Resources given to follow what they learned
Training
High Quality Evaluations
Useful Information
Better Decisions
Needs of trainees met
Report writing skills adequate to communicate with government
Enough time to learn
IPDET © 2009

27. Theory of Change with Assumptions and Context
Macro economic environment
Foundation programs
Resources given to follow what they learned
Training
High Quality Evaluations
Useful Information
Better Decisions
Needs of trainees met
Report writing skills adequate to communicate with government
Enough time to learn
Government- funded related programs
NGO’s programs
Worker remittances
Other bilateral micro-financed MDM programs
IPDET © 2009

28. Flow Chart of Classic Logic Model
Inputs
Activities
Outputs
Outcomes
Impacts
IPDET © 2009

29. Exercise: Unlabeled Theory of Change
Visits by teachers to students’ homes
Sharing of views by parent and teacher
Teachers’ understanding of the home culture
Teachers’ sympathy with children and their view of the world
Teaching in terms comfortable and understandable to students
Student morale
Achievement in reading
Parents’ knowledge of school’s expectations for students
Parental support and encouragement with child’s homework and school assignments
Conscientiousness of work by students
Identification of special problems that retard student’s achievement (health, emotional, etc.)
Parents’ knowledge of school’s expectations for students
Student’s receipt of special help
Improvement of condition (health, emotional)
Parental support for better attendance at school
Student attendance
IPDET © 2009

30. Impact nutrition, health, and clothing
Women have limited access to economic opportunities due to lack of access to credit and other productive resources and to social control
Project offers credit, technical assistance and group information
Women create business
Generate profits
Profits re-invested
Short-term improvement in household welfare
Permanent improvement in household welfare
Improved housing
Improved education for girls
Economic improvements
Business sustained
Government- funded related programs
Macro economic environment
Worker remittances
Other bilateral micro-financed MDM programs
Foundation programs
NGO’s programs
IPDET © 2009

31. Model with Assumptions & Influences
Inputs Activities Outputs Outcomes Impact
Influence
Assumptions
IPDET © 2009

32. Theories of Change should Address these Questions:
Is the intervention built on a knowledge base?
Are there gaps in the logic chain?
Are interrelationships clear and plausible?
Is it realistic to assume that the intervention will result in the attainment of stated goals?
On what critical assumptions is the intervention based?
What else is operating in the context of the intervention that might affect results?
IPDET © 2009

33. A Final Note…. Questions?
“One does not discover new lands without consenting to lose site of the shore for a long time.” -- Unknown
Questions?
IPDET © 2009