1. Types of Evaluation

2. Different types of evaluation
Formative evaluations examine the delivery of the program or technology, the quality of its implementation, and the assessment of the organizational context, personnel, procedures, inputs, etc
Needs assessment
Process evaluation
Summative Evaluations describe what happens subsequent to delivery of the program or technology; assessing whether the object can be said to have caused the outcome; determining the overall impact of the causal factor beyond only the immediate target outcomes; and, estimating the relative costs associated with the object.
c. Impact evaluation
d. Cost-benefit analysis

3. Needs Assessment Should Provide
Clear sense of target population
Students who are not responding to other inputs
Students who are falling behind
Clear sense of need program will fill
What are teachers lacking?
How to deliver? How much? What are potential barriers?
Clear articulation of program benefits
Is a wrong being righted? Is a right being expanded?
Clear sense of alternatives
Is this the most effective, efficient, cost-effective method of meeting teacher/student needs?
Tools – focus group discussions, structured
unstructured surveys
Implementor should know what, why, for whom, it wants to do the program

4. Process evaluation Are the services being delivered?
Money is being spent
Textbooks are reaching the classroom, being used
Can same service be delivered at lower cost?
Substituting expensive inputs with cheaper ones
Are the services reaching the right population?
Are the books reaching students? Which students?
Are the clients satisfied with service?
Teachers’, students’ response to teaching method
Tools/resources – administrative data, surveys, group
discussions

5. Impact evaluation The program happened, how did it change lives?
What does ToC say we might expect to change?
Intermediate indicators
Final outcomes
Primary: did textbooks cause children to learn more
Secondary:
Distributional questions: who learned more?
If several treatments: what was the best program design? 5 5

6. How impact differs from process?
When we answer a process question, we need to describe what happened.
This can be done from reading documents, interviewing people, admin records, etc.
When we answer an impact question, we need to compare what happened to what would have happened without the program
There are various ways to get at this, but all of them have in common that they need to re-create what did not happen.

7. Impact Evaluation Techniques
Experimental Evaluation – Assignment of treatment is random
Quasi-Experimental – There are multiple waves of data or multiple groups available. But the treatment assignment is not random
Non-Experimental – Only a single snapshot measurement avaialble
Do a blackboard demo: The problem of constructing a counterfactual

8. Evaluation and cost-benefit analysis
Needs assessment gives you the metric for defining the cost/benefit ratio
Process evaluation gives you the costs of all the inputs
Impact evaluation gives you the quantified benefits
Identifying alternatives allows for comparative cost benefit

9. Example: Comparative cost benefit
In comparison to other programs, deworming has been found to be the most cost-effective

10. Linking back to objectives for evaluation
Accountability
Did we do what we said we were going to do?
Process evaluation determines whether books delivered and used
Did we have a positive impact on people’s lives?
Impact evaluation of link between books and test scores
Lesson learning
Particular programs do or do not work
Impact evaluations of similar programs in different situations
What is the most effective route to achieve a certain outcome?
Cost benefit analysis comparing several programs
Similarities in strategies that are successful, for example, in changing behavior, even across fields?
Linking results back to theory
Reduced poverty through more effective programs
Future decisions based on lessons learned
Solid reliable impact evaluations are the building blocks for more general lesson learning

11. Things to be very clear about
a. Validity – internal and external b. Study Ethics

12. a. Internal Validity How well the study was run (research design,
operational definitions used, how variables were
measured, what was/wasn't measured, etc.), and
In case of an impact evaluation how confidently can
one conclude that the change in the dependent
variable was produced solely by the independent
variable and not extraneous ones

13. a. External Validity
The extent to which a study's results (regardless of whether the study is descriptive or experimental) can be generalized/applied to other people or settings reflects its external validity.
Typically, group research employing randomization/randomized-selection will initially possess higher external validity than will studies (e.g., case studies and single-subject experimental research) that do not use random selection/assignment.

14. b. Ethical Principles Voluntary participation Informed consent
Risk of harm
Confidentiality
Institutional Review Boards – Enforces established procedures that researchers will consider all relevant ethical issues in formulating research plans.