1. Identifying Issues & Formulating Questions
46-566: Program Evaluation

2. What could go wrong? Most Important Industry Measure
Organization wishes to design a new bonus system
Problems: The old system
Promise: The new system
The new system
Two Components
Profit
Customer Loyalty
Most Important Industry Measure
Is what is desired what is being rewarded?

3. Important Considerations
What makes a good evaluation question?
Determining the specific questions
Setting priorities

4. A good evaluation question…
Dimensions of performance
Reasonableness test
Answerable
Criteria for making judgement
Establishing criteria
Living by the criteria
The role of program theory

5. Evaluation Hierarchy Considerations: Questions at each level
Interdependencies among levels
Need for program
Program Design
Operations
Impact
Efficiency

6. Selecting specific questions: Working with sponsor & stakeholders
Defining the range in the hierarchy
Gathering input
Snowball approach
Key Informant approach
Semi-structured interviews
Backward mapping
What would a useful evaluation look like?
What are the most important questions to answer?
Sample results

7. Program Theory & Assumptions
Divining theory
Archival documents
Meeting notes
Presentations made to key stakeholders/ sponsors, etc.
Using the theory against them
Pithiness
Setting priorities

8. Our Reading
What examples can you think of?
How pervasive?