Identifying Issues & Formulating Questions 46-566: Program Evaluation
What could go wrong? Most Important Industry Measure Organization wishes to design a new bonus system Problems: The old system Promise: The new system The new system Two Components Profit Customer Loyalty Most Important Industry Measure Is what is desired what is being rewarded?
Important Considerations What makes a good evaluation question? Determining the specific questions Setting priorities
A good evaluation question… Dimensions of performance Reasonableness test Answerable Criteria for making judgement Establishing criteria Living by the criteria The role of program theory
Evaluation Hierarchy Considerations: Questions at each level Interdependencies among levels Need for program Program Design Operations Impact Efficiency
Selecting specific questions: Working with sponsor & stakeholders Defining the range in the hierarchy Gathering input Snowball approach Key Informant approach Semi-structured interviews Backward mapping What would a useful evaluation look like? What are the most important questions to answer? Sample results
Program Theory & Assumptions Divining theory Archival documents Meeting notes Presentations made to key stakeholders/ sponsors, etc. Using the theory against them Pithiness Setting priorities
Our Reading What examples can you think of? How pervasive?