PAI786: Urban Policy Class 2: Evaluating Social Programs

Urban Policy: Evaluating Social Programs Class Outline ▫Positive vs. normative analysis ▫The role of program evaluation ▫Basic principles of program evaluation ▫Program evaluation and decision making

Urban Policy: Evaluating Social Programs Positive vs. Normative Analysis ▫Positive analysis: How do people behave?  How are prices and quantities determined in a particular market?  What is the impact of a particular government program and people’s behavior?  If the necessary data are available, positive statements can be tested.

Urban Policy: Evaluating Social Programs Positive vs. Normative Analysis ▫Normative analysis: What is a good outcome?  What are the appropriate objectives for government intervention in a given market?  Which objectives are the most important?  Normative statements cannot be tested, but they certainly can be debated!

Urban Policy: Evaluating Social Programs The Role of Program Evaluation ▫Government programs change behavior. ▫You cannot determine whether the outcome of a government program meets your own objectives without determining how it changes behavior. ▫Program evaluation is necessary to identify the programs that best meet your own objectives!

Urban Policy: Evaluating Social Programs What is the basic problem facing someone wanting to evaluate any public program? ▫What you want is to know how one place differs with and without the program. ▫What you observe is either (a) what the world is like after and before the program or (b) what one place is like with the program and another is without it. ▫Thus, you cannot be sure that the effects you observe are not due to non-program differences over time or across places.

Urban Policy: Evaluating Social Programs What is the basic problem facing someone wanting to evaluate any public program? ▫Another way to put this is that the great challenge of any program evaluation is to identify the “counterfactual”, that is, to identify what would have happened if the program had not been implemented. ▫The counterfactual cannot be observed directly, so all evaluation methods are attempts to estimate the counterfactual in an unbiased way.

Urban Policy: Evaluating Social Programs Example (from Hollister) ▫Consider a program that provides training and counseling to improve participants’ employability. ▫Suppose a high share of previously unemployed participants become employed after leaving the program. ▫Does this evidence indicate that the program is effective?

Urban Policy: Evaluating Social Programs Example (continued) ▫Answer: No! ▫People tend to move from unemployment to employment over time, and programs tend to select people who are unemployed.  So the increase in employment may reflect the natural process of moving to employment, not program impact.  This is called regression to the mean. ▫Local labor market conditions might have improved at the time the program was implemented.  So the increase in employment might reflect factors other than the impact of the program.  This is called omitted variable bias.

Urban Policy: Evaluating Social Programs The two ways to estimate program impacts are ▫random assignment ▫statistical control. ▫Random assignment ensures that differences across time and place are not correlated with program participation. ▫Statistical controls can account for observable differences across place or time and for certain kinds of unobservable factors.

Urban Policy: Evaluating Social Programs Random assignment is the preferred method in most cases. ▫It provides results that are intuitively compelling and scientifically sound. ▫If you want to know a program’s impacts, become an advocate for evaluation using random assignment!

Urban Policy: Evaluating Social Programs Random assignment can be applied at many different scales. ▫Some evaluations randomly assign treatment to individuals. ▫Others randomly assign treatment to organizations (firms, schools, etc.) ▫Still others randomly assign treatment to communities.

Urban Policy: Evaluating Social Programs Random assignment has been used to study: ▫Welfare-to-work programs ▫Unemployment insurance ▫Job training ▫Income maintenance ▫Housing assistance ▫Electricity pricing ▫Education ▫Early childhood development ▫Criminal justice policy ▫Child health and nutrition

Urban Policy: Evaluating Social Programs Random assignment is not always feasible. A huge literature indicates that the best statistical studies: ▫must have extensive data to ensure that differences aren’t due to unobservable factors. ▫must have comparable experimental and control groups based on observable factors.

Urban Policy: Evaluating Social Programs Comparable Control Groups Recent advances in evaluation methodology have discovered that it is very important to have comparable control groups, because the impact of a program may depend on the traits of the recipients. This leads to matching techniques, which focus treatment-control comparisons on groups that are comparable on all observable traits. Matching cannot solve the often-encountered problem that treatment and control groups may differ on unobserved traits—a problem that arises in many statistical studies.

Urban Policy: Evaluating Social Programs For the case of community economic development programs, Hollister discusses several evaluations that do not use random assignment. ▫This discussion gives you a sense of what to look for in statistical studies. ▫You may want to return to it when we discuss community economic development!

Urban Policy: Evaluating Social Programs Formal evaluation of programs or management reforms are often not available. Thus, it is appropriate for you (when you become public officials!) to use your own judgment: ▫to select programs and reforms that appear to have worked in other places ▫to design new programs and reforms

Urban Policy: Evaluating Social Programs Evaluations of intermediate results can also be helpful. Here is the figure in Hollister: Your Planned Work Your Intended Results ImpactResources/ Inputs ActivitiesOutputsOutcomes 12435

Urban Policy: Evaluating Social Programs But evaluation should always be in the back of your mind. ▫Search for evaluations of the programs or reforms you are interested in. ▫Make an honest judgment about the quality of existing evaluations. ▫Informally apply basic evaluation principles to programs and reforms you are considering. ▫Implement formal evaluations whenever possible!

Urban Policy: Evaluating Social Programs Informal Evaluations ▫Informal evaluations can be very helpful. ▫Learn about the market in which the program will operate—that is about the economic and social factors that influence the behavior of market participants. ▫Think about how various government programs change the incentives of people in this market. ▫Use your understanding from other cases to make an educated guess about the impact of these changes in incentives on behavior—and hence on your objectives.

Urban Policy: Evaluating Social Programs The Punchline ▫You may undermine your own objectives if you don’t take program evaluation seriously ▫Look for (and advocate!) high-quality program evaluation studies. ▫When these studies are not available, make your best judgment about the relevant positive analysis using the best evidence and analysis you can find.