Evaluation Research Appropriate for any study of planned or actual social intervention. Goal is to determine whether a social intervention has produced the intended result. Results are not always well received. Probably the most difficult type of research to do well

Steps in Evaluation Research 1. Specifying Goals/Success* Ambiguous goals – Vietnam/Iraq war Conflicting goals – Who Killed the Electric Care 2. Identifying Unintended Consequences 3. Identifying Who Benefits/Who Suffers 4. Developing Methods and Measures of Evaluation *Evaluability Assessment

Types of Measurement in Evaluation Research Outcome (dependent variable) Contextual conditions - aspects of the context of an experiment that might affect the experiment. –Examples: the economy, inflation, job growth, etc.

Types of Measurement in Evaluation Research Experimental Stimulus (interventions) –Examples: Job training program English immersion program Tax incentive program Minnesota unemployment benefit experiment Population - demographic variables as well as variables defining the population. Process variables/measures

Types of Evaluation Needs Evaluability Assessment Process Impact Analysis (summative evaluation) Efficiency Analysis –Cost-benefit Analysis –Cost-effectiveness Analysis

Evaluation Research Designs Experimental designs –Difference of Means Tests Quasi-experimental designs –Time-series design –Nonequivalent control groups –Correlational design Qualitative evaluations –Focus groups, interviews

Ethical Issues Social interventions being evaluated may raise ethical issues. –Identification of drug users, sex offenders –Deciding who gets treatment and who doesn’t

1. Who Evaluates? Elites: Government Agencies, Public Universities, Think Tanks Interest Groups: AAA, ACLU, AMA, Business groups, etc. 2. Evaluation and Bureaucratic Oversight (at least) Two reasons for evaluation: Are the policy solutions working Are the policy solutions really being implemented

STAGES OF THE POLICY PROCESS Problem Identification Policy Formation Policy Analysis Policy Choice Policy Legitimation Policy Implementation Policy Evaluation Policy Formulation, etc.

Why Results Are Ignored Too Complicated: Implications may not be presented in a way that nonresearchers can understand. Results can sometimes be counter-intuitive = Results sometimes contradict deeply held beliefs. Vested interest in a program. Who evaluates government policies? Answer: the government.

Cost-Benefit Analysis 6 important stages in cost-benefit analysis

1. Problem and Objective Identification: Goals and Values 2. Identification of Potential Solutions 3. Data Collection and Analysis

4. Estimation of Costs and Benefits –Monetization –Human v. Monetary Costs and Benefits –Intangibles –Discounted Value (money now v. money later) –Risk Assessment

5. Decision Criteria –Democratic: fair, politically acceptable –Benefits outweigh costs –Competitive –Scientific –Pareto Principle –Kaldor-Hicks

How do we estimate costs & benefits? Identify relevant impacts –Geographic targets –Persons and Preferences Future generations Monetizing Impacts –Valuing Inputs Taking into account opportunity costs

Valuing Outcomes –Willingness to pay –Benefits equal the net revenue generated plus any social surplus –Social surplus depends on efficiencies of the market –Government Day Care: direct benefit of those in program, indirect benefit of shifting supply schedule to the right. –Secondary markets need to be considered Stocking lake with game fish example Sales of bait and equipment go up, but golfing goes down

Estimating the demand for nonmarket goods –Hedonic Price Models Public safety; quality schools increase housing value, but how do we put a price on a specific public good? Statistical techniques; control variables –Opinion Surveys Ask people how much the value a good Difficult to describe goods accurately Hypotheticals not really forcing respondent to make an economic choice –Activity Surveys Survey respondents on their behavior Regional park example: examine travel, time, distance costs to estimate value.

Discounting for Time & Risk The concept of Present Value –Most of us would be unwilling to loan someone $1000 today for a promise of repayment for $1000 next month. Why? –Formula: (B t -C t )/(1+d) t –How do we calculate d? In an efficient market the market interest rate is an appropriate estimate –Example, a city want to know if buying larger trash trucks would save money. They estimate cost of larger fleet to be $500,000, but would save $100,000 a year in fuel, wages, benefits, maintenance. –Inflation is 4% and current interest rate for city is.10 –Will sell trucks in 5 th year because of a new plant

City Trash Truck Example

Expected Value adds another level of complexity. –The value of improving the levies in New Orleans depends on the probability of a hurricane and the expected damage –Need a Risk Assessment: Past occurrences, past damage, estimate probabilities –Example: 2 major hurricanes per 100 year. Probability of a major flood every 20 years =.33 ( ). Cost of flood= $25 million. What if no flood happens. –(.33)($25 million) + (.67)(-$5million) = $4.9 million

Applying the cost-benefit analysis: Is it appropriate to talk in terms of cost/benefit when dealing with all policies? What about child abuse policy? How do we handle the meaning or definition of child abuse and weigh in the rights of parents and maintain due process? Cost-Effectiveness analysis is more appropriate in this case

Some problems are not easily applied to economic models, market incentives, or cost/benefit analysis. Very few firms take on the problems of housing, education, and medically caring for the old, abating drug abuse, or dealing with crime.

Fairness is often a major goal/value when we are trying to solve a social problem and fairness can conflict with goals of efficiency or cost/benefit analysis. Protection of due process creates a social surplus which is hard to monetize.

Although “Cost/Benefit Analysis” sounds objective it is often applied very subjectively and strategically The process is likely to be affected by: –The values and perceptions of the researcher or funding agency –Local stadiums: how to value product, social capital; cannibalization of tourism, local consumption; spill-over effects