1. Meta-analysis and benefit-cost analysis
Or, “Show me the money!”

2. Meta-analysis
The value of an evaluation is always limited by its methodological constraints
Findings will always be subject to uncertainty
Was sample representative?
Was control/comparison group truly equivalent?
Were measurements accurate?
Are the finding generalizable?
Meta-analysis helps to address these issues by aggregating findings of other evaluations

3. Meta-analysis
Is critical to identifying “what works” – our collective knowledge of whether specific interventions are successful in achieving desired outcomes
Statistically combines outcome measurements across rigorous evaluations to generate more valid and reliable predictions of program impacts

4. Meta-analysis General approach
Identify prior impact evaluations of specified program
Screen evaluations to identify subset that used rigorous and valid measures
Statistically aggregate the impact measurements of selected evaluations to produce ‘effect size’ estimates and standard error

5. It’s a wonderful world This is highly complex and time-consuming work
Fortunately, several groups are already doing it and publishing results
“What works” clearinghouses
Cochrane and Campbell academic collaboratives

6. What Are Clearinghouses?
Purpose is to identify “what works”
Review and summarize rigorous evaluations of interventions
Assign ratings based on evidence (e.g., model, promising, mixed effects)
Use slightly different methodologies, criteria and terminology
Policy area specific
What Works Clearinghouse = Education
CrimeSolutions.gov = Criminal Justice
Many are part of the federal government. We also included some non-profits.
Only high-quality since these are the only ones you can trust

7. Clearinghouses
Results are available on-line and are significant resource to evaluators
Gives you ready access to prior evaluations to borrow from their design and data collection instruments
Can readily compare your results to those from other rigorous evaluations
Can supplement your formative evaluations with published meta-analysis results to better inform stakeholders about program

8. It gets even better
Results First Clearinghouse Database contains information from 8 clearinghouses
Organized by policy area, over 1,400 interventions rated
Provides links to program page
Reconciles clearinghouse rating systems with traffic light colors
First time all of the info has been put in one place!
28 program areas

9. Results First Clearinghouse Database
Why? Different ratings
Have changed the nomenclature
Highest- generally requires RCT and statistically significant
Second- more quasi- experimental and generally stat sig
Lot going on here.
Important information is the top row and far right column.
You can see the 8 different clearninghouses we have included in our database on the left most column.
The next 5 columns include information on the ratings each clearinghouses use (note that some, such as Blueprints, only include programs shown to be model or promising excluding no evidence, mixed, or negative effects). To create a common language, we have color coded the findings from the clearninghouse illustrated in the top row.

10. Reconciles clearinghouse ratings
RATING COLOR
BROAD TIER OF EVIDENCE RATING
DEFINITION
Highest rated
Research with the highest level of rigor shows a statistically significant positive impact (Strong Evidence)
Second-highest rated
Research with a high level of rigor shows a positive impact (Good Evidence)
No effects
No evidence of impact
Mixed effects
Evidence differs on effectiveness: at least one study shows outcome had a positive effect while another shows the same outcome had a negative effect
Negative effects
Evidence of a negative impact
Why? Different ratings
Have changed the nomenclature
Highest- generally requires RCT and statistically significant
Second- more quasi- experimental and generally stat sig

11. Results First Clearinghouse Database

12. Let’s try it

13. Cost-effectiveness and benefit-cost analysis
Extensions of evaluation that compare the cost of programs to the benefits that they achieve
Enables evaluators to address 2 critical questions:
“Is a program worth funding - Does it generate enough benefits to outweigh its costs?
Which alternative would generate the most benefits per dollar invested?
Both types of studies are growing in popularity

14. Why important?
An evaluation finds that a students in new smoking prevention program are 10% less likely to smoke than those in comparison group
Control group: 30% smoked
Treatment group: 27% smoked
Program costs $10,000 per person
Is it worth funding? What is the value of the 10% reduction in smoking and is more than the program cost?

15. Cost-effectiveness analysis
Compares costs of programs to the units of effectiveness they achieve (outputs/outcomes)
Dollars per lives saved (seat belts)
Dollars per polio cases prevented (vaccines)

16. Benefit-cost analysis
Goes further and puts a dollar value on the outcomes that a program achieves and compares it to program costs
Reported as return on investment ratio
Value of outcomes achieved for every dollar spent on program
Example: 3:1 ROI for drug courts
$3 in benefits achieved for each $1 invested n program

17. General approach
Decide scope of analysis - what costs and benefits are important?
Identify and measure costs and benefits
Project costs and benefit over time
Monetize costs and benefits
Discount $ to present value
Compute CBA/CEA ratios
Conduct sensitivity analyses

18. Cost-effectiveness or Benefit-cost?
Cost-effectiveness best when it is difficult to put a dollar value on program outcomes and when a single outcome is of primary importance
Benefit-cost is best when you can monetize outcomes and you care about multiple elements of program impacts

19. Decide scope of analysis
You can’t measure everything
Decide who & what has “standing” and is important enough to measure
Example: juvenile justice
Include:
Costs: Direct program
Benefits: Reduced recidivism (avoided state and social costs); Improved high school graduation rate for participants (income)
Exclude: Indirect costs of crime on economic activity, intergenerational impacts

20. Measure costs & impacts
HOW: Your friend the impact evaluation
Or better yet, findings of LOTS of impact evaluations (meta-analysis)
Must decide how far to track costs
Generally focus on major cost drivers
Measure total, average and marginal costs (cost to serve a single client)
Measure outcomes that have ‘standing’ , both tangible and intangible ones as possible

21. Project costs over time
Many costs and benefits occur over period of time
Increased income of high school graduates
Costs of treating diabetes over lifetime
Develop projections for when costs and benefits occur (from long-term impact evaluations)
Importance: Critical to deciding what programs will achieve positive ROI in specified time periods

22. Monetize costs and benefits
Must determine what costs to include
Must determine what outcomes are worth in monetary terms

23. How can you assess costs?
Fortunately, there are reliable estimates for many things that are difficult to directly measure
Cost of crime victimization
Value of college degree
Can estimate other values through willingness-to-pay studies
Value of clean parks; wilderness protection
Many values have been estimated by economists (your friend the google)

24. Discount to present value
Costs and benefits that occur in the future have a lower economic value than those that occur immediately
That’s why we charge interest on loans
Must discount value of future costs and benefits to ‘present value’
By discounting them by discount rate (money value of time)

25. Compute CER/ROI Cost effectiveness ratio: Cost-benefit ratio:
Present value of costs/units of effectiveness
Expressed as “dollars per dropout prevented”
Cost-benefit ratio:
Net present value of benefits/net present value of costs
“Taxpayers and society receive $15,481 for each person who completes dropout prevention program
Return on Investment ratio: $5.87 in benefits for each $1 invested OR

26. Sensitivity analysis How risky is the investment in the program?
All projections are based on projections that are subject to uncertainty (error)
Sensitivity analysis determines how much the results would change if assumptions were varied
Typically done with Monte-Carlo analysis
Runs analysis many times under varying assumptions

27. Things to keep in mind
BCA and CEA estimate the economic value of investments in programs
BUT, economic efficiency isn’t the only thing we care about
Equity, civil rights, due process are also important but not measured by BCA/CEA
Should recognize this caveat when reporting results

28. Fortunately, we have the technology…
Sophisticated benefit-cost models have been developed programs to examine programs in many social policy areas
Based on meta-analysis program impact estimates
Are readily customizable to specific jurisdictions by specifying local program costs and population cohorts
Have monetization & sensitivity analysis built-in

29. Example: Meta-analysis Functional Family Therapy
RECIDIVISM RATES REDUCED BY 16%
Here’s a more concrete example of how the model works using data from Washington state.
Considering their juvenile populations, they determined that without programming, WA juveniles recidivate at a rate of 60% within the first 3 years and 72% 15 years later. FFT is a program that integrates the family into the rehabilitative process and is found, through meta-analysis, to reduce recidivism in juveniles by about 16%.
The model then calculates the benefits that reduction achieves – or monetizes this shaded blue area. Because when you invest in a corrections program, you’re buying recidivism reduction and skill building in the participant – and when you buy something, you want to know how much it’s worth.
Source: Based on Washington data

30. CBA of Functional Family Therapy
OUTCOMES FROM PARTICIPATION
MAIN SOURCE OF BENEFITS
Reduced crime
$20,740
Lower state & victim costs
Increased high school graduation
$8,220
Increased earnings
Reduced health care costs
$66
Lower public costs
Total Benefits
$29,026
Cost
$3,406
Net Present Value
$25,620
Benefits per Dollar of Cost
$8.52
Source: Based on Washington data

31. The Killer Ap
BCA and CBA become incredibly useful if you can provide comparative assessments of alternative ways to achieve a goal
Such as ranking criminal justice program alternatives by their return on investment (think Consumer Reports ranking)

32. Comparison of CJ Programs
COSTS
BENEFITS
BENEFIT TO COST RATIO
Correctional education
$1,180
$21,720
$18.40
Vocational education
$1,645
$19,594
$11.91
Correctional industries
$1,485
$6,818
$4.59
Drug courts
$4,951
$15,361
$3.10
Intensive supervision (surveillance only)
$4,305
-$1,139
-$0.26
JUVENILE JUSTICE PROGRAMS
Aggression Replacement Training
(within institutions)
$1,575
$16,827
$10.68
Functional Family Therapy (probation)
$3,406
$29,026
$8.52
$3,275
$8,110
$2.48
Multidimensional Treatment Foster Care
$8,232
$20,065
$2.44
Scared Straight
$67
-$12,319
-$183.87
Then compare those to one another by a common denominator. The cost-benefit ratio identifies the return on investment over the same time period. This is moving beyond just having a list of programs and what they cost – where an inexpensive program (like Scared Straight) might seem like a safe bet and a more expensive program (like formalized Aggression Replacement Training) might scare us away – to taking a closer look at what those investments actually BUY (those long-term benefits), and identifying which investments are wiser bets over the long term.
Just a caveat to using a table like this. These are data from Washington State – the cost-benefit ratios are based on their costs and benefits. The LFC has published a similar table of programs using NM-state data, which Charles has presented before. The full program inventory would help you determine where program dollars are going; the NM-specific consumer reports guide to programs identifies how effective those program can be and which are wiser investments here in NM.
Source: Based on Washington data

33. Example: Results First
Benefit-cost model that can assess a wide range of programs across criminal justice, child welfare, K-12 education, prevention, and health care
Based on meta-analyses of rigorous evaluations
Based on work of Washington State Institute for Public Policy
Now used by 22 states and 7 local governments

34. Results First Model Output
Source: Based on Washington data

35. Results First Model Output: Cash Flow Analysis
Source: Based on Washington data

36. Results First Model Output: Benefits by Perspective
Source: Based on Washington data

37. Results First Model Output: Taxpayer Benefits by Budget Area
Source: Based on Washington data

38. Results First Model Output: Taxpayer Benefits by Governmental Level
Source: Based on Washington data