Building a Strong Outcome Portfolio Section 2: Evidence and Evidence-Based Jeffrey A. Butts, Ph.D. Research and Evaluation Center John Jay College of Criminal Justice City University of New York September 2018

What are Evidence-Based Programs? X X Good Programs Programs That Always Work X Programs That Usually Work X Proven Programs X X Most Effective Programs Best Programs

What are Evidence-Based Programs? Replicable interventions based on sound theory with reliable effects on relevant outcomes as demonstrated by multiple evaluations using credible research designs accounting for all reasonable threats to validity, both internal and external. Good Programs Programs That Work Programs That Always Work Proven Programs Most Effective Programs Best Programs

What are Evidence-Based Programs? Before After 100% 0% 80% = 30% 50%

What are Evidence-Based Programs? What if the difference was less than 30%? X 55% Before After 100% 0% What if the difference was 3% or 4%? X 45%

What are Evidence-Based Programs? Even a difference of 3% or 4% could qualify a program as “evidence-based” IF it was a reliable difference detected by valid evaluation designs.

Two Types of Threats to Validity External: Something about the way the study was conducted makes it inappropriate to generalize the findings beyond the particular study, sample, or population. Can the findings of effectiveness be transferred to other settings, other circumstances, and other populations? Internal: The study failed to establish credible evidence that the intervention (e.g., services, policy changes) affected the outcomes in a causal way and that the association was not likely due to other factors. Can we really say that A > caused > B?

Two other important concepts: Interpreting Effects Two other important concepts: Statistical Significance — How confident can we be that differences in outcome are really there and not just due to dumb luck? Effect Size — How meaningful are the differences in outcome? Differences can be statistically significant, but trivial in terms of their application and benefit in the real world.

Statistical Confidence Comes From our Knowledge of Distributions

Percent Change in Recidivism One study outcome or one person’s behavior - 20% -10% 0% 10% 20% Percent Change in Recidivism

Percent Change in Recidivism - 20% -10% 0% 10% 20% Percent Change in Recidivism

Percent Change in Recidivism - 20% -10% 0% 10% 20% Percent Change in Recidivism

Not “evidence-based” … - 20% -10% 0% 10% 20% Percent Change in Recidivism

Maybe “evidence-based” … - 20% -10% 0% 10% 20% Percent Change in Recidivism

Cause and Effect Evaluators assess not only outcomes, but whether changing outcomes are attributable to a program or policy: Outcome Level is the status of an outcome at some point in time (e.g., drug use among teenagers) Outcome Change is the difference between outcome levels at different points in time or between groups Program Effect is the portion of a change in outcome that can be attributed uniquely to a program as opposed to the influence of other factors Rossi, P., M. Lipsey and H. Freeman (2004). Evaluation: A Systematic Approach (7th Edition), p. 208. Sage Publications.

Model Development RANDOMIZED TRIALS EVALUATION FIDELITY ASSURANCE REPLICATION INNOVATION THEORY

Many Types of Evidence Stage of Development Question to be Answered Evaluation Function 1. Assessment of social problems and needs To what extent are community and standards met? Needs assessment; problem description 2. Determination of goals What must be done to meet those needs and standards? Needs assessment; service needs 3. Design of program alternatives What services could be used to produce the desired changes? Assessment of program logic or theory 4. Selection of alternative Which of the possible program approaches is best? Feasibility study; formative evaluation 5. Program implementation How should the program be put into operation? Implementation assessment 6. Program operation Is the program operating as planned? Process evaluation; program monitoring 7. Program outcomes Is the program having the desired effects? Outcome evaluation 8. Program efficiency Are program effects attained at a reasonable cost? Cost-benefit analysis; cost-effectiveness analysis Rossi, P., M. Lipsey and H. Freeman (2004). Evaluation: A Systematic Approach (7th Edition), p. 40. Sage Publications; adapted from Pancer & Westhues 1989).

Focus of Evidence-Based Practices and Policy Many Types of Evidence Stage of Development Question to be Answered Evaluation Function 1. Assessment of social problems and needs To what extent are community and standards met? Needs assessment; problem description 2. Determination of goals What must be done to meet those needs and standards? Needs assessment; service needs 3. Design of program alternatives What services could be used to produce the desired changes? Assessment of program logic or theory 4. Selection of alternative Which of the possible program approaches is best? Feasibility study; formative evaluation 5. Program implementation How should the program be put into operation? Implementation assessment 6. Program operation Is the program operating as planned? Process evaluation; program monitoring 7. Program outcomes Is the program having the desired effects? Outcome evaluation 8. Program efficiency Are program effects attained at a reasonable cost? Cost-benefit analysis; cost-effectiveness analysis Focus of Evidence-Based Practices and Policy Rossi, P., M. Lipsey and H. Freeman (2004). Evaluation: A Systematic Approach (7th Edition), p. 40. Sage Publications; adapted from Pancer & Westhues 1989).

Section 3: Evidence Starts with Theory Next Section 3: Evidence Starts with Theory