Evaluation of the Fighting Back Initiative by Kay E. Sherwood Presented by Maddie Velez Presented by Maddie Velez

What is Fighting Back? A community-based drug abuse prevention program

Why study this Initiative? Shows the importance of taking context into evaluations Raises questions about how community interventions are conceptualized and evaluated Provides a warning about the manageability of large-scale, comprehensive evaluations. Shows the importance of taking context into evaluations Raises questions about how community interventions are conceptualized and evaluated Provides a warning about the manageability of large-scale, comprehensive evaluations.

Background: Duration and Scale Contribute to Complexity ◊A 12 year initiative ◊Original stakeholders differed greatly from the stakeholders involved 12 years later ◊Reduction of intervention sites went from 15 to 5 ◊First evaluation team replaced after 2 years. ◊High staff / leadership turnover ◊Original key leader retires ◊Few examples of credible, successful evaluations that truly measured the interventions impact ◊A 12 year initiative ◊Original stakeholders differed greatly from the stakeholders involved 12 years later ◊Reduction of intervention sites went from 15 to 5 ◊First evaluation team replaced after 2 years. ◊High staff / leadership turnover ◊Original key leader retires ◊Few examples of credible, successful evaluations that truly measured the interventions impact

The Foundation Takes on Substance abuse ◊Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse. ◊First grant was made to Vanderbilt University for $26.4 million in 1988 ◊Foundation explores addressing the national problems of substance abuse and dependence ◊Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse. ◊First grant was made to Vanderbilt University for $26.4 million in 1988 ◊Foundation explores addressing the national problems of substance abuse and dependence

Continued… ◊July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.” ◊The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.” ◊Project STAR and ALERT ◊Poly abuse - combination of mental health problems and substance abuse occurring ◊July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.” ◊The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.” ◊Project STAR and ALERT ◊Poly abuse - combination of mental health problems and substance abuse occurring

New Leadership: Kathryn Edmundson ◊New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement? ◊An element of racism and elitism in the law enforcement ◊Expected outcomes ◊New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement? ◊An element of racism and elitism in the law enforcement ◊Expected outcomes

Evaluation I: Lost time, Money, and Credibility ◊ ◊The first evaluation team replaced, 4 years, $4.6 million, and a baseline ◊Division between stakeholders missed changes ◊Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data. ◊ ◊The first evaluation team replaced, 4 years, $4.6 million, and a baseline ◊Division between stakeholders missed changes ◊Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data.

A 1996 Watershed ◊Become unified with an emphasis on prevention, early intervention, treatment, and aftercare. ◊NPO (National Program Office) moved to Boston University School of Public Health. ◊NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom. ◊Board of Trustees makes a recommendation to give the program Fighting Back more time. ◊Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect. ◊Become unified with an emphasis on prevention, early intervention, treatment, and aftercare. ◊NPO (National Program Office) moved to Boston University School of Public Health. ◊NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom. ◊Board of Trustees makes a recommendation to give the program Fighting Back more time. ◊Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect.

A National Program Office Change ◊Fighting Back reduces # of sites eligible for new funding. ◊Measure most substance abuse within the communities to be able to do something measurable at community level. ◊Increasing treatment and treatment capacity an important goal. ◊Fighting Back reduces # of sites eligible for new funding. ◊Measure most substance abuse within the communities to be able to do something measurable at community level. ◊Increasing treatment and treatment capacity an important goal.

Evaluation II ◊Consensus 2nd evaluation team does an credible job with difficult circumstances. ◊1st Evaluators spend $4.6 million dollars with little to show for it. ◊Fighting Back Program and evaluation staff is moving forward w/out replacement dollars. ◊Consensus 2nd evaluation team does an credible job with difficult circumstances. ◊1st Evaluators spend $4.6 million dollars with little to show for it. ◊Fighting Back Program and evaluation staff is moving forward w/out replacement dollars.

Relying on Survey Data ◊Phone surveys throughout the community. ◊Management Information Systems (MIS). ◊Ethnographic Studies. ◊Community Indicators ◊Four Research questions were identified by the 2nd evaluation team. ◊Strong correlations between strategies and outcomes. ◊Community Indicators ◊School survey data difficult to use. ◊Phone surveys throughout the community. ◊Management Information Systems (MIS). ◊Ethnographic Studies. ◊Community Indicators ◊Four Research questions were identified by the 2nd evaluation team. ◊Strong correlations between strategies and outcomes. ◊Community Indicators ◊School survey data difficult to use.

The Price of Relying on Survey Data ◊199 6 residue of distrust ◊Saxe ’ s research team became known as the “ national evaluation ◊Community has been seen as the “ human subject ” ◊National evaluation offer no alternative to outcomes perspective ◊High emotions surrounding analysis emerged accusations ◊Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions ◊199 6 residue of distrust ◊Saxe ’ s research team became known as the “ national evaluation ◊Community has been seen as the “ human subject ” ◊National evaluation offer no alternative to outcomes perspective ◊High emotions surrounding analysis emerged accusations ◊Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions

The Evaluation’s Ability to Explain ◊Evaluation illustrate all central problems for evaluation ◊Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it ◊Fighting Back site activities revised after an initial publication in 1997 ◊Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals ◊Evaluation illustrate all central problems for evaluation ◊Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it ◊Fighting Back site activities revised after an initial publication in 1997 ◊Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals

Measuring and Interpreting Outcomes ◊Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data ◊3 waves of surveys- 1995, 1997, 1999 ◊Jellinek described early thinking on the evaluation ◊Presentation of Results - A second area of disagreement ◊Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data ◊3 waves of surveys- 1995, 1997, 1999 ◊Jellinek described early thinking on the evaluation ◊Presentation of Results - A second area of disagreement

The Continuing Debate and the Foundation’s Takeaway ◊Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04 ◊Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes ◊Teams were formed ◊Substance abuse- D.A.R.E. and treatment reform ◊Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04 ◊Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes ◊Teams were formed ◊Substance abuse- D.A.R.E. and treatment reform