Evaluation of the Fighting Back Initiative by Kay E. Sherwood Presented by Maddie Velez Presented by Maddie Velez.

Slides:



Advertisements
Similar presentations
TCE Board Presentation February, 2006 Evaluating the Initiative Oakland, CA - Seattle, WA.
Advertisements

Oakland EMA Patricia LaBrie Calloway, R.N., P.H.N.
Key Challenges in the Field of Violence Against Women with Disabilities and Deaf Women Overview Overarching Challenges Barriers to Services Barriers to.
Reclaiming Futures as part of the OJJDP required Three-Year State Plan A Presentation to State SAGs Tom Begich.
Complexities of Co-occurring Disorders STATE AGENCY PERSPECTIVE June 24, 2004 Renata J. Henry, M.Ed.
Understanding the pathway: barriers to data collection and onward referral to specialist hepatitis C services for PWUD in London.
Bureau of Justice Assistance JUSTICE AND MENTAL HEALTH COLLABORATIONS Bureau of Justice Assistance JUSTICE AND MENTAL HEALTH COLLABORATIONS Presentation.
Title I Schoolwide Providing the Tools for Change Presented by Education Service Center Region XI February 2008.
Laura Nissen National Program Director February 2009.
Systems Approach Workbook A Systems Approach to Substance Use Services and Supports in Canada Communication Tools: Sample PowerPoint presentation The original.
 The slides in this set are made available for use in presentations and educational sessions by health departments.  The information is provided for.
Socially constructed exclusion: Understanding and mitigating the pervasive influences of HIV-related stigma John de Wit, Professor and Director Centre.
Caring for our future Caring for our future: shared ambitions for care and support Emerging thinking: Markets November 2011.
Sustaining Community Based Programs CYFAR Conference Boston, 2005.
ARQ part II data management Training pack 2: Monitoring drug abuse for policy and practice.
Federal Budget Measure Drug and Alcohol and Mental Health Counselling Services within Universities.
John R. Kasich, Governor Tracy Plouck, Director Orman Hall, Director.
Healthy North Carolina 2020 and EBS/EBI 101 Joanne Rinker MS, RD, CDE, LDN Center for Healthy North Carolina Director of Training and Technical Assistance.
Commonwealth of Massachusetts Executive Office of Health and Human Services Improving the Commonwealth’s Services for Children and Families A Framework.
FUNDING OPPORTUNITIES. BACKGROUND New category of funding in the FY13 Harold Rogers Prescription Drug Monitoring Program Official title is “Category 3:
PHAB Slide Set 2013 The slides in this set are made available for use in presentations and educational sessions by health departments. The information.
Shared Decision Making: Moving Forward Together
Canada’s Drug Strategy. 2 Purpose Provide an overview of Canada’s renewed National Drug Strategy  Historical context  Impetus for change  Renewed National.
BUILDING CAPACITY FOR UNIVERSAL PREVENTION THROUGH STATE-NONPROFIT-UNIVERSITY- SCHOOL SYSTEM PARTNERSHIPS Philip J. Leaf, Ph.D. Johns Hopkins University.
Resident Centred Care Through Service Excellence Introduction to the Resident Centred Care and Service Excellence Project.
Some Emerging Characteristics of Sustainable Practices Ronnie Detrich Randy Keyworth Jack States Wing Institute.
Page 1 DBHDS Virginia Department of Behavioral Health and Developmental Services Systemic Therapeutic Assessment Respite and Treatment (START) Bob Villa.
“Wraparound Orange”- Addressing the Children’s Mental Health System of Care December 1, 2009.
Investing in Change: Funding Collective Impact
Children’s Mental Health: An Urgent Priority for Illinois.
Energy Issues in Peru and the Andes: Environmental and Social Aspects George Washington University January 28, 2005 Dr. Robert H. Montgomery Head, Environmental.
NEW MEXICO STATE COVERAGE INITIATIVE New Mexico Human Services Department June, 2004 Carolyn Ingram, Director Medical Assistance Division.
J. Douglas Willms University of New Brunswick School Completion in Canadian Schools: Findings from Tell Them From Me.
Monitoring and Evaluation in MCH Programs and Projects MCH in Developing Countries Feb 10, 2011.
KENTUCKY YOUTH FIRST Grant Period August July
ADD Perspectives on Accountability Where are We Now and What does the Future Hold? Jennifer G. Johnson, Ed.D.
Safe and Healthy Schools: Move to a State and Local Strategy.
Welcome! Please join us via teleconference: Phone: Code:
Overview NIATx Overview. NIATx Mission To improve care delivery to help people live better lives To become the premier resource for systems and process.
Transforming Community Services Commissioning Information for Community Services Stakeholder Workshop 14 October 2009 Coleen Milligan – Project Manager.
Comprehensive Educator Effectiveness: New Guidance and Models Presentation for the Virginia Association of School Superintendents Annual Conference Patty.
TitleHow do you know if you have got it right? Evaluation and Indicators Professor Vanessa Burholt.
Federal Advisory Committee on Juvenile Justice (FACJJ) 2009 Annual Request for Information (ARI) Presented by CSR, Incorporated August 2009.
Warm-up Question 1 What is your favorite winter activity? If you could travel anywhere this coming weekend, where would you go?
Third Sector Evaluation: Challenges and Opportunities Presentation to the Public Legal Education in Canada National Conference on “Making an Impact” 26.
Virginia WebEx Program Division Support for Substantial School Improvement 1.
United Way of Mid Coast Maine Mental Health Initiative Presentation to Midcoast District Coordinating Council March 8, 2011.
Monitoring and Evaluation in MCH Programs and Projects MCH in Developing Countries Feb 24, 2009.
Health Reform: Local Safety Net Implications Karen J. Minyard, Ph.D., Executive Director, Georgia Health Policy Center, Georgia State University.
@theEIFoundation | eif.org.uk Early Intervention to prevent gang and youth violence: ‘Maturity Matrix’ Early intervention (‘EI’) is about getting extra.
Partnership for Change Drug Free Communities Baseline Evaluation October 13, 2015 Presented by: Linda M. Bosma, PhD Bosma Consulting, LLC Presented by:
Salinas, California Addressing the Roots of Violence Through a Collective Impact Approach.
Solano County Behavioral Health MHSA Innovation Plan A Joint Project Between Solano County and the UC Davis Center for Reducing Health Disparities.
Monitoring and Evaluation in MCH Programs and Projects MCH in Developing Countries Feb 9, 2012.
Barnstable County Regional Substance Abuse Council Updated October 2015 Barnstable County Department of Human Services |
Blaenau Gwent County Borough Council Social Services CSSIW Performance Evaluation Report 2014–15.
1 Center Mission Statements SAMHSA ? CSAT Improving the Health of the Nation by Bringing Effective Alcohol and Drug Treatment to Every Community CMHS Caring.
Background Objectives Methods Study Design A program evaluation of WIHD AfterCare families utilizing data collected from self-report measures and demographic.
Child Protection and Social Protection “Social Protection must be regarded as one element in a broad strategy aimed at ensuring protection of children”
PATIENT SAFETY AND QUALITY IMPROVEMENT EDUCATIONAL STRATEGY.
Partnering with Political Leaders to Evaluate Recommendations for State-Level Mental Health Policy Change: The Role of Responsive, Participatory Evaluation.
Nevada’s Epidemiological Profile Julia Peek, MHA Deputy Administrator, Community Services April 29, 2015 Division of Public and Behavioral Health (DPBH)
How can we evaluate the impact of projects designed to prevent people from sleeping rough? Lucy Spurling (DCLG research) Ellen Reaich (DCLG policy)
Digital Technology Overview
Community Corrections 2019 Budget
As we reflect on policies and practices for expanding and improving early identification and early intervention for youth, I would like to tie together.
Director’s Report Quarter Two Board summary 10 November 2017
The Communities That Care System
Community Organizing: The Path to Building Power
Presentation transcript:

Evaluation of the Fighting Back Initiative by Kay E. Sherwood Presented by Maddie Velez Presented by Maddie Velez

What is Fighting Back? A community-based drug abuse prevention program

Why study this Initiative? Shows the importance of taking context into evaluations Raises questions about how community interventions are conceptualized and evaluated Provides a warning about the manageability of large-scale, comprehensive evaluations. Shows the importance of taking context into evaluations Raises questions about how community interventions are conceptualized and evaluated Provides a warning about the manageability of large-scale, comprehensive evaluations.

Background: Duration and Scale Contribute to Complexity ◊A 12 year initiative ◊Original stakeholders differed greatly from the stakeholders involved 12 years later ◊Reduction of intervention sites went from 15 to 5 ◊First evaluation team replaced after 2 years. ◊High staff / leadership turnover ◊Original key leader retires ◊Few examples of credible, successful evaluations that truly measured the interventions impact ◊A 12 year initiative ◊Original stakeholders differed greatly from the stakeholders involved 12 years later ◊Reduction of intervention sites went from 15 to 5 ◊First evaluation team replaced after 2 years. ◊High staff / leadership turnover ◊Original key leader retires ◊Few examples of credible, successful evaluations that truly measured the interventions impact

The Foundation Takes on Substance abuse ◊Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse. ◊First grant was made to Vanderbilt University for $26.4 million in 1988 ◊Foundation explores addressing the national problems of substance abuse and dependence ◊Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse. ◊First grant was made to Vanderbilt University for $26.4 million in 1988 ◊Foundation explores addressing the national problems of substance abuse and dependence

Continued… ◊July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.” ◊The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.” ◊Project STAR and ALERT ◊Poly abuse - combination of mental health problems and substance abuse occurring ◊July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.” ◊The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.” ◊Project STAR and ALERT ◊Poly abuse - combination of mental health problems and substance abuse occurring

New Leadership: Kathryn Edmundson ◊New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement? ◊An element of racism and elitism in the law enforcement ◊Expected outcomes ◊New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement? ◊An element of racism and elitism in the law enforcement ◊Expected outcomes

Evaluation I: Lost time, Money, and Credibility ◊ ◊The first evaluation team replaced, 4 years, $4.6 million, and a baseline ◊Division between stakeholders missed changes ◊Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data. ◊ ◊The first evaluation team replaced, 4 years, $4.6 million, and a baseline ◊Division between stakeholders missed changes ◊Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data.

A 1996 Watershed ◊Become unified with an emphasis on prevention, early intervention, treatment, and aftercare. ◊NPO (National Program Office) moved to Boston University School of Public Health. ◊NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom. ◊Board of Trustees makes a recommendation to give the program Fighting Back more time. ◊Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect. ◊Become unified with an emphasis on prevention, early intervention, treatment, and aftercare. ◊NPO (National Program Office) moved to Boston University School of Public Health. ◊NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom. ◊Board of Trustees makes a recommendation to give the program Fighting Back more time. ◊Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect.

A National Program Office Change ◊Fighting Back reduces # of sites eligible for new funding. ◊Measure most substance abuse within the communities to be able to do something measurable at community level. ◊Increasing treatment and treatment capacity an important goal. ◊Fighting Back reduces # of sites eligible for new funding. ◊Measure most substance abuse within the communities to be able to do something measurable at community level. ◊Increasing treatment and treatment capacity an important goal.

Evaluation II ◊Consensus 2nd evaluation team does an credible job with difficult circumstances. ◊1st Evaluators spend $4.6 million dollars with little to show for it. ◊Fighting Back Program and evaluation staff is moving forward w/out replacement dollars. ◊Consensus 2nd evaluation team does an credible job with difficult circumstances. ◊1st Evaluators spend $4.6 million dollars with little to show for it. ◊Fighting Back Program and evaluation staff is moving forward w/out replacement dollars.

Relying on Survey Data ◊Phone surveys throughout the community. ◊Management Information Systems (MIS). ◊Ethnographic Studies. ◊Community Indicators ◊Four Research questions were identified by the 2nd evaluation team. ◊Strong correlations between strategies and outcomes. ◊Community Indicators ◊School survey data difficult to use. ◊Phone surveys throughout the community. ◊Management Information Systems (MIS). ◊Ethnographic Studies. ◊Community Indicators ◊Four Research questions were identified by the 2nd evaluation team. ◊Strong correlations between strategies and outcomes. ◊Community Indicators ◊School survey data difficult to use.

The Price of Relying on Survey Data ◊199 6 residue of distrust ◊Saxe ’ s research team became known as the “ national evaluation ◊Community has been seen as the “ human subject ” ◊National evaluation offer no alternative to outcomes perspective ◊High emotions surrounding analysis emerged accusations ◊Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions ◊199 6 residue of distrust ◊Saxe ’ s research team became known as the “ national evaluation ◊Community has been seen as the “ human subject ” ◊National evaluation offer no alternative to outcomes perspective ◊High emotions surrounding analysis emerged accusations ◊Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions

The Evaluation’s Ability to Explain ◊Evaluation illustrate all central problems for evaluation ◊Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it ◊Fighting Back site activities revised after an initial publication in 1997 ◊Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals ◊Evaluation illustrate all central problems for evaluation ◊Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it ◊Fighting Back site activities revised after an initial publication in 1997 ◊Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals

Measuring and Interpreting Outcomes ◊Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data ◊3 waves of surveys- 1995, 1997, 1999 ◊Jellinek described early thinking on the evaluation ◊Presentation of Results - A second area of disagreement ◊Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data ◊3 waves of surveys- 1995, 1997, 1999 ◊Jellinek described early thinking on the evaluation ◊Presentation of Results - A second area of disagreement

The Continuing Debate and the Foundation’s Takeaway ◊Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04 ◊Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes ◊Teams were formed ◊Substance abuse- D.A.R.E. and treatment reform ◊Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04 ◊Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes ◊Teams were formed ◊Substance abuse- D.A.R.E. and treatment reform