Fundamental Issues for Evaluating Program Implementation: A Decision Framework Robert Owens, MA LPRC Learning & Performance Research Center.

Slides:

Advertisements

Similar presentations

Introduction to the unit and mixed methods approaches to research Kerry Hood.

Advertisements

Primary Project A Proposal for Schools-R-Us Heather Loud & Ashley Wagner.

Understanding and changing professional practice: the use of behaviour change technique methodology Susan Michie and Robert West Professors of Health Psychology,

USING PRACTICE-BASED EVIDENCE TO ASSESS AND IMPROVE INTEGRATED CARE: THE INTEGRATED CARE EVALUATION PROJECT Jim Fauth & George Tremblay Clinical Psychology.

Planning an improved prevention response up to early childhood Ms. Giovanna Campello UNODC Prevention, Treatment and Rehabilitation Section.

Mixed methods in health services research: Pitfalls and pragmatism Robyn McDermott Mixed Methods Seminar JCU October 2014.

Program Evaluation. Lecture Overview  Program evaluation and program development  Logic of program evaluation (Program theory)  Four-Step Model  Comprehensive.

Program Evaluation and Measurement Janet Myers. Objectives for today… To define and explain concepts and terms used in program evaluation. To understand.

An overview of NREPP and how to use it National Registry of Evidence-Based Programs and Practices.

Prepared by the Justice Research and Statistics Association IMPLEMENTING EVIDENCE-BASED PRACTICES.

Lessons Learned About Fidelity & Adaptation from the Massachusetts SIG Wayne Harding, Ed.M., Ph.D. Social Science Research & Evaluation, Inc. 21-C Cambridge.

Principles of Effective Prevention Programs Karol L. Kumpfer, Ph.D. Professor Dept. of Health Promotion and Education University of Utah.

UNDERSTANDING DATA AND INFORMATION FLOW Session 4.

Presented by Juli R. Skinner, MSW.  Bachelor’s degree in Psychology (1999)  Master’s of Social Work with a concentration in Administrative Community.

The Importance of Fidelity as a Means for Enhancing and Assessing Program Quality A Case Study of the RESPECT Program.

CAPP Evaluation: Implementing Evidence Based Programs in NYS Jane Powers ACT for Youth Center of Excellence 2011 A presentation for Comprehensive Adolescent.

School Social Work National Model An Overview. Rationale Purpose – Describes skills and services – Promotes consistency – Provides a framework for the.

Using a Board game to enhance mentor engagement within nurse education in practice settings Jo Hirdle and Belinda Humphries University Practice Learning.

In Shape From: National Registry of Evidence- based Programs and Practices (NREPP) Trey Thomas 11/19/2012 Health 313_01 Drugs and Human Behavior.

A Conceptual Framework for Co- Occurring Disorders within a Behavioral Health Care System Reference: National Dialogue on Co-occurring Mental Health and.

School Social Work National Model

1 From Evidence to Action Replicating and Adapting Evidence-Driven Interventions at the Local Level Shannon Thomas Ryan David Lewis-Peart.

Presentation to GSSD Oct 20/2012 Lois Okrainec Manager Mental Health and Addiction Services Child and Youth Sunrise Health Region.

Using Implementation Research to Inform Technical Assistance Practice Sam Morgan Peggy Malloy

Participants Adoption Study 109 (83%) of 133 WSU Cooperative Extension county chairs, faculty, and program staff responded to survey Dissemination & Implementation.

SHAHRP School Health and Alcohol Harm Reduction Project Maura Kearney and Kate Watson September 2012 Glasgow Psychological Service & Inverclyde Educational.

Improving Implementation Research Methods for Behavioral and Social Science Working Meeting Measuring Enactment of Innovations and the Factors that Affect.

Implementation Science Vision 21: Linking Systems of Care June 2015 Lyman Legters.

KENTUCKY YOUTH FIRST Grant Period August July

Strategic Prevention Framework Overview Paula Feathers, MA.

Digging Deeper LKSD Principal Inservice Feb. Our approach for our time together…

Understanding TASC Marc Harrington, LPC, LCASI Case Developer Region 4 TASC Robin Cuellar, CCJP, CSAC Buncombe County.

June 11, IOM, Reducing Suicide, 2002 Statement of Task w Assess the science base w Evaluate the status of prevention w Consider strategies for studying.

A cluster randomised controlled trial to assess the effectiveness of knowledge translation strategies for obesity prevention Elizabeth Waters, Boyd Swinburn,

Dissemination and Implementation Ellen Goldstein, MA, Kevin Grumbach, MD Translating Practice into Evidence: Community Engaged Research.

Implementation and process evaluation: developing our approach Ann Lendrum University of Manchester Neil Humphrey University of Manchester Gemma Moss Institute.

Applying Science to Transform Lives TREATMENT RESEARCH INSTITUTE TRI science addiction Mady Chalk, Ph.D Treatment Research Institute CADPAAC Conference.

Multidimensional Family Therapy for Adolescent Drug Abuse: Research Update and Illustration of Core Clinical Methods Presented at the 2005 Joint Meeting.

IMPLEMENTATION QUALITY RESEARCH OF PREVENTION PROGRAMS IN CROATIA MIRANDA NOVAK University of Zagreb, Faculty of Education and Rehabilitation Sciences.

CJ411-Seminar 8 What are the major treatment strategies currently used for drug and alcohol abuse?

Chapter 1 Introduction to Clinical Sport Psychology By San-Fu Kao.

Substance Abuse Prevention Fulfilling the Promise Linda Dusenbury, Ph.D. Tanglewood Research.

Care Packages in Substance Misuse Treatment Development of MH Care Clusters: overview  Service users in MH, clinicians found: idiosyncratic referral pathways.

College of Education Helping Schools Evaluate Needs and Select Best Practices in Childhood Mental Health.

In Highland Bill Alexander Director of Care and Learning June 2014.

: The National Center at EDC

Practical Challenges Integrating EBP into Addiction Treatment Programs Dan Kivlahan, Ph.D. VA Puget Sound & University of Washington APA, San Francisco.

Researchers Without Borders Webinar 4 A Framework and Suite of Instruments for Examining Fidelity of Implementation Jeanne Century Center for Elementary.

World Bank’s Approach on Youth March 31, 2005 Viviana Mangiaterra, Children and Youth Advisor, HDNCY.

High Quality Implementation: Meeting the Challenge Linda Dusenbury, Ph.D. Tanglewood Research.

SRBI/PBIS Implementation: Considerations George Sugai & Jen Freeman Center for Behavioral Education & Research Center on Positive Behavioral Interventions.

A Multi-Level Framework to Understand Factors Influencing Program Implementation in Schools Celene E. Domitrovich, Ph.D. Penn State Prevention Research.

Session 4 Agenda 1. Strategic Prevention Framework Sustainability Step 4: Implementation Step 5: Evaluation 2. Bringing It All Together 2.

INTRODUCING THE PSBA-GTO ACT FOR YOUTH CENTER OF EXCELLENCE IN CONSULTATION WITH HEALTHY TEEN NETWORK Planning for Evidence-Based Programming.

1 Center Mission Statements SAMHSA ? CSAT Improving the Health of the Nation by Bringing Effective Alcohol and Drug Treatment to Every Community CMHS Caring.

Why It’s Important To Know We’re Doing The Right Thing… Presented by Dr Roberta Ryan and Dr Ania Wilczynski August 2011 Assessments Of ‘Appropriateness’

The Promise of Prevention Linda Dusenbury, Ph.D. Tanglewood Research.

Keeping our Commitments to Collaborative Children’s Services.

Introduction 2. Pharmaceutical care is a professional patient care practice, which, when provided as an organized service, is experienced, documented,

Sandra F. Naoom, MSPH National Implementation Research Network Frank Porter Graham Child Development Institute University of North Carolina- Chapel Hill.

Mental Health in Complex Emergencies and Refugee Settings: The case of Syria Tahilia J. Rebello, PhD Columbia Global Mental Health Program WHO Collaborating.

Training Personnel Using Autism online ebp Modules

Evidence Based Practice Process

Melanie Bania Benjamin Roebuck Beth O’Halloran Vanessa Chase 2017

Models, Theories and Frameworks

The Strengthening Families Program

Fidelity of Implementation in Scaling-up Highly Rated Science Curriculum Units for Diverse Populations Carol O’Donnell and Sharon Lynch The George Washington.

Chapter 15 Community As Client: Applying the Nursing Process

Program Planning: Models and Theories

Presentation transcript:

Fundamental Issues for Evaluating Program Implementation: A Decision Framework Robert Owens, MA LPRC Learning & Performance Research Center

Purpose and Overview Purpose: Present fundamental issues in a decision framework Importance of Implementation Assessment Fundamental Issues: Decisions Theoretical approach What to measure How to measure Grain size Level of involvement Complete Framework Contributions & Limitations

Why Assess Implementation? ProgramOutcomes  Implementation  Feedback

Fundamental Issues: Decisions, Decisions, Decisions Theoretical Approach – Strict fidelity approach (McGrew et al., 1994) – Adaptation approach (Castro et al., 2004) – A little of both?? (Blakely et al., 1997; Dusenbury et al., 2005)

Fundamental Issues: Decisions, Decisions, Decisions What to measure – Delivery or Receipt Orwin’s (2002) distinction – Both are necessary for successful outcomes

Fundamental Issues: Decisions, Decisions, Decisions How to measure? (Dane & Schneider, 1998; Dusenbury et al., 2003) – Quality or how well Quality-competence or proficiency of delivery Participant responsiveness-participant engagement – Quantity or how much Adherence-the degree to which the program is delivered as intended Dosage-the amount of program activities participants receive

Fundamental Issues: Decisions, Decisions, Decisions How to measure? Continued (Chen, 1990) – Structure Facilitator dimension, Mode of delivery Participant dimension

Fundamental Issues: Decisions, Decisions, Decisions Grain size (Similar to Program Differentiation from Dane & Schneider, 1998; Dusenbury et al., 2003) – Global implementation Cantu et al. (2010) – Specific components McGrew et al. (1994) – How do we divide a program into components?

Fundamental Issues: Decisions, Decisions, Decisions Level of involvement – Systematic manipulation Kumpfer et al. (2002) – Naturalistic observation (Dusenbury et al., 2005; Blakely, 1997; McGrew, 1997; and Cantu & Hill, 2010;)

Complete Framework Context of Theoretical Orientation to Fidelity and Adaptation How to measureInvolvment What to measureQuantityQualityStructure Systematic Manipulation DeliveryAdherenceQuality Facilitator Dimension & Delivery Mode ReceiptDosage Participant Responsiveness Participant Dimension Grain Size Global Program Differentiation Specific Components Naturalistic Observation

Contributions & Limitations Contributions – Clarity on fundamental issues – Guide for practitioners – Especially useful for novices Limitations – Does not address larger context of implementation – Provides little guidance on actual methods

References Blakely, C.H., Mayer, J. P., Gottschalk, R.G., Schmitt, N., Davidson, W., Roitman, D.B. and Emshoff, J.G. (1987) The fidelity–adaptation debate: implications for the implementation of public sector social programs. American Journal of Community Psychology, 15. Cantu, A., Hill, L., & Becker, L. (2010). Implementation quality of a family-focused preventive intervention in a community-based dissemination. Journal of Children’s Services, 5(4), Castro, F. G., Barrera, M., & Martinez, C. R. (2004). The cultural adaptation of prevention interventions: Resolving tensions between fidelity and fit. Prevention Science, 5(1), Chen, H.T. (1990). Theory-Driven Evaluations, Sage, London. Dane, A. & Schneider, B. (1998). Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical Psychology Review, 18. Dusenbury, L., Brannigan, R., Falco, M. & Hansen, W. (2003). A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Education Research, 18. Dusenbury, L., Brannigan, R., Hansen, W.,Walsh, J., & Falco, M. (2005). Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Journal of Alcohol and Drug Education, 20. Kumpfer, K., Alvarado, R., Smith, P., Bellamy, N. (2002). Cultural sensitivity and adaption in family-based prevention interventions. Prevention Science 3. McGrew, J. H., Bond, G. R., Dietzen, L. and Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62. Orwin, R.G. (2000) Assessing program fidelity in substance abuse health services research. Addiction, 95.