Sandra F. Naoom, MSPH National Implementation Research Network Frank Porter Graham Child Development Institute University of North Carolina- Chapel Hill.

Slides:



Advertisements
Similar presentations
Integrated Systems to Support Implementation
Advertisements

Scaling Up: From Research to National Implementation
Dean L. Fixsen, Karen A. Blase, Michelle A. Duda, Sandra F. Naoom, Melissa Van Dyke National Implementation Research Network Implementation and System.
Karen A. Blase, PhD Dean L. Fixsen, PhD, Melissa Van Dyke, LCSW, Allison Metz, PhD, Sandra Naoom, MSPH, Michelle Duda, Ph D From Outcomes to Implementation.
Establishing an Effective Network of PB4L: School wide Coaches
The Council of Chief State School Officers State Consortium on Educator Effectiveness (SCEE) February 14, 2012 Implementing Your Educator Effectiveness.
Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. & Karen A. Blase Ph.D., Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University.
Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. & Karen A. Blase Ph.D., Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University.
STAFF COACHING The Pathway to Successful Outcomes OJACC 2012 Jennifer Kisela & Kelli Held.
Planning for Success: Using Implementation Data to Action Plan for Full and Sustained Implementation Barbara Sims Caryn Ward National SISEP Center National.
Installing Alternative Response in Ohio
April 29, 2014 WA OSPI SISEP Active State State Capacity Assessment Results.
RE-EXAMINING THE ROLE OF PROFESSIONAL DEVELOPMENT AND TRAINING EVALUATION THROUGH AN IMPLEMENTATION SCIENCE LENS MICHELLE GRAEF & ROBIN LEAKE NHSTES June.
DEC RECOMMENDED PRACTICES ARE HERE! NOW…HOW CAN WE ENSURE THEY ARE USED? Pam Winton, DEC Recommended Practice Commissioner Carol Trivette, DEC Board Member.
Dean L. Fixsen, Ph.D. Karen A. Blase, Ph.D. National Implementation Research Network Louis de la Parte Florida Mental Health Institute Implementing Innovations.
The District Role in Implementing and Sustaining PBIS
Implementation Strategy for Evidence- Based Practices CIMH Community Development Team Model Pam Hawkins, Senior Associate Association for Criminal Justice.
Allison Metz, Ph.D., Karen Blase, Ph.D., Dean L. Fixsen, Ph.D., Rob Horner, Ph.D., George Sugai, Ph.D. Frank Porter Graham Child Development Institute.
No, Not the Drivers Again! (Or, RTI is a Journey Not a Destination)
Building Capacity with Implementation Drivers
ISLLC Standard #2 Implementation
V Implementing and Sustaining Effective Programs and Services that Promote the Social-Emotional Development of Young Children Part I Karen Blase, Barbara.
Implementation Science 101 Vestena Robbins, PhD Kentucky Dept for Behavioral Health, Developmental and Intellectual Disabilities.
Implementation Science Vision 21: Linking Systems of Care June 2015 Lyman Legters.
Using Implementation Science To Inform Professional Development
SectionVideo/PresentSlidesTotal Time Overview + Useable Intervention8:30 min Stages7:19 min Teams PDSA Terri present Drivers8:50 min Lessons Learned +
“Current systems support current practices, which yield current outcomes. Revised systems are needed to support new practices to generate improved outcomes.”
Implementation and Scaling Literacy Programs
Program Fidelity Influencing Training Program Functioning and Effectiveness Cheryl J. Woods, CSW.
Selecting, Training and Coaching for Effective and Sustainable RtI Implementation A Model for Leadership Teams Margie McGlinchey; Ph.D. Kim St. Martin;
Implementation and process evaluation: developing our approach Ann Lendrum University of Manchester Neil Humphrey University of Manchester Gemma Moss Institute.
Critical Elements Effective teaching Alignment of curriculum and instruction to standards Instructional program coherence Fidelity of implementation Evaluation.
Dean L. Fixsen, Karen A. Blase, Michelle Duda, Sandra Naoom, Melissa Van Dyke, Frances Wallace National Implementation Research Network Louis de la Parte.
SISC Implementation Science Simcoe Muskoka Kathy Simpson.
Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. & Karen A. Blase Ph.D., Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University.
Barbara Sims Brenda Melcher Dean Fixsen Karen Blase Michelle Duda Washington, D.C. July 2013 Keep Dancing After the Music Stops OSEP Project Directors’
Rob Horner OSEP Center on PBIS Jon Potter Oregon RTI David Putnam Oregon RTI.
DEVELOPING AN EVALUATION SYSTEM BOB ALGOZZINE AND STEVE GOODMAN National PBIS Leadership Forum Hyatt Regency O’Hare Rosemont, Illinois October 14, 2010.
Copyright © Dean L. Fixsen and Karen A. Blase, 2008.
Barbara Sims Dean L. Fixsen Karen A. Blase Michelle A. Duda
Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. & Karen A. Blase Ph.D., Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University.
Scaling Up Improved Education Dean L. Fixsen, Ph.D. & Karen A. Blase, Ph.D. University of North Carolina - Chapel Hill Rob Horner, Ph.D. University of.
Integrated System for Student Achievement SISEP in Illinois.
Dean L. Fixsen, Karen A. Blase, Sandra F. Naoom, Melissa Van Dyke, Frances Wallace National Implementation Research Network Louis de la Parte Florida Mental.
Implementation Conversations Jennifer Coffey Audrey Desjarlais Steve Goodman.
: The National Center at EDC
Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. &
Karen A. Blase, PhD, Allison Metz, PhD and Dean L. Fixsen, PhD Frank Porter Graham Child Development Institute University of North Carolina at Chapel Hill.
State Scaling-up Workgroup March 6, SWG Agenda 9:00 Welcome & Introductions 9:15 Review Agenda/TIPS Meeting Form 9:20 Purpose of the Scaling-Up.
Welcome To Implementation Science 8 Part Webinar Series Kathleen Ryan Jackson Erin Chaparro, Ph.D University of Oregon.
Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. & Karen A. Blase Ph.D., Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University.
Barbara Sims Debbie Egan Dean L. Fixsen Karen A. Blase Michelle A. Duda Using Implementation Frameworks to Identify Evidence Based Practices 2011 PBIS.
National Child Welfare Resource Center for Organizational Improvement A service of the Children’s Bureau, Member of the T/TA Network Readiness for Systemic.
Connecting Resources, Impacting Lives FRCA Quarterly Fall Retreat Tuesday, November 3, 2015 Chris Rubino Director, Strengthening Families Programs.
Copyright © Dean L. Fixsen and Karen A. Blase, Dean L. Fixsen, Ph.D. National Implementation Research Network Louis de la Parte Florida Mental Health.
Help to develop, improve, and sustain educators’ competence and confidence to implement effective educational practices and supports. Help ensure sustainability.
Karen A. Blase, PhD National Implementation Research Network FPG Child Development Institute University of North Carolina at Chapel Hill MiBLSi Coaches.
1 Organizational Readiness Jan M. Markiewicz, M.Ed. National Center August 13, 2007.
Min.cenmi.org Michigan Implementation Network Providing Support through District Leadership and Implementation Team April 29, 2010 Michigan Implementation.
Translating Aspiration to Operation: Developing Active Implementation Capacity and Infrastructure W. Oscar Fleming; MSPH Tuesday, September 29 th 2:30-4:00pm.
An Introduction to Implementation Tools to Help Build Implementation Capacity SPDG Evaluators May 2012 Michelle A. Duda, Dean L. Fixsen,
District Leadership Team Sustainability Susan Barrett Director, Mid-Atlantic PBIS Network Sheppard Pratt Health.
State Scaling-up Workgroup
An Introduction to Implementation Tools to Help Build Implementation Capacity SPDG Evaluators May 2012 Michelle A. Duda, Dean L. Fixsen,
Using the Hexagon tool to Effectively select a practice
CEEDAR-IRIS Cross State Convening
Assessing Readiness Erin Chaparro, Ph.D. Kimberly Ingram-West, Ph.D.
Bringing RTI to Scale in Oregon
Creating a Coherent Communication System
Installation Stage and Implementation Analysis
Presentation transcript:

Sandra F. Naoom, MSPH National Implementation Research Network Frank Porter Graham Child Development Institute University of North Carolina- Chapel Hill Assessing Fidelity: A Necessary Component of Effective Implementation Child Care Policy Research Consortium Annual Meeting 2009

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 What is Implementation? The act of accomplishing some aim or executing some order To put into practical effect; carry out Pursue to a conclusion – Dictionary.com

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Stages of Implementation Major Implementation Initiatives occur in stages: Exploration Installation Initial Implementation Full Implementation Innovation Sustainability 2 – 4 Years

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 INTEGRATED & COMPENSATORY CONSULTATION & COACHING STAFF EVALUATION STAFF EVALUATION FACILITATIVE ADMINISTRATIVE SUPPORTS RECRUITMENT AND SELECTION PRESERVICE TRAINING SYSTEMS INTERVENTIONS SYSTEMS INTERVENTIONS Implementation Drivers (Processes) DECISION SUPPORT DATA SYSTEMS

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Implementation and Fidelity The bridge between a promising idea and the impact on students is implementation, but innovations are seldom implemented as intended”. “The bridge between a promising idea and the impact on students is implementation, but innovations are seldom implemented as intended”. -Berman & McLaughlin, 1976

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Fidelity and Fidelity Measures Fidelity is the degree to which a program as implemented corresponds with the program as described. (Fixsen, Naoom, Blase, Friedman & Wallace, 2005, National Implementation Research Network) Fidelity measures are tools to assess the adequacy of the implementation of the program as described.

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Fidelity in the Field: The Prevalence Implementation efforts in the ‘real’ world, are often unsuccessful and highly variable both in terms of the fidelity and sustainability of the program or practice. (Elliott & Mihalic, 2004). Implementation efforts in the ‘real’ world, are often unsuccessful and highly variable both in terms of the fidelity and sustainability of the program or practice. (Elliott & Mihalic, 2004). About 23%-81% of program activities may be omitted during implementation (Durlak, 1998). About 23%-81% of program activities may be omitted during implementation (Durlak, 1998). “Surprisingly, many of the highest-quality programs fail to take adequate steps to monitor and verify program integrity...This weakens the conclusions that can be drawn regarding the program outcomes and reduces the likelihood that replications will resemble the original program” (Domitrovitch and Greenberg, 2000). “Surprisingly, many of the highest-quality programs fail to take adequate steps to monitor and verify program integrity...This weakens the conclusions that can be drawn regarding the program outcomes and reduces the likelihood that replications will resemble the original program” (Domitrovitch and Greenberg, 2000).

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Why is Fidelity Important? Basis for program effectiveness Basis for program effectiveness Helps to avoid Type III Error Helps to avoid Type III Error Promotes Internal and External Validity Research has linked fidelity with positive outcomes Research has linked fidelity with positive outcomes Adult Mental Health- ACT; Children’s Services- FFT, MST, Wraparound; Medicine- DOTS, Texas Algorithm

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Why Measure Fidelity? Measure Fidelity AND Outcomes BECAUSE you need to know: Are we having an implementation problem? Low fidelity + Poor outcome = Implementation problem Implementation problem Are we having an effectiveness problem? High fidelity + Poor outcome = Effectiveness problem

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Common Dimensions of Intervention Fidelity Adherence Adherence Dosage/Exposure Dosage/Exposure Quality of Program Delivery Quality of Program Delivery Participant Responsiveness Participant Responsiveness Program Differentiation Program Differentiation (e.g. Dane & Schneider, 1998; Dusenbury, 2003)

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 NIRN’s Fidelity Dimensions Context Context Pre-requisites and pre-conditions for a program to operate. Compliance Compliance Extent to which practitioner uses intervention processes prescribed by the program and avoids those proscribed by the program. Competence Competence Level of skill demonstrated by practitioner while delivering services or treatment to the consumer.

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Fidelity Assessments Fidelity assessments should include aspects of : Structure Structure The framework for service delivery. Process Process The way in which services are delivered.

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Methods to Develop a Fidelity Assessment Requires the specification of a theory of change; core components of intervention need to be identified and measured. Drawing from a specific program model with proven efficacy and effectiveness Drawing from a specific program model with proven efficacy and effectiveness Gathering expert opinion Gathering expert opinion Survey of experts and/or literature reviews to identify active ingredients or the core components of the program. Opinions of users and advocates regarding what works or site visits of diverse currently operating programs. Opinions of users and advocates regarding what works or site visits of diverse currently operating programs.

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Measuring Intervention Fidelity Most common methods: Ratings by experts, based on project documentation, client records, site observations, interviews and/or videotaped sessions. Ratings by experts, based on project documentation, client records, site observations, interviews and/or videotaped sessions. Surveys or interviews completed by the individuals delivering the services or those who receive them. Surveys or interviews completed by the individuals delivering the services or those who receive them.

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 For More Information Sandra F. Naoom, M.S.P.H At the Frank Porter Graham Child Development Institute University of North Carolina Chapel Hill, NC

Copyright © Dean L. Fixsen and Karen A. Blase, 2008 For More Information Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: Monograph/ Monograph/ To order the monograph go to: