Prepared by the Justice Research and Statistics Association IMPLEMENTING EVIDENCE-BASED PRACTICES

THE IMPORTANCE OF EVIDENCE-BASED PROGRAMS The evidence-based movement is one of the most significant developments to occur in criminal and juvenile justice in recent years. Evidence-based programs (EBPs) are important because they can be expected to produce results and a sound return on taxpayer investment. Funding sources are increasingly promoting the use of EBPs. 2 Prepared by the Justice Research and Statistics Association

IMPLEMENTATION MATTERS Identifying and adopting an EBP is not enough to achieve successful outcomes. An evidence-based program still has to be delivered with fidelity/integrity in diverse and complex real- world settings. 3 Prepared by the Justice Research and Statistics Association

HOW ARE EBPS IDENTIFIED? EBPs are deemed to be effective based on rigorous scientific research. While specific criteria may vary, all organizations assess the quality and consistency of evidence when determining program effectiveness. 4 Prepared by the Justice Research and Statistics Association

THE IMPORTANCE OF THE PROGRAM MODEL Conclusions about a program’s effectiveness are derived from evaluations of a specific set of program components and activities, typically referred to as the program model. An EBP should be implemented in a manner that is consistent with its program model. Implementing an EBP in a way that deviates from the program model changes the program and invalidates expectations regarding the program’s effectiveness. 5 Prepared by the Justice Research and Statistics Association

IMPLEMENTATION FIDELITY Implementation fidelity refers to the degree to which a program’s real-world implementation matches the program model. Research has shown that programs implemented with a high degree of fidelity are more likely to be successful. Deviating from the program model not only can degrade program effectiveness, but also can result in an otherwise effective program doing more harm than good. 6 Prepared by the Justice Research and Statistics Association

ORDINARY CIRCUMSTANCE CAN BE AN OBSTACLE TO IMPLEMENTATION What appears to be simple and straightforward in the implementation process often turns out to be more complex than anticipated. We often underestimate the number of steps involved, the number of separate decisions that have to be made or the number of participants whose preferences have to be taken into account. Because of the complexity involved, implementation, even under the best circumstances, is exceedingly difficult. 7 Prepared by the Justice Research and Statistics Association

HIGH-QUALITY IMPLEMENTATION IS DIFFICULT TO ACHIEVE A 2011 Westat study found that only about 3.5% of all school-based programs to prevent youth substance abuse and school crime are research-based and well-implemented. A 2013 EPISCenter study found that nearly half of the Blueprint program implementations in PA involved adaptation, and the majority (53%) of these adaptations were predicted to negatively impact program effectiveness. 8 Prepared by the Justice Research and Statistics Association

IMPLEMENTATION SCIENCE Research-based insights and tools that can be used to support high-quality implementation in complex real-world settings 9 National Implementation Research Network (NIRN) Prepared by the Justice Research and Statistics Association

IMPLEMENTATION SCIENCE: LESSONS LEARNED Implementation is a process, not an event. Implementation occurs in stages: – Exploration: acquisition of information – Installation: preparation, behind-the-scenes tasks – Initial implementation: small scale, trial change in practice – Full implementation: the innovation becomes accepted practice; full staffing and client loads Entire process takes 2-4 years. 10 Prepared by the Justice Research and Statistics Association

TRADITIONAL STRATEGIES TO PROMOTE EBPS ARE INSUFFICIENT TO ACHIEVE SOUND IMPLEMENTATION Diffusion and dissemination of information Passing laws or using other mandates Providing funding, or using funding incentives Training Organizational development 11 Prepared by the Justice Research and Statistics Association

FACTORS THAT AFFECT IMPLEMENTATION FIDELITY Complexity of the program Specificity of information in the program model Facilitation strategies available to the implementing organization Responsiveness of clients and staff 12 Prepared by the Justice Research and Statistics Association

CORE PROGRAM COMPONENTS Core components are the parts of a program that are vital for program success and should not be eliminated or altered prior to or during implementation When core components are known, organizations should ensure they are implemented with the highest degree of fidelity possible. 13 Prepared by the Justice Research and Statistics Association

ADAPTATION The deliberate or accidental modification of a program – Deletions, additions or enhancements of program components – Changes in the manner or intensity of administration of program components – Cultural and other modifications required by local circumstances Excessive program adaptation is a common problem. 14 Prepared by the Justice Research and Statistics Association

ADAPTATION Finding the right balance between fidelity and adaptation is difficult. An evidence-based program should be implemented with the highest degree of fidelity possible. Adaptation is likely to be advantageous only when it is highly strategic, pursued with caution, and monitored to prevent potentially harmful effects. 15 Prepared by the Justice Research and Statistics Association

SYSTEMS TRUMP PROGRAMS Systems are not very accommodating to innovation and change. Systems will exert pressures to alter a new initiative so it fits into existing structures with the least amount of change. 16 Prepared by the Justice Research and Statistics Association

IMPLEMENTATION DEPENDS ON: Core components of implementation that facilitate implementation fidelity Based on the commonalities found among successfully implemented programs Integrated and compensatory in nature 17 Prepared by the Justice Research and Statistics Association

IMPLEMENTATION DRIVERS Staff selection Pre-service and in-service training On-the-job coaching Staff and program evaluation to identify problems and guide mid-course corrections Facilitative administrative support to address technical and adaptive problems 18 Prepared by the Justice Research and Statistics Association

OVERCOMING RESISTANCE Implementation teams – Must have the knowledge, skill, freedom, and authority to act within the larger organization or a collaboration of agencies – Actively support and facilitate implementation on a daily basis Communities of practice – Mechanisms for practitioners to interact and exchange craft knowledge Practice to policy feedback loops 19 Prepared by the Justice Research and Statistics Association

ASSESSING IMPLEMENTATION FIDELITY Program structure – the framework for service delivery – The presence of key program elements – Participation of requisite partner organizations – Resources allocated to the program – Staffing levels and characteristics Program processes - the way services are delivered – Adherence to program protocols or procedures – Frequency and intensity of services provided – The quality with which program staff delivers services 20 Prepared by the Justice Research and Statistics Association

ASSESSING IMPLEMENTATION FIDELITY Context - prerequisites that must be in place for the program to operate – Staffing qualifications and numbers – Practitioner-consumer ratio (i.e., caseload size) – Completion of requisite training Compliance with the intervention components prescribed in the program model Competence or level of skill shown by the program staff 21 Prepared by the Justice Research and Statistics Association

ASSESSING IMPLEMENTATION FIDELITY Adherence to the program model - is the program being delivered as it was designed? Dosage or Exposure - do program participants receive the amount of the intervention prescribed in the program model? Quality of program delivery - do staff members deliver the program with skill? Participant responsiveness - are program participants engaged by program activities? 22 Prepared by the Justice Research and Statistics Association