The Standardized Program Evaluation Protocol ~SPEP~ Division of Adult Correction and Juvenile Justice The Standardized Program Evaluation Protocol ~SPEP~ Understanding the Tool and Program Improvement Implications

First State-wide community-based alternatives (CBA) ~ late 1970s Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices First State-wide community-based alternatives (CBA) ~ late 1970s First “wraparound” model in the US via Willie M. Program~ 1979 class action lawsuit 1998 NC Juvenile Justice Reform Act ~ Second state to mandate evidence-based services I review these “firsts” because I think it is important to stress that NC has been the first with the aide of JCPCs to move much of the juvenile justice reform for NC in the positive direction that it has taken. NC adopted evidenced-based practices very early on and mandated it by legislation (point out that this was an unfunded mandate). This leads to further discussion of how the SPEP was first implemented in NC.

Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices First state to fully adopt the OJJDP Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders First legislated statewide, county-level Juvenile Crime Prevention Councils (also first statewide use of a risk factor assessment beginning in the late 1990’s) Most effective statewide use of a juvenile offender risk and needs assessment instrument along with a disposition matrix to reduce confinement (Nov, 2001)

Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices First state to begin implementation of a state-wide gang prevention and intervention initiative (Community-Based Youth Gang Violence Prevention Project) 2009 (Use of Recovery Act funds) (THE first U. S. experiment to fund statewide Comprehensive Gang Model programming!)

Historical Context of SPEP in NC 1998 NC Juvenile Justice Reform Act required a statewide evaluation of programs funded through the JCPC The Standardized Program Evaluation Protocol ~SPEP~ was developed specifically for this purpose in NC. 2006 First statewide Level II services based on SPEP primary service types (2010) Here is the discussion about the context of SPEP and historically how it fits with what we are currently doing.

Most (57%) JJ programs reduce recidivism: Outcomes of 556 studies (Dr Most (57%) JJ programs reduce recidivism: Outcomes of 556 studies (Dr. Mark Lipsey, 2002) SPEP 1.0 lessons learned included control studies whereby meta-analysis indicated that juvenile justice programs were effective. We learned that there were several domains that we focused upon in this literature…….primary/supplemental services, frequency and duration, and targeted risk level youth. The purpose of this training is to building upon the research and learning about the changes in the tool in order to impact programming at the local level to effect recidivism of youth served.

The SPEP Process Key Steps to SPEP Score

EVALUATE/PROGRAM IMPROVEMENT IDENTIFY SERVICES SPEP SCORING PROCESS MATCH AGAINST RESEARCH-BASED CATEGORIES DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE SPEP SCORE This process for programs will be completed step by step with the ultimate goal of program improvement. EVALUATE/PROGRAM IMPROVEMENT

IDENTIFY/ MATCH SERVICES DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE) Each of the factors found in the meta-analysis to be importantly related to program effectiveness is represented in the SPEP and associated with a certain maximum number of points to provide a score. The first step is identification of the primary service and a possible qualifying supplemental service. A crosswalk into the appropriate therapeutic approach and service type is important to understand. Sixty-five percent of the scoring comes from data generated by the program through the use of various tools (quality of service), duration/frequency(hrs) NCALLIES, and Level of Risk (risk assessment tool). NCALLIES will be an integral part of the scoring. Point out that the Primary Service Type groupings are directly proportionate to the contributing factors associated with that type of program group that effects recidivism. There are some program types that are more effective than others. The risk levels of juveniles served must be based on the results of a valid risk assessment instrument (North Carolina’s Assessment of Juvenile Risk of Future Offending ). Ratings of the amount of service must be based on management information system data, in this case, NCALLIES, that report the service received by each juvenile. SPEP is data driven.

Classification of Program Intervention Getting Started……. Classification of Program Intervention

SPEP 101 Definitions Structure or Format: Setting within which the program services are delivered Example: Residential, YDC, Day Treatment Program: Services that are delivered within the context of a structure or format. Service: The treatment ingredient of the program. Services target key criminogenic risk factors and treatment needs and are direct interventions with the juvenile or with others (peers or family) in order to alter juvenile behavior that leads to reduced recidivism. For continuation programs and JCPC members that have been with us for the long-haul , this was the basic teaching in the first phase of SPEP. Review.

Identification of “Philosophies” toward Altering Juvenile Behavior Program Services Fall into 3 Broad Categories: External Control Techniques Behavioral change through instilling discipline, fear, or detection of bad behavior in the absence of treatment Therapeutic Techniques Behavioral change through improved skills, relationships, insight. These are the only services for which SPEP ratings are completed. Other :Services for which there is insufficient research to estimate the effects on recidivism Meta-analysis is limited to recidivism effects in the study. The focus is upon recidivism with the SPEP. Meta-analysis allows for sorting of the program types that have the greatest effect on recidivism. The sorting within the meta-analysis generated broad categories of interventions/ techniques that fell into 2 main categories: External Control Techniques: discipline, fear (bootcamp) programs; Therapeutic Techniques: those that improved skills, relationships, insight. Focus was upon the therapeutic approaches given that they had positive impact on recidivism reduction. JCPC need to understand that they need to fund therapeutic approaches. There will be structures, however, that are important to the continuum of services that play a vital role in providing services to youth.

Program Types Sorted by General Approach: Average Recidivism Effect Multiple services Counseling* Skill building** Restorative Surveillance Deterrence Discipline -10 -5 5 10 15 % Recidivism Reduction from .50 Baseline Therapeutic approaches Control approaches Control approaches Broad categories were generated from the meta-analysis. Programs sort into two large / broad categories. Control approaches , not very effective, Discipline: like “Scared Straight” and boot camp/para military, “pushups before breakfast” style programs actually have a negative impact on recidivism. Deterrence programs such jail visitations , scared straight programs. Surveillance: intensive surveillance supervision, positive effects Therapeutic programs and approaches reduce recidivism and some approaches are clearly more effective than others. All on average have positive effects on recidivism. It is important to note here that there will be a great variance in the recidivism reduction. Point out that programs along a bar graph, though they may have positive effects, their positive effects can vary greatly from one program to another in the meta-analysis studies.

Further Sorting by Intervention Type within, e. g Further Sorting by Intervention Type within, e.g., Counseling Approaches Point out that within program groupings, there is further distinction between the program type and impact on recidivism; though all have some impact on recidivism reduction. Make a point of stressing that there is great variance in the effects of programming on recidivism, some with greater effects (i.e. mentoring…some as great as about 22% reduction) but yet there were some programs in the meta-analysis data that had very little, yet positive effects. All on average show positive effects on recidivism, but there is also variation. Some approaches seem to have greater effects than others. Mentoring is grouped with counseling because the counseling approach is considered to be . . . “a personal relationship between the offender and a responsible adult who attempts to exercise influence on the juvenile’s feelings, cognitions, and behavior.” Mark Lipsey

Further Sorting by Intervention Type within, e. g Further Sorting by Intervention Type within, e.g., Skill-building Approaches Ditto Last category does not show strong positive effects in this literature, but keep in mind the average age in about 14 in the literature. Point out variation of effects.

SPEP 101 Definitions Primary Service: Main focus of the program Most time/effort spent on this Recognized as the dominant theme Other program elements support its effectiveness

SPEP 101 Definitions Supplemental Service: Other distinct service Less time/effort spent on this Serves to reinforce or complement the primary service Note: Supplemental service can be deemed as a “qualifying” one in order to receive additional points on the SPEP tool OR points may be automatic based on lack of evidence in the research base to couple a supplemental service with the identified primary service.

WHAT IS THE PROGRAM’S PRIMARY SERVICE? IN WHICH OF THE 5 SERVICE GROUPS DOES THE SERVICE FALL? IS THERE A QUALIFYING SUPPLEMENTAL SERVICE ?

Things to Consider……… JCPCs will need to fund structures or formats. (Teen Court, Structured Day Programs) It is important that an analysis of key therapeutic element (s) within the structures be explored. JCPCs need to fund Therapeutic Approaches/Interventions JCPCs need to understand the SPEP process in order to understand SPEP scores and their implications for program improvement.

DATA Examining the Service Quality, Amount of Service (Duration and Contact Hours), and Risk Level

Data Used for SPEP Scores Quality of Service: SPEP Quality of Services Checklist~ Completed with the Area Consultant Examines the following: Written program protocol ~ program manual; written protocol that describes how the intended services is delivered Staff training /staff retention Monitoring of Program effectiveness, protocol and corrective action to correct deviations

SPEP Quality of Services Checklist Score Point out that the Quality Services Checklist is a tool used by the consultant to review with the provider those areas noted in the previous slide. The score from the checklist would generate up to 20 points on the SPEP.

Data Used for SPEP Scores Amount of Service: (NCALLIES) Duration of Service Target number of weeks specified for each service type % of youth who received targeted weeks of service Maximum number of points ~10 Contact Hours Target number of hours specified for each service type % of youth who received targeted hours of service SPEP scores are data driven and 20 points would come from information derived from the % of terminated clients during the examination period that meet the targeted number of weeks/targeted number of hours specified for the service type.

Data generated from NC ALLIES Again, the SPEP is data driven based in the data entered by the program. I think it is important here to stress that this data is based on the % of terminated youth that meet the targeted number of weeks/targeted number of hours specified for the service type during the examination period.

Data Used for SPEP Scores Risk level of Youth Served: Program’s use of a validated risk assessment instrument/tool to provide juvenile risk level data on each juvenile receiving services North Carolina’s Assessment of Juvenile Risk of Future Offending ~ Validated tool currently being used by Juvenile Court Services. This risk assessment tool has been selected for use by program providers. The department will train providers on its use. NC ALLIES will be modified with the addition of a Risk Level field. Risk Level Scores from referrals from Juvenile Court would be entered as well as risk level scores generated by the program. Future modifications will include incorporating the risk assessment tool into NCALLIES. Consider that there are generally 2/3rds of youth that are low risk for reoffending in NC. The research supports this and NC data from NCJOIN also suggests this. It is important to realize that the SPEP research centers around recidivism reduction of youth that have involvement with the juvenile justice system. Many JCPC programs serve a variance of youth risk levels. Court services utilizes the Juvenile Risk of Future Offending tool which has been validated in 6 studies. There is a strong leaning toward use of this tool by providers to determine the level of risk for reoffending for youth being served. Risk levels/scores would come with youth referred from juvenile court services and programs would generate risk levels based on its administering of a risk tool for each youth served. (The department has now decided to use the North Carolina Assessment of Juvenile Risk of Future Offending Tool. Providers will be trained on the use of the tool.)

Data Generated via a Validated Risk Assessment Tool and entered into NC ALLIES

The SPEP SCORE Examines how a specific program is performing compared to the effective practice for that service type in the research Provides opportunities for program improvement for programs that fall short of the evidence-based effective practice profiles.

Systematic Correction to Optimize Recidivism reduction Effectively SPEP allows for a systematic means by which we (the JCPC and the Department) can help programs correct their programming/agency practices to optimize recidivism reduction in an effective and meaningful manner.

Scoring = Improvement Opportunity Initial Score Understanding the Score Implications for Improvement Program Improvement Plan Subsequent Scoring Repeat Scoring is just an improvement or enhancement opportunity. SPEP 2.0 will provide a “baseline” score; with time and planning for improvement; score again; cyclical.

Program Improvement Program Improvement Plans address the areas needing improvement as identified through the program’s SPEP score. Timeframes for program improvement will be established Monitoring of the status of program improvement will be incorporated into the JCPC monitoring/planning processes and Consultant Program Monitoring JCPCs would continue to evaluate the effectiveness of their funded programs through general oversight and supporting program improvement plans. There is a role for the JCPCs in the program improvement of their locally funded JCPC programs. JCPCs can be supportive to the program improvement planning efforts through their local JCPC monitoring/planning processes.

Central Area Consultant Division of Adult Correction and Juvenile Justice Questions? Ronald Tillman Central Area Consultant Ronald.tillman@ncdps.gov 919-323-6845