Research & Evaluation

Defining Recidivism  Felony adjudication (conviction) within 3 years of release from closed custody or commitment to probation.

Defining the Assessments  ORRA: OYA Recidivism Risk Assessment. Identifies the likelihood a youth will recidivate after release from closed custody or commitment to probation.  ORRA-V: OYA Recidivism Risk Assessment–Violent. Identifies the likelihood a youth will VIOLENTLY recidivate after release from closed custody or commitment to probation. Recognizes the propensity for violence or threatening crimes that may result in physical harm.

Why Develop a Risk Assessment  Program evaluation  Placement and treatment decisions  Parole Decisions  Sentencing practices  RNA fails to differentiate risk populations  Poor predictive accuracy

The Data  15,968 total youth Date range of population January 2005 to May 2007  Four Youth Populations County Probation Committed to OYA Probation Released from OYA Close Custody Facility Released from OYA Close Custody Facility to Supervision in the adult system

Dozens of Variables were Considered  Age at first referral  Total prior sex offense referrals  Total prior felony referrals  Total prior theft referrals  Total prior runaway referrals  Total prior property referrals

Dozens of Variables were Considered (cont.)  Total burglary referrals  Total prior misdemeanor referrals  Total prior robbery referrals  Total prior violation referrals  Total prior dependency referrals  Total prior harassment referrals

Variables contributing to the ORRA Scores and their effects  Prior felony AOD referral (Y/N)  Prior weapon referral (Y/N)  Age  Criminal mischief referral (Y/N)  No. prior misdemeanor referrals  No. prior theft referrals  Adjudicated delinquent (Y/N)  No. prior AOD referrals  Current sex offense (Y/N)  No. prior runaway referrals  Gender (male higher risk)  Interactions – mischief referral by No. prior misdemeanors No. prior theft referrals No runaway referrals +25.5% Flattens out Increases Flattens out

So… What exactly does this mean?

Meet the Twins… Age 15 Male 3 Runaways 1 Prior Felony Drug Referral 1 Prior Misdemeanor Referral

Interpreting Scores Each youth has a score between 0% and 100% The score approximates the probability that the youth will recidivate ○ For example, a youth with a score of 40% has a 40% probability they will recidivate ○ This also means the youth has a 60% approximate score that the youth will NOT recidivate.

Model Accuracy  Overall Accuracy for ORRA = 73% Accurate for all subpopulations  Accuracy of 50% suggests poor predictive accuracy  Accuracy of 100% suggests perfect predictive accuracy

Comparison of the Four Populations

Differences in Youth Populations

Predictive Accuracy

Interpreting Scores ORRA and ORRA-V scores can also be evaluated for a specific population The average score for a group of youth estimates the expected recidivism rate for the group

Program Evaluation Actual vs. Expected Recidivism  Calculate risk scores Expected (based on average risk of youth served) Actual (based on recidivism of youth served)  Determine Relative increase or decrease Facilitates meaningful comparisons across providers Providers Serving 30 or more youth from 1/1/2000 to 12/13/2007; 36-Month Recidivism Treatment Provider Expected Rate Actual Rate Percent Increase or Decrease Provider A18.8%13.0%-31.0% Provider B20.3%16.3%-19.5% Provider C21.9%17.9%-18.2% Provider D26.8%22.2%-17.1% Provider E16.5%14.0%-15.4% Provider F22.4%19.2%-14.3% Provider G27.6%25.0%-9.3% Provider H14.9%13.6%-8.4% Provider I30.2%29.7%-1.7% Provider J28.2%28.0%-0.7% Provider K36.8%40.4%9.9% Provider L24.5%27.1%10.6% Provider M26.4%29.9%13.2% Provider N26.1%30.0%14.7% Provider O25.3%37.1%46.5%

Things Done and Things Still to Do  Done -- Test for all OYA youth groups Males/Females Minorities Crime Type  Still to Do -- Make the ORRA dynamic Incidence Revocations Programming

ORRA-V  Used the same dataset  Used “violent recidivism” – a subset of recidivism  Violent recidivism includes murder, arson…robbery, assault, and burglary

Variables contributing to the ORRA-V Scores and their effects  Male  Prior weapon referral (Y/N)  No. prior misdemeanor referrals  No. prior felony referrals  Prior felony assault referral (Y/N)  Prior felony theft referrals (Y/N)  Misdemeanor theft referrals (Y/N)  Prior curfew violation (Y/N)  No. prior runaway referrals  Interactions: Weapons X felony theft Misdemeanor Referrals X felony referrals % Flattens out

Differences between the ORRA and the ORRA-V Variable ORRAORRA-V Male % Weapon offense Misdemeanor referrals Runaway referrals Felony referrals + 9 Felony assault referrals + 32 Felony theft referrals + 36 Misdemeanor theft referrals + 20 Curfew violation + 22 Felony AOD referral + 26 Age + 5 Mischief referral + 83 Number theft referrals + 5 Prior adjudication + 22 Number AOD referrals + 11 Current sex offense - 40

OVIRA and ONIRA  OVIRA measures the likelihood a youth will engage in a violent act in the first six months of closed custody  ONIRA measures the likelihood a youth will engage in numerous non-violent incidents in the first six months of closed custody

Data for OVIRA and ONIRA  Youth admitted to OYA between November 2007 and December 2009  N = 1,258  90% male and 10% female  27% property crime, 25% sexual offenses, and 9% robbery  64% YCF, 11% DOC, and 11% revoked

Variables considered for OVIRA and ONIRA  ORRA and ORRA-V  RNA data – aggression, drugs/alcohol, mental health, employment, relationships, attitudes, etc.  Gender  Age  Sexual offender  Special education and learning disability  Other variables

OVIRA – OYA Violent Incident Risk Assessment  Violence considered an assault or peer fight resulting in isolation/segregation  Considered “immediately threatening to life, health, or facility safety, security, or good order.”

ONIRA – OYA Nuisance Incident Risk Assessment  Considered four or more non-violent incidents in the first months of closed custody

Variables contributing to OVIRA and ONIRA scores VariableOVIRA ONIRA Age at admission -20% - 27% Male -43 SED Sex offender Mental health protective - 9 Full relationship risk +29 Belief in fighting/aggression +49 RNA prescreen social score - 11 Mental health risk + 28 Aggression protective - 22 Parental authority/control + 50 ORRA+1224 (HR=13.2) ORRA-V - 95 (HR=.05)

Typologies

C E B D A F

Type A Description  Highest need population  AOD use is high both current and historical  Poor relationships and likely lack relationship skills  Highest on aggression and attitude issues  History of Mental Health = ADD/ADHD or mental health diagnosis – recommend analysis of RNA items 15.5 and 15.6 to differentiate ADD/ADHD versus Formal MH Diagnosis  Education issues are prominent – recommend analysis of RNA item 3.1 for potential responsivity issues 3.1 = Special Education or Formal Diagnosis of Special Education Need (LD, SED, MRDD Indicators)

Treatment Recommendation  Estimated to require longest dosage of treatment (e.g., months)  Group may require more stabilization than other groups due to co-occurring mental health and learning concerns  AOD Treatment (longer in duration due to persistency)  MH treatment with QMHP  Educational intervention  Social Skills/Relationship Skills development (intensive)  Engagement in prosocial activities that can foster protective factors  Potential family therapy component  Aggression Replacement Training (intensive)  Cognitive Behavioral program to address thinking

Type E Description 66% of this cluster is SO Highest on protective factors Low need for MH = ADD/ADHD or mental health diagnosis – recommend analysis of RNA items 15.5 and 15.6 to differentiate ADD/ADHD versus Formal MH Diagnosis Education issues are low – recommend analysis of RNA item 3.1 for potential responsivity issues 3.1 = Special Education or Formal Diagnosis of Special Education Need (LD, SED, MRDD Indicators)

Treatment Recommendations (Type E) Sex Offender Treatment when appropriate (Abbreviated Kaufman or general cognitive behavioral treatment) Capitalize on whatever activities youth engaged in prior to coming as leverage for treatment engagement Seek opportunities for continued engagement

Optimal Length of Stay  Calculated length of stay in months  Plotted LOS against recidivism for the overall sample  On average, providers reduce recidivism by approximately 3% per month of supervision  But, there may be a window of time where providers are most effective

Program Evaluation Continuum

Summary  ORRA  ORRA-V  OVIRA  ONIRA  Typology (being completed)  Optimum dose (next project)  Program continuum (being developed)  LOS report  Recidivism report  Timing study for JJPOs  Revocation (being completed)  Culture climate survey (data collection completed)  Staff-management/leadership survey (data collection now)  PREA – identifying vulnerable youth (surveyed thru October)

Close Custody Populations Making comparisons while controlling for risk

Why  Problems with the RNA Not valid for OYA females Approximately 85% of the youth in Close Custody were High Risk – little practical information The Area Under the Curve (AUC) was.56  DOC had the solution Methodology for developing risk tool based on local data The AUC for their risk tool was.78

How  Methodology Subjects ○ N = 28,431 dispositions (19,309 unique youth) ○ Qualifying events occurred between 1/1/2005 and 5/14/2010 ○ Youth qualified if they were: Placed on county probation Committed to OYA probation Released from an OYA close custody facility Release from OYA close custody to supervision in the adult system

What’s Next  ONIRA: OYA Nuisance Incident Risk Assessment  OVIRA: OYA Violent Incident Risk Assessment

How  Methodology continued Omitted disposition records of youth: ○ Supervised under interstate compact ○ Returned to DOC to complete their sentences in adult institutions ○ Committed to OYA or county probation who were subsequently committed to an OYA YCF without recidivating Randomly selected one disposition per youth Final dataset: N = 15,986

How  Methodology continued Dependent (Outcome) Variable ○ Recidivism event: OYA official recidivism measure Felony Adjudication or Conviction Disposition of formal supervision ○ Groups ○ Tracking Dates ○ Tracking Periods: 12-, 24-, 36-, 48-Month

How  Methodology continued Independent Variables ○ Over 50 starting variables ○ Bootstrap Re-sampling Run 1000 randomly sampled logistic regressions for each tracking period Lists the proportion of time each variable is significantly related to the outcome variable Selected the top 30% of the variables to develop the model ○ Run stepwise Logistic Regression for each tracking period

How  Methodology continued Developing the Model ○ Run stepwise Logistic Regression for each tracking period ○ Determine the concordance rate for each model ○ Test for interactions ○ Run stepwise Logistic Regression for each tracking period including significant interaction variables

How  Methodology continued Selecting and refining the final model ○ 36-Month tracking period had the highest concordance rate (.73) and included 12 predictor variables 3 interaction terms

How  Results Model Accuracy: ○ AUC =.72 ○ Estimates Actual Recidivism

How  The Model PREDICTOR VARIABLES VALUES ODDS RATIO Prior felony drug or alcohol referralNo = 0, Yes = Prior weapon offense referralNo = 0, Yes = Age at start trackingAge at probation start or release to community from close custody Prior criminal mischief referralNo = 0, Yes = Total prior misdemeanor referralsSum (maximum = 20)1.103 Total prior theft referralsSum (no maximum)1.052 Prior delinquency adjudicationNo = 0, Yes = Total prior drug or alcohol referralsSum (no maximum)1.111 Current sex offense dispositionNo = 0, Yes = Total prior runaway referralsSum (maximum = 20)1.114 Total prior felony referralsSum (maximum = 6)1.204 MaleFemale = 0, Male = Interaction: prior criminal mischief referral & total prior misdemeanor referrals Product of the two variable terms specified0.897 Interaction: prior criminal mischief referral & total prior theft referrals Product of the two variable terms specified1.108 Interaction: prior criminal mischief referral & total prior runaway referrals Product of the two variable terms specified0.935

What for  Interpreting ORRA Scores Each youth get a score between 0 and 1 The score represents the probability that the youth will recidivate ○ For example, a youth with a score of.42 has a 42% probability they will recidivate The average score for a group of youth estimates the expected recidivism rate for the group ○ For example, the average ORRA score for females on OYA probation was 13.1 and the actual recidivism rate was 13.0.

What for  ORRA has multiple uses Placement and treatment decisions Parole decisions Program evaluations Sentencing practices Foundation for future improvement in risk assessment

What for Making comparisons while controlling for risk

What for Making comparisons while controlling for risk

What for Making comparisons while controlling for risk

Other risk equations  ORRA+  ORRA-V  Risk of being involved in a violent incident in the first year in OYA close custody

Implementing ORRA Scores…An Example  Used ORRA Scores in Evaluating the Effectiveness of Residential Programs Is the actual recidivism rate different than the predicted recidivism rate? Is there an optimal length of stay? With which youth is a program most effective?  All youth in residential programs from 2000 to 2007 Used official OYA definition for recidivism

Actual vs. Expected Recidivism  Calculated risk scores Expected (based on average risk of youth served) Actual (based on recidivism of youth served)  Determined Relative increase or decrease Facilitates meaningful comparisons across providers Providers Serving 30 or more youth from 1/1/2000 to 11/1/2007; 36-Month Recidivism Treatment Provider Youth Served Expecte d Rate Actual Rate Percent Increase or Decrease A % B % C % D % E % F % G % H % I % J % K % L % M % N % O %

Next step? Right Youth…Right Program  In depth analysis about who programs are most effective with Potential variables include sex, age, offense type  In addition to understanding which youth are most effectively served by a specific program, this analysis may identify gaps and determine which youth are not served well by current provider resources

Questions  Contact Research: Lance Schnacker (503) ○ Paul Bellatty ○

Current OYA Population

Why  Problems with the RNA Not valid for OYA females Approximately 85% of the youth in Close Custody were High Risk – little practical information The Area Under the Curve (AUC) was.56  DOC had the solution Methodology for developing risk tool based on local data The AUC for their risk tool was.78

How  Methodology Subjects ○ N = 28,431 dispositions (19,309 unique youth) ○ Qualifying events occurred between 1/1/2005 and 5/14/2010 ○ Youth qualified if they were: Placed on county probation Committed to OYA probation Released from an OYA close custody facility Release from OYA close custody to supervision in the adult system

How  Methodology continued Omitted disposition records of youth: ○ Supervised under interstate compact ○ Returned to DOC to complete their sentences in adult institutions ○ Committed to OYA or county probation who were subsequently committed to an OYA YCF without recidivating Randomly selected one disposition per youth Final dataset: N = 15,986

How  Methodology continued Dependent (Outcome) Variable ○ Recidivism event: OYA official recidivism measure Felony Adjudication or Conviction Disposition of formal supervision ○ Groups ○ Tracking Dates ○ Tracking Periods: 12-, 24-, 36-, 48-Month