1. Concurrent Validity of Alternative CANS Outcome Metrics William A. Shennum Julian Leiro Delisa Young Five Acres Altadena, California

3. What is test validity? The concept of validity was formulated by T.L. Kelley in 1927, who stated that a test is valid if it measures what it claims to measure. For example a test of intelligence should measure intelligence and not something else, such as memory. Simplypsychology.org/validity

4. Concurrent validity Concurrent validity is a measure of how well a particular test correlates with a previously validated measure of the same construct. Concurrent validity relies upon measurements that take place at the same time. Explorable.com

5. Who we are Non profit organization serving greater Los Angeles community Research & Evaluation Department – Outcome & performance measurement – Quality & business process improvement – Program evaluation – Project management

6. Five Acres’ service programs

7. Why CANS? Need for a common outcome measure across programs for use in planning & CQI. CANS roll-out Training Inspiration – Dr. Lyons’ visit Technology Reporting

8. Individual Score Report (Excerpt) Client:“I” CANS Date:9/24/20132/3/2014 Life Domain Functioning IntakeDischarge 1. Family 30 2. Living Situation 20 8. Medical 00 10. Sexuality 00 13. School Achievement 21 14. School Attendance 10 Youth Strengths 15. Family 31 16. Interpersonal 11 18. Educational 21

9. What About Program –Level CANS Outcomes? Several scoring options available Which ones work best? – Easily understood by stakeholders – Closely associated with other outcomes important to the organization (concurrent validity)

10. Study Design Three service levels were included. 1. Community based mental health 2. Wraparound 3. Residential treatment

11. Study Design - continued Youth intake and discharge CANS were compared (N=233) Five CANS outcome scoring metrics were computed for each youth Statistical associations of each CANS metric with concurrently measured discharge outcomes were examined.

12. Concurrently Measured Outcomes Service Intensity ProgramProgram Outcome/Success Measures Lower Higher Community based mental health % Service completion (EBP’s) Avg. YOQ points improved (caregiver) Wraparound % Program completion/ graduation % In community placement at discharge Residential treatment (RBS) % Program completion/ graduation % In less restrictive placement at discharge

13. CANS Outcome Metrics Examined in This Study

14. Metric 1. Intake-discharge change scores Method: Compute the average change from intake to discharge, for total score and for each domain Strength- easy to calculate Weaknesses – unit of measurement not common-sense, not tied to clients’ specific needs

15. Metric 1 – Intake/Discharge Change Scores, Example Youth # CANS Total Score Intake CANS Total Score DischargeChange RBS_215.510.45.1 RBS_38.15.22.9 RBS_41.66.3-4.7 RBS_510.73.96.7 RBS_613.312.01.3 RBS_76.74.62.1 RBS_87.44.13.3 RBS_1211.16.64.6 RBS_133.03.7-0.7 RBS_1411.33.67.7 Total Score8.86.0 2.8

16. Program Results for Metric 1 Community Based Services WraparoundResidential Treatment Avg. Change2.00.71.9 SignificanceP<.001P<.004P<.02

17. Metric 2 - Reliable Change Index Analysis Method: Determine whether each individual’s change score is statistically significant, by comparing it with a Reliable Change cutoff value RCI Formula: change in an individual’s score divided by the standard error of the difference for the test. Parameters in formula include Intake and Discharge scores, test standard deviations, and test (rater) reliability. Result is a yes-no score for each youth, for CANS total score, and each domain. – Strength: statistically sound – Weaknesses: difficult to compute and explain to stakeholders

18. How to Calculate the RCI M. Horsill, www.psy.uq.edu.auwww.psy.uq.edu.au 1. Calculate standard error of measurement 2. Calculate standard error of the difference 3. Calculate RCI If RCI is 1.96 or greater, the difference is statistically significant (95% confidence level, 2-tailed). 1.65 = one-tailed test cutoff

19. Metric 2 - Reliable Change Index Analysis, Example Youth # CANS Total Score ChangeRCI Score >=1.96? (1=yes, 0-no) RBS_25.12.231 RBS_32.91.290 RBS_4-4.7-2.080 RBS_56.72.981 RBS_61.30.590 RBS_72.10.950 RBS_83.31.440 RBS_124.62.031 RBS_13-0.7-0.290 RBS_147.73.401 % Exceed RCI Cutoff 40.0%

20. Program Results for Metric 2 RCI Community Based Services WraparoundResidential Treatment >=1.9614.9%10.2%24.2% >=1.6524.1%12.6%30.3%

21. Metric 3. Reduction in Number of Actionable Needs Method: Compare the number of actionable needs at intake and discharge, total and by domain. – Strength- addresses standout needs – Weakness- not tied to an individual’s specific needs identified at intake.

22. Metric 3. Reduction in Number of Actionable Needs, Example Youth # # Actionable Needs Intake # Actionable Needs DischargeChange RBS_22324 RBS_319136 RBS_4310-7 RBS_523716 RBS_632293 RBS_71266 RBS_815213 RBS_12301416 RBS_1356 RBS_1424618 Total Score18.611.7 6.9

23. Program Results for Metric 3 Community Based Services WraparoundResidential Treatment Avg. Change4.51.44.4 SignificanceP<.001P<.12P<.03

24. Metric 4 - % Actionable Needs Resolved Method: Track specific actionable items identified at intake, and calculate the % of these that are no longer actionable at discharge, total and by domain. – Strengths - tailored to individual, easy for stakeholders to understand – Weakness – somewhat complicated scoring

25. Metric 4 - % Actionable Needs Resolved, Example CANS Domain: Youth Strengths IntakeDischarge Need Resolved? (1=yes, 0=no) 15. Family 311 16. Interpersonal 11 17. Optimism 211 18. Educational 211 19. Vocational UU 20. Talents/Interests 211 21. Spiritual/Religious 220 22. Community Life 211 23. Relationship Permanence 220 24. Resiliency 11 25. Resourcefulness 211 % Needs Resolved 75%

26. Program Results for Metric 4 Community Based Services WraparoundResidential Treatment % Needs Met72.8%49.2%61.6%

27. Metric 5 – Actionable Needs Improved Method: Track specific actionable items identified at intake, and calculate % of these that show any improvement, even if still actionable at discharge, total and by domain. – Strength - tailored to individual, easy for stakeholders to understand – Weaknesses – includes needs still remaining at discharge, somewhat complicated scoring

28. Metric 5 – Actionable Needs Improved, Example Youth Behavioral/Emotional Needs IntakeDischarge Any Improvement? (1=yes, 2=no) 41. Psychosis 00 42. Impulse/Hyper 201 43. Depression 10 44. Anxiety 11 45. Oppositional 321 46. Conduct 211 47. Adjustment to Trauma 321 48. Anger Control 220 49. Substance Use 00 % Any Improvement in Actionable Needs 80%

29. Program Results for Metric 5 Community Based Services WraparoundResidential Treatment % Needs Improved 75.3%59.4%66.6%

30. Concurrent Validity of the 5 CANS Program Outcome Metrics Are the metrics statistically associated with concurrently collected measures of program success?

31. All CANS Metrics Significantly Associated with Concurrent Program Outcomes CANS Outcome Metrics Association w/ Concurrent Outcomes 1. Avg. Change 2. RCI Cutoff 3. Change in # Needs 4. Needs Resolve d 5. Needs Improved Median Statistical Significance p =.020 p =.043 p =.009 p =.013 p =.018

32. Specific CANS Domains were Positively Associated with Concurrent Outcomes CANS Outcome Metrics for Selected Domains FunctioningYouth Strength Family Needs & Strengths Behavior & Emotional Needs Median Correlation w/ Outcomes r =.23 r =.33 r =.22 r =.28

33. Some Differences Across Program Were Evident CANS Outcome Metrics Program / OutcomeAvg. Change RCI Cutoff Change in # Needs Needs Resolved Needs Improved Community Based Service completionp<.001p<.05p<.001p<.01p<.05 YOQ Pointsp<.05 p<.01 Wraparound Program completion p<.001 In communityp<.01nsp<.05p<.10ns Residential (RBS) Program completion p<.10p<.05 Less restrictivens p<.05p<.10

34. CANS outcomes for ‘successful’ youth CANS Outcome Metrics Program Avg. Change in Total Score % Youth exceeding RCI 1.65 Avg. Change in # of Actionable Needs Avg. % Actionable Needs Met Avg. % Actionable Needs Improved Community MH 2.329.0%5.676.7%78.9% Wraparound 2.634.8%7.068.9%79.9% Residential 3.136.8%8.170.5%74.6% Average 2.733.5%6.972.0%77.8%

35. Trends “Reduction in number of actionable needs” and “% of actionable needs resolved” metrics tended to work best for most programs “Average change score” metrics worked well in program serving less severe youth Suggestive evidence favored youth strengths domain as a driver of overall improvement CANS outcome profiles for successful youth were similar across programs – benchmarks would assist in setting performance targets.

36. Take Aways All CANS metrics examined showed evidence of concurrent validity across a range of programs All CANS domain improvements were related to concurrent outcomes, with suggestive evidence that “youth strengths” had strongest relationship to success. Programs serving higher severity needs may benefit from measuring metrics related to actionable need resolution and improvement. Benchmark databases could assist programs in setting outcome targets for youth.

37. Contact Information Bill Shennum Director of Research & Evaluation Five Acres (626)798-6793 bshennum@5acres.org www.5acres.org