Concurrent Validity of Alternative CANS Outcome Metrics William A. Shennum Julian Leiro Delisa Young Five Acres Altadena, California.

Slides:



Advertisements
Similar presentations
Item Analysis.
Advertisements

Measurement Concepts Operational Definition: is the definition of a variable in terms of the actual procedures used by the researcher to measure and/or.
This outcome report is based on data from clients who completed a Functional Restoration Programme at the RealHealth Treatment Centre in Coventry between.
Skagit Wraparound Data and Statistics. Client Demographics Total of 28 youth and their families. One (1) youth was a readmission which results in an actual.
Cal State Northridge Psy 427 Andrew Ainsworth PhD
Research Insights from the Family Home Program: An Adaptation of the Teaching-Family Model at Boys Town Daniel L. Daly and Ronald W. Thompson EUSARF 2014/
Measuring strengths & recovery Observations on the Mental Health Center of Denver.
Hypothesis Testing Developing Null and Alternative Hypotheses Developing Null and Alternative Hypotheses Type I and Type II Errors Type I and Type II Errors.
Research Methods for Counselors COUN 597 University of Saint Joseph Class # 8 Copyright © 2015 by R. Halstead. All rights reserved.
Keys to Data-Driven Decision Making in Complex Service Systems: Availability, Reliability, & Palatability Alyssa M. Ward, Ph.D, LCP Director of Clinical.
Youth Who Thrive Developed from a review of the critical factors and effective programs for year olds Prepared for the YMCA of Greater Toronto and.
Item Analysis Ursula Waln, Director of Student Learning Assessment
PSY 307 – Statistics for the Behavioral Sciences
Comprehensive Children’s Mental Health Act
1 Behavioral Health/Juvenile Justice (BH/JJ) Evaluation Report ( ) Presented by Jeff Kretschmar, Ph.D. Project Director: Institute for the Study.
Outcomes Tool Selection Committee May 27, Introduction of Committee Children’s Behavioral Health Care Commission Terry Lawler Hot Springs School.
Children Receiving Targeted Case Management:
Developing a Resource Guide for CANS Data Analysis and Reporting Vicki Sprague Effland, Ph.D.
TAYLOR HOWARD The Employment Interview: A Review of Current Studies and Directions for Future Research.
Multiple Regression Farrokh Alemi, Ph.D. Kashif Haqqi M.D.
1 Reducing the duration and cost of assessment with the GAIN: Computer Adaptive Testing.
The Effective Management of Juvenile Sex Offenders in the Community Section 6: Reentry.
Idaho Department of Health and Welfare
11 Triple P Outcomes in California Arizona Child Trauma Summit April 9, 2013 Cricket Mitchell, PhD Senior Associate, CiMH.
1-2 Training of Process FacilitatorsTraining of Coordinators 5-1.
May 17, 2012 Electronic Information Exchange for Children in Foster Care Beth Morrow Director, Health IT Initiatives The Children’s Partnership Congressional.
Nursing Care Makes A Difference The Application of Omaha Documentation System on Clients with Mental Illness.
WRAPAROUND MILWAUKEE “Never doubt that a small group of committed citizens can change the world: indeed, it’s the only thing that ever does.” Margaret.
Michael D. Pullmann, Eric J. Bruns, Univerisity of Washington Jody Levison-Johnson, Keith Durham, Louisiana DHH – OBH EVALUATING YOUTH AND SYSTEM OUTCOMES.
Psychometrics William P. Wattles, Ph.D. Francis Marion University.
Department of Human Development (0416), Department of Psychology (0436) & Center for Gerontology (0426), Virginia Polytechnic Institute and State University,
Experimental Research Methods in Language Learning Chapter 11 Correlational Analysis.
Innovation in Monroe County Jody Levison-Johnson Director Child & Family Service Quality & System Development Coordinated Care Services, Inc. Technical.
Appraisal and Its Application to Counseling COUN 550 Saint Joseph College For Class # 3 Copyright © 2005 by R. Halstead. All rights reserved.
Managing Residential Care to Improve Permanency Outcomes Presented by: Dr. Peter Mendelson, Chief, Bureau of Behavioral Health and Medicine, DCF Lori Szczygiel,
Practice Area 1: Arrest, Identification, & Detention Practice Area 2: Decision Making Regarding Charges Practice Area 3: Case Assignment, Assessment &
Behavioral and Emotional Rating Scale - 2 Understanding and Sharing BERS-2 Information and Scoring with Parents, Caregivers and Youth May 1, 2012.
Chapin Hall Center for Children Measuring Changes in Reclaiming Futures Communities: National Evaluation Results Annual Meeting of the Coalition for Juvenile.
Introduction Results and Conclusions On counselor background variables, no differences were found between the MH and SA COSPD specialists on race/ethnicity,
Chapter 10 The t Test for Two Independent Samples
A joint Australian, State and Territory Government Initiative Experiences and lessons from benchmarking Older Persons Mental Health Services Dr Rod McKay.
Applied Quantitative Analysis and Practices LECTURE#31 By Dr. Osman Sadiq Paracha.
Early Intervention Program & Early Family Support Services: Analyzing Program Outcomes with the Omaha System of Documentation Presented to: Minnesota Omaha.
© Copyright McGraw-Hill 2004
Behavioral Health DATA BOOK A quarterly reference to community mental health and substance abuse services Fiscal Year 2015 Quarter 1 March 10, 2015
Chapter 6 - Standardized Measurement and Assessment
Georgia DFCS Outcome- Based Permanency Initiative A Proposal to Introduce Performance- Based Contracting and Partner for CFSR Success.
Department of Juvenile Justice
Portland State University
Maryland Healthy Transition Initiative
Concept of Test Validity
CANS in WISe: Empowering Youth and Family Voices
Classroom Assessment Validity And Bias in Assessment.
Human Resource Management By Dr. Debashish Sengupta
Week 3 Class Discussion.
Treating Alcohol Abuse
Eating Disorders Result in Different Eating Disorder Outcomes?
4 Domains Child Welfare, Juvenile Education and Mental/Health
A Shared Developmental Approach: Meeting Well-Being Needs and Addressing Trauma to Promote Healthy Development CLARE ANDERSON, DEPUTY COMMISSIONER ADMINISTRATION.
JAMA Pediatrics Journal Club Slides: Effect of Attendance of the Child in Childhood Obesity Treatment Boutelle KN, Rhee KE, Liang J, et al. Effect of attendance.
Performance Improvement Projects: From Idea to PIP
First 5 Sonoma County Triple P Implementation & Evaluation
IV-E Prevention Family First Implementation & Policy Work Group
Cal State Northridge Psy 427 Andrew Ainsworth PhD
Gary Morse, Ph.D. Mary York, LMSW Nathan Dell, AM, LMSW
[Group Name].
Toronto Child & Youth Advocacy Centre (CYAC)
Arely M. Hurtado1,2, Phillip D. Akutsu2, & Deanna L. Stammer1
Predicting School Difficulties for Culturally Diverse Adolescents Referred to Mental Health Treatment Deanna L. Stammer1, Phillip D. Akutsu2, & Arely.
Patient-reported Outcome Measures
Presentation transcript:

Concurrent Validity of Alternative CANS Outcome Metrics William A. Shennum Julian Leiro Delisa Young Five Acres Altadena, California

What is test validity? The concept of validity was formulated by T.L. Kelley in 1927, who stated that a test is valid if it measures what it claims to measure. For example a test of intelligence should measure intelligence and not something else, such as memory. Simplypsychology.org/validity

Concurrent validity Concurrent validity is a measure of how well a particular test correlates with a previously validated measure of the same construct. Concurrent validity relies upon measurements that take place at the same time. Explorable.com

Who we are Non profit organization serving greater Los Angeles community Research & Evaluation Department – Outcome & performance measurement – Quality & business process improvement – Program evaluation – Project management

Five Acres’ service programs

Why CANS? Need for a common outcome measure across programs for use in planning & CQI. CANS roll-out Training Inspiration – Dr. Lyons’ visit Technology Reporting

Individual Score Report (Excerpt) Client:“I” CANS Date:9/24/20132/3/2014 Life Domain Functioning IntakeDischarge 1. Family Living Situation Medical Sexuality School Achievement School Attendance 10 Youth Strengths 15. Family Interpersonal Educational 21

What About Program –Level CANS Outcomes? Several scoring options available Which ones work best? – Easily understood by stakeholders – Closely associated with other outcomes important to the organization (concurrent validity)

Study Design Three service levels were included. 1. Community based mental health 2. Wraparound 3. Residential treatment

Study Design - continued Youth intake and discharge CANS were compared (N=233) Five CANS outcome scoring metrics were computed for each youth Statistical associations of each CANS metric with concurrently measured discharge outcomes were examined.

Concurrently Measured Outcomes Service Intensity ProgramProgram Outcome/Success Measures Lower Higher Community based mental health % Service completion (EBP’s) Avg. YOQ points improved (caregiver) Wraparound % Program completion/ graduation % In community placement at discharge Residential treatment (RBS) % Program completion/ graduation % In less restrictive placement at discharge

CANS Outcome Metrics Examined in This Study

Metric 1. Intake-discharge change scores Method: Compute the average change from intake to discharge, for total score and for each domain Strength- easy to calculate Weaknesses – unit of measurement not common-sense, not tied to clients’ specific needs

Metric 1 – Intake/Discharge Change Scores, Example Youth # CANS Total Score Intake CANS Total Score DischargeChange RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ Total Score

Program Results for Metric 1 Community Based Services WraparoundResidential Treatment Avg. Change SignificanceP<.001P<.004P<.02

Metric 2 - Reliable Change Index Analysis Method: Determine whether each individual’s change score is statistically significant, by comparing it with a Reliable Change cutoff value RCI Formula: change in an individual’s score divided by the standard error of the difference for the test. Parameters in formula include Intake and Discharge scores, test standard deviations, and test (rater) reliability. Result is a yes-no score for each youth, for CANS total score, and each domain. – Strength: statistically sound – Weaknesses: difficult to compute and explain to stakeholders

How to Calculate the RCI M. Horsill, 1. Calculate standard error of measurement 2. Calculate standard error of the difference 3. Calculate RCI If RCI is 1.96 or greater, the difference is statistically significant (95% confidence level, 2-tailed) = one-tailed test cutoff

Metric 2 - Reliable Change Index Analysis, Example Youth # CANS Total Score ChangeRCI Score >=1.96? (1=yes, 0-no) RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ RBS_ % Exceed RCI Cutoff 40.0%

Program Results for Metric 2 RCI Community Based Services WraparoundResidential Treatment >= %10.2%24.2% >= %12.6%30.3%

Metric 3. Reduction in Number of Actionable Needs Method: Compare the number of actionable needs at intake and discharge, total and by domain. – Strength- addresses standout needs – Weakness- not tied to an individual’s specific needs identified at intake.

Metric 3. Reduction in Number of Actionable Needs, Example Youth # # Actionable Needs Intake # Actionable Needs DischargeChange RBS_22324 RBS_ RBS_ RBS_ RBS_ RBS_71266 RBS_ RBS_ RBS_1356 RBS_ Total Score

Program Results for Metric 3 Community Based Services WraparoundResidential Treatment Avg. Change SignificanceP<.001P<.12P<.03

Metric 4 - % Actionable Needs Resolved Method: Track specific actionable items identified at intake, and calculate the % of these that are no longer actionable at discharge, total and by domain. – Strengths - tailored to individual, easy for stakeholders to understand – Weakness – somewhat complicated scoring

Metric 4 - % Actionable Needs Resolved, Example CANS Domain: Youth Strengths IntakeDischarge Need Resolved? (1=yes, 0=no) 15. Family Interpersonal Optimism Educational Vocational UU 20. Talents/Interests Spiritual/Religious Community Life Relationship Permanence Resiliency Resourcefulness 211 % Needs Resolved 75%

Program Results for Metric 4 Community Based Services WraparoundResidential Treatment % Needs Met72.8%49.2%61.6%

Metric 5 – Actionable Needs Improved Method: Track specific actionable items identified at intake, and calculate % of these that show any improvement, even if still actionable at discharge, total and by domain. – Strength - tailored to individual, easy for stakeholders to understand – Weaknesses – includes needs still remaining at discharge, somewhat complicated scoring

Metric 5 – Actionable Needs Improved, Example Youth Behavioral/Emotional Needs IntakeDischarge Any Improvement? (1=yes, 2=no) 41. Psychosis Impulse/Hyper Depression Anxiety Oppositional Conduct Adjustment to Trauma Anger Control Substance Use 00 % Any Improvement in Actionable Needs 80%

Program Results for Metric 5 Community Based Services WraparoundResidential Treatment % Needs Improved 75.3%59.4%66.6%

Concurrent Validity of the 5 CANS Program Outcome Metrics Are the metrics statistically associated with concurrently collected measures of program success?

All CANS Metrics Significantly Associated with Concurrent Program Outcomes CANS Outcome Metrics Association w/ Concurrent Outcomes 1. Avg. Change 2. RCI Cutoff 3. Change in # Needs 4. Needs Resolve d 5. Needs Improved Median Statistical Significance p =.020 p =.043 p =.009 p =.013 p =.018

Specific CANS Domains were Positively Associated with Concurrent Outcomes CANS Outcome Metrics for Selected Domains FunctioningYouth Strength Family Needs & Strengths Behavior & Emotional Needs Median Correlation w/ Outcomes r =.23 r =.33 r =.22 r =.28

Some Differences Across Program Were Evident CANS Outcome Metrics Program / OutcomeAvg. Change RCI Cutoff Change in # Needs Needs Resolved Needs Improved Community Based Service completionp<.001p<.05p<.001p<.01p<.05 YOQ Pointsp<.05 p<.01 Wraparound Program completion p<.001 In communityp<.01nsp<.05p<.10ns Residential (RBS) Program completion p<.10p<.05 Less restrictivens p<.05p<.10

CANS outcomes for ‘successful’ youth CANS Outcome Metrics Program Avg. Change in Total Score % Youth exceeding RCI 1.65 Avg. Change in # of Actionable Needs Avg. % Actionable Needs Met Avg. % Actionable Needs Improved Community MH % %78.9% Wraparound % %79.9% Residential % %74.6% Average % %77.8%

Trends “Reduction in number of actionable needs” and “% of actionable needs resolved” metrics tended to work best for most programs “Average change score” metrics worked well in program serving less severe youth Suggestive evidence favored youth strengths domain as a driver of overall improvement CANS outcome profiles for successful youth were similar across programs – benchmarks would assist in setting performance targets.

Take Aways All CANS metrics examined showed evidence of concurrent validity across a range of programs All CANS domain improvements were related to concurrent outcomes, with suggestive evidence that “youth strengths” had strongest relationship to success. Programs serving higher severity needs may benefit from measuring metrics related to actionable need resolution and improvement. Benchmark databases could assist programs in setting outcome targets for youth.

Contact Information Bill Shennum Director of Research & Evaluation Five Acres (626)