Children’s Evaluation, Outcomes and Fidelity CMHACY Conference 2007 Todd Sosna, Ph.D.

Slides:



Advertisements
Similar presentations
Integrating the NASP Practice Model Into Presentations: Resource Slides Referencing the NASP Practice Model in professional development presentations helps.
Advertisements

ORCHARD ACADEMY Best Practices in Person-Centered Transition Planning and Programming that Deliver Positive Outcomes Presented by… Tim Bobrowski, Director.
Making a Difference Improving the Quality of Life of Individuals with Developmental Disabilities and their families.
Customised training: Learner Voice and Post-16 Citizenship.
LIFE, LEARNING AND ACHIEVEMENT
Educational Specialists Performance Evaluation System
Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
From QA to QI: The Kentucky Journey. In the beginning, we were alone and compliance reigned.
Research Insights from the Family Home Program: An Adaptation of the Teaching-Family Model at Boys Town Daniel L. Daly and Ronald W. Thompson EUSARF 2014/
Representing Central Government in the South East Monday, 27 April 2015 Vivien Lines DCSF Safeguarding Adviser VCS Safeguarding Seminar 17 December 2009.
Each Year, nationwide, 1.2 million students fail to graduate from high school!
Planning an improved prevention response up to early childhood Ms. Giovanna Campello UNODC Prevention, Treatment and Rehabilitation Section.
Our Mission Community Outreach for Youth & Family Services, Inc. is dedicated to improving the quality of life for both the youth and adult population.
Healthy Child Development Suggestions for Submitting a Strong Proposal.
Presented at Annual Conference of the American Evaluation Association Anaheim, CA, November 2011 Lessons Learned about How to Support Outcomes Measurement.
Katie A: From Compliance to Transformation CMHACY: May 14, 2015 Sylvia Deporto, MS Lynn Dolce, MFT Bonnie Friedman, LCSW Jennifer Clancy, MSW.
1 Minority SA/HIV Initiative MAI Training SPF Step 3 – Planning Presented By: Tracy Johnson, CSAP’s Central CAPT Janer Hernandez, CSAP’s Northeast CAPT.
Presented By: Tracy Johnson, Central CAPT
Goal 3, Volunteer Development and Systems to Support Youth: Logic Model and Communications Plan Situation Statement During 2005, over 11,218 adult volunteers.
Community Planning Training 1-1. Community Plan Implementation Training 1- Community Planning Training 1-3.
Early Childhood Mental Health Consultants Early Childhood Consultation Partnership® Funded and Supported by Connecticut’s Department of Children and Families.
VISITATION 1. Competencies  SW Ability to complete visitation plans that underscore the importance of arranging and maintaining immediate, frequent,
Pennsylvania Child Protective Services Law: Module 4: Reporting and the Role of the Child Welfare Professional Transfer of Learning The Pennsylvania Child.
GSU-NACDD-CDC Chronic Disease and Public Health Workforce Training Training Needs Survey and Public Health Certificate in Chronic Disease Training for.
IFAS Extension Goal 3, Logic Model and Communications Plan Life Skills Developed in Youth Through Subject Matter Experiences Situation Statement Florida.
The Role of Collaboration in Improving Children’s Mental Health Services Mary I. Armstrong, Ph.D. Presented at the 2007 National Association of Mental.
Our three year strategy >Our vision >Children and young people in families and communities where they can be safe, strong and thrive. >Our mission >Embed.
Prevention and Early Intervention Programme Presentation to the Trinity College Summer School, August 2012.
Interstate New Teacher Assessment and Support Consortium (INTASC)
May 20, Purpose of the Self- Assessment Required by the Head Start Performance Standards (i)(1) Head Start Ac 2007 Head Start Act Section.
Implementation Strategy for Evidence- Based Practices CIMH Community Development Team Model Pam Hawkins, Senior Associate Association for Criminal Justice.
Bringing Protective Factors to Life in the Child Welfare System New Hampshire.
Population Parameters  Youth in Contact with the Juvenile Justice System About 2.1 million youth under 18 were arrested in 2008 Over 600,000 youth a year.
KENTUCKY YOUTH FIRST Grant Period August July
Bethany Geldmaker RN, PNP, PhD
Chapter 10 Counseling At Risk Children and Adolescents.
Measuring and Improving Practice and Results Practice and Results 2006 CSR Baseline Results Measuring and Improving Practice and Results Practice and Results.
Ingham Healthy Families. History: Why Healthy Families America? Michigan Home Visiting Initiative Exploration & Planning Tool (Fall 2013)  Ingham County.
Understanding TASC Marc Harrington, LPC, LCASI Case Developer Region 4 TASC Robin Cuellar, CCJP, CSAC Buncombe County.
Components of a national drug prevention system Ms. UNODC.
Practice Model Elements Theoretical framework Values and principles Casework components Practice elements Practice behaviors.
Family Advocates Home Visiting Program. Mission Strengthen families and communities so they can be safe, healthy and thriving. We do this by providing.
6 Key Priorities A “scorecard” for each of the 5 above priorities with end of 2009 deliverables – with a space beside each for a check mark (i.e. complete)
Child/Youth Care Management 2015 training. WELCOME!
Practice Area 1: Arrest, Identification, & Detention Practice Area 2: Decision Making Regarding Charges Practice Area 3: Case Assignment, Assessment &
Children’s Evaluation: Outcomes and Fidelity Full Service Partnerships January , 2007.
Basic Nursing: Foundations of Skills & Concepts Chapter 9
State of California Department of Alcohol and Drug Programs State Incentive Grant Project Overview Michael Cunningham Deputy Director, Program Services.
Family Care Community Partnerships (FCCP) Selected Logic Model Outcomes in the System of Care CY14 3 rd and 4 th Quarters Rhode Island Department of Children,
Project KEEP: San Diego 1. Evidenced Based Practice  Best Research Evidence  Best Clinical Experience  Consistent with Family/Client Values  “The.
FOSTER CARE: MODULE #4 Meeting the Needs of Children and Families.
Surveillance Evaluation Assist Prof Dr. Kwankate Kanistanon, DVM, MS, PhD.
California Implementation of Functional Family Therapy (FFT) California Institute for Mental Health (CiMH) Todd Sosna Ph.D., Senior Associate, Center for.
Community Planning 101 Disability Preparedness Summit Nebraska Volunteer Service Commission Laurie Barger Sutter November 5, 2007.
Monitoring MCHB’s Six Core Outcomes for CSHCN Paul Newacheck, DrPH MCH Policy Research Center.
M eaningful Quality Measures for Children with Behavioral Health Conditions Discussion with the NYS Conference of Local Mental Health Hygiene Directors.
Senate Select Committee on Aging and Long Term Care Final Report and Progress in 2015 Suzanne Reed, Chief of Staff Senator Carol Liu (Chair)
Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center
1-2 Training of Process Facilitators Training of Process Facilitators To learn how to explain the Communities That Care process and the research.
About District Accreditation Mrs. Sanchez & Mrs. Bethell Rickards Middle School
1 Module 1 Introduction: The Role of Gender in Monitoring and Evaluation.
Training of Process Facilitators 1- Training of Process Facilitators 5-1.
Legislative Enhancements to Behavioral Health. Recent Legislation Behavioral Health Enhancements HB 7019/SB 7068 (2015) SB 12/HB 7097 (2016) Housing Assistance.
What Is Child Find? IDEA requires that all children with disabilities (birth through twenty-one) residing in the state, including children with disabilities.
Standards and Competencies for Cancer Chemotherapy Nursing Practice in Canada: CANO/ACIO AN INTRODUCTION.
The PDA Center is funded by the US Department of Education Office of Special Education Programs Stories from the Field and from our Consumers Building.
KITS V JUNE , 2014 BREAKING DOWN AND UNDERSTANDING THE PSYCHOLOGICAL : WHAT YOU DON’T KNOW CAN HURT YOU M. Connie Almeida, PhD, LSSP, Licensed Psychologist.
1 This project was supported by the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services (HHS) under.
NYU Child Study Center: Bridges Program Caring Across Communities: Annual Grantee Meeting April
Presented at Annual Conference of the American Evaluation Association Anaheim, CA, November 2011 Lessons Learned about How to Support Outcomes Measurement.
Presentation transcript:

Children’s Evaluation, Outcomes and Fidelity CMHACY Conference 2007 Todd Sosna, Ph.D.

CMHACY 2007Outcome Evaluation Model2 Topics Fidelity monitoring-outcome evaluation model Measuring fidelity Measuring practice-level outcomes Measuring community-level outcomes Target population characteristics Evaluating your programs

CMHACY 2007Outcome Evaluation Model3 Optimal Service System Full (easy) access to comprehensive (any and all) individualized Effective services Provided with high quality

CMHACY 2007Outcome Evaluation Model4 Levels of Responsibility Agency is responsible for delivery of high quality service (e.g. hiring, training, supervision) Practice is responsible for promoting early stages of positive outcomes (e.g. engagement, skills development, positive attributions, improved mood) Public service system is responsible for promoting access to comprehensive, coordinated services and supports Community is responsible for the establishing priorities and resources across systems

CMHACY 2007Outcome Evaluation Model5 Accountability Fidelity is closely related to the capabilities of the provider agency Practice specific outcomes are closely related to the effectiveness of the practice (assumes that it is provided with strong fidelity) System level outcomes are closely related to the responsiveness of the system (assumes that individual practices are effective) Community level outcomes are closely related to the responsiveness of policy makers, advocates, citizens (assumes that service systems are accessible and collaborative)

CMHACY 2007Outcome Evaluation Model6 Critical Evaluation Questions Are programs/practices provided with model adherence (fidelity)? Does each individual program/practice result in the level of expected child and family improvement (practice specific outcomes)? Does the system insure access to needed services and supports (community level outcomes)? Does the community insure that services systems are available, funded, and coordinated?

CMHACY 2007Outcome Evaluation Model7 Interpretation Low fidelity The agency has not provided the level of expected quality Low practice outcomes If fidelity is high, then the practice has not been effective If fidelity is low, the effectiveness of the practice cannot be determined

CMHACY 2007Outcome Evaluation Model8 Interpretation Low system-level outcomes If practices are effective, then the system has failed to identify needs and make services accessible If practice outcomes are low, the effectiveness of the system cannot be determined Low community-level outcomes If the service system is insuring access to services and supports, then the community has failed to establish priorities, programs and funding If the system-level outcomes are low, the effectiveness of the community cannot be determined

CMHACY 2007Outcome Evaluation Model9 Measures Fidelity tools are unique to each practice. No two practices use the same measure of fidelity Practice specific outcome measures need to be sensitive to change that is realistically expected to occur concurrent with or immediately after a course of treatment Practices that promote achievement of the same goals can be evaluated using the same measures System and community level outcome measures need to be sensitive to long term functional achievement, and could consist of one set of measures system-wide and/or community-wide

CMHACY 2007Outcome Evaluation Model10 Measures Fidelity measures are collected on an ongoing or periodic basis (fidelity can wax and wane) Practice specific outcomes are collected pre- and post System level outcomes are collected when an individual first receives any service from the service system (baseline) and then collected on a fixed interval (e.g. annually) while they are receiving one or more services (and afterwards for a follow-up evaluation) Community level outcomes are collected on a fixed interval independent of individual consumers starting or ending services

CMHACY 2007Outcome Evaluation Model11 Fidelity Measures Adherence to key structural components Staff hiring, roles and responsibilities Completion of specific training Demonstration of key (delineated) processes Live observation Video or audio observation Self rating Youth/caregiver interviews Achievement of key milestones Retention rates Attainment of treatment phases or goals Graduation rates

CMHACY 2007Outcome Evaluation Model12 Practice Specific Measures Age 0-5 Elementary School Junior & High School Adult Older Adult Disorder Externalizing Depression Trauma Other anxiety Bipolar Early psychosis Consumer rating Caregiver rating Practitioner rating Behavioral indicator

CMHACY 2007Outcome Evaluation Model13 System & Community Level Measures Physically health/access to health care Safe and nurturing family/home Pro-social friendships Happy Success in school/work Law abiding

CMHACY 2007Outcome Evaluation Model14 Population Characteristics Consumer attributes or historical events, for example-- Age Gender Ethnicity History of neglect, abuse, placement or law violations Population characteristics can be mediating factors Does (1) access to care, (2) completion of services, or (3) outcomes vary as a function of these characteristics?

CMHACY 2007Outcome Evaluation Model15 Evaluating Your Programs Simply start simply List one of your programs Describe its distinguishing features? What is the target population? What outcomes are you trying to achieve? Pick 4 population characteristics Select a single fidelity monitoring method Select 2 practice level outcomes (1 age- disorder rating measure, 1 behavioral indicator)

CMHACY 2007Outcome Evaluation Model16 Evaluating Your Programs Involve/orient practitioners to the fidelity monitoring and outcome evaluation protocol--why and how What data is already being collected? Add new measures to intake procedure Use a simple (Excel) database Have all completed measures sent to a designated individual for entry Create simple graphs for reporting

CMHACY 2007Outcome Evaluation Model17 Evaluating Your Programs Simple reports disseminated soon Characteristics of youth Average fidelity ratings Average pre- post- changes Relationships between youth characteristics and changes Age, gender, or ethnicity and outcomes Relationships between fidelity ratings and changes