Documenting quality in complex residential child care systems: Challenges with assessing fidelity to a principle-based program model Lisa McCabe and Michael Nunno Cornell University NHSTES Berkeley, CA May

Documenting quality: Challenges with assessing fidelity to a principle-based program model How do you create an instrument that measures adherence to principles? – Not measuring specific skills – Principles provide boundaries and direction – Many right ways

Active Ingredients Kemp, 2016

Intervention Fidelity: What is it? Extent to which delivery of a program is consistent with intended program model. – Adherence (elements are present) – Dosage (exposure, frequency) – Quality (how close use resembles ideal) – Differentiation (different than business as usual)

Why Measure Fidelity? 1.Assist programs to self assess and implement well 2.Identify common deviations in program delivery 3.Assess how program components relate to outcomes

Steps for Assessing Fidelity 1.Criteria based on program theory 2.Criteria operationally defined 3.Development of instruments 4.Test and report reliability and validity

Holden, 2009

What is CARE? Program model implemented in residential agencies (and beyond) – Setting level – Congruence throughout agency Research-informed Principle-based

CARE Principles Relationship based Trauma informed Developmentally focused Family involved Competence centered Ecologically oriented

How do we CARE? Personnel Education Exposure to concepts and principles Practice recommendations Modeling and role playing strategies and skills Consultation about implementation Data based feedback and reflection

CARE Children And Residential Experiences: Theory of Change Personnel Education Exposure to concepts & principles Practice recommend- ations Modeling and role- play of strategies and skills Intervention CARE- Consistent Practices Staff Outcomes Childcare beliefs Understand principles Motivation to apply principles & strategies Youth Experience Quality Interaction With Staff Child Outcomes Policies supporting CARE Imp Climate Culture Congruence Data-based decision-making Organizational Outcomes Consultation about implementation Data-based feedback & Reflection Youth Behavior Social & emotional Well-being

Steps for Assessing Fidelity 1.Criteria based on program theory 2.Criteria operationally defined 3.Development of instruments 4.Test and report reliability and validity

Structure – Training – Resources for implementation – Forums/workgroups for planning, review, reflection – Policies, written materials reflect CARE – Data gathered to support implementation Process – Application of CARE principles in: Staff-youth Interactions Supervisor-staff Interactions Staff-family interactions Active Ingredients

Steps for Assessing Fidelity 1.Criteria based on program theory 2.Criteria operationally defined 3.Development of instruments 4.Test and report reliability and validity

Active Ingredients: Structure Supervisors are trained in the CARE model No superviso rs trained in CARE model Planning underway to train superviso rs in CARE model Some (<50%) supervis ors trained in CARE model Most (51%- 89%) supervis ors trained in CARE model All (90% or more) supervis ors trained in CARE model 1 Not Attempted/ Not Yet/ Inadequate 2 Planning 3 In Progress: Partial 4 In Progress: Consistent 5 Fully Met: Exemplary

Active Ingredients: Process How do you create an instrument that measures adherence to principles? 2. Criteria operationally defined 3. Development of instruments

CARE principles in action: Relationship based Observable behaviors or actions of staff that illustrate positive relational experiences? Observable behaviors, actions of youth that illustrate positive relational experiences? Measurable youth outcomes that illustrate positive relational experiences?

Reading list

For More Information Lisa McCabe Michael Nunno Martha Holden 20