1. Documenting quality in complex residential child care systems: Challenges with assessing fidelity to a principle-based program model Lisa McCabe and Michael Nunno Cornell University NHSTES Berkeley, CA May 2016 1

2. Documenting quality: Challenges with assessing fidelity to a principle-based program model How do you create an instrument that measures adherence to principles? – Not measuring specific skills – Principles provide boundaries and direction – Many right ways

3. Active Ingredients Kemp, 2016

5. Intervention Fidelity: What is it? Extent to which delivery of a program is consistent with intended program model. – Adherence (elements are present) – Dosage (exposure, frequency) – Quality (how close use resembles ideal) – Differentiation (different than business as usual)

6. Why Measure Fidelity? 1.Assist programs to self assess and implement well 2.Identify common deviations in program delivery 3.Assess how program components relate to outcomes

7. Steps for Assessing Fidelity 1.Criteria based on program theory 2.Criteria operationally defined 3.Development of instruments 4.Test and report reliability and validity

8. Holden, 2009

9. What is CARE? Program model implemented in residential agencies (and beyond) – Setting level – Congruence throughout agency Research-informed Principle-based

10. CARE Principles Relationship based Trauma informed Developmentally focused Family involved Competence centered Ecologically oriented

11. How do we CARE? Personnel Education Exposure to concepts and principles Practice recommendations Modeling and role playing strategies and skills Consultation about implementation Data based feedback and reflection

12. CARE Children And Residential Experiences: Theory of Change Personnel Education Exposure to concepts & principles Practice recommend- ations Modeling and role- play of strategies and skills Intervention CARE- Consistent Practices Staff Outcomes Childcare beliefs Understand principles Motivation to apply principles & strategies Youth Experience Quality Interaction With Staff Child Outcomes Policies supporting CARE Imp Climate Culture Congruence Data-based decision-making Organizational Outcomes Consultation about implementation Data-based feedback & Reflection Youth Behavior Social & emotional Well-being

13. Steps for Assessing Fidelity 1.Criteria based on program theory 2.Criteria operationally defined 3.Development of instruments 4.Test and report reliability and validity

14. Structure – Training – Resources for implementation – Forums/workgroups for planning, review, reflection – Policies, written materials reflect CARE – Data gathered to support implementation Process – Application of CARE principles in: Staff-youth Interactions Supervisor-staff Interactions Staff-family interactions Active Ingredients

15. Steps for Assessing Fidelity 1.Criteria based on program theory 2.Criteria operationally defined 3.Development of instruments 4.Test and report reliability and validity

16. Active Ingredients: Structure Supervisors are trained in the CARE model No superviso rs trained in CARE model Planning underway to train superviso rs in CARE model Some (<50%) supervis ors trained in CARE model Most (51%- 89%) supervis ors trained in CARE model All (90% or more) supervis ors trained in CARE model 1 Not Attempted/ Not Yet/ Inadequate 2 Planning 3 In Progress: Partial 4 In Progress: Consistent 5 Fully Met: Exemplary

17. Active Ingredients: Process How do you create an instrument that measures adherence to principles? 2. Criteria operationally defined 3. Development of instruments

18. CARE principles in action: Relationship based Observable behaviors or actions of staff that illustrate positive relational experiences? Observable behaviors, actions of youth that illustrate positive relational experiences? Measurable youth outcomes that illustrate positive relational experiences?

19. Reading list

20. For More Information Lisa McCabe (LAM4@cornell.edu) Michael Nunno (man2@cornell.edu) Martha Holden (mjh19@cornell.edu) 20