1. No Place Like Home Cross-Site Evaluation Training

2. General Staff Survey

3. NPLH Summary Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services –Process What does practice look like? Tools: focus groups, observations, surveys –Outcomes Placements (number and type), re-reports, recurrence of maltreatment, etc. Satisfaction and service provision Tools: administrative data, surveys

4. Your role… Is critical! To provide data and feedback about your experience working in child welfare and with family meetings To recruit families to participate, so that they can provide data and feedback about their experience To participate in the feedback loop – ongoing communication

5. WhoWhat is the value? …the fieldResearch is mixed and evidence-base of FGDM is moderate— more research is needed Additional research can support the national spread of family meetings or suggest the need for alternative practices …FamiliesFamilies receive interventions with known effectiveness Opportunity for families to provide input about their experiences with CPS …CPS Staff and Agencies (You!) Accountability. Opportunity to get input from staff who know the most about the families (You!) Knowledge of effectiveness of family meetings can increase investment in the practice Visibility and respect for advancing the field and systematically examining internal practices Increased knowledge about the use and potential of evaluation Results can impact system and practice change What’s in it for…?

6. Human Subjects What is the CITI program? History and Ethical Principles The Belmont Report

7. Human Subjects Defining Research with Human Subjects Federal Regulations Assessing Risks Basic Institutional Review Board (IRB) Regulations and Review Process

8. Human Subjects Informed Consent –Risks to you personally are minimal –There are potential benefits to children, families, and to staff in the agency –Your participation is voluntary Privacy and Confidentiality –All survey information is confidential and used only for the study –Survey data is maintained by Kempe Center NPLH evaluation team –Individually identifiable information will NOT be shared outside of the Kempe Center evaluation team

9. Human Subjects Records-Based Research Research with Protected Populations – Vulnerable Subjects Unanticipated Problems and Reporting Requirements

10. Evaluation Basics Evidence-Based Practice Confidentiality vs. Anonymity Bias Design Types –Experimental –Quasi-Experimental Process vs. Outcome evaluation –How vs. What Qualitative vs. Quantitative methods –Depth vs. Breadth

11. Process Evaluation Was the program implemented as planned? Did the program meet its process objectives? What type and amount of services were provided? Who was served? Did the program meet its objectives and goals? Did the program make a difference? Generally, process evaluations use qualitative research methods like interviews and focus groups. Generally, outcome evaluations use quantitative research methods and designs to establish causality (i.e., experimental* and quasi-experimental) Can the program be causally linked to outcomes? Typically, the best research designs combine both process and outcome evaluations and rely on mixed methods. A cost analysis links benefits to costs for comparison of effectiveness in policy-relevant terms. Example evaluation questions from: Handler, A. (2002). Role of evaluation in policy development and implementation. Retrieved from http://www.uic.edu/sph/mch/evaluation/class.ppt Outcome Evaluation Process vs. Outcome Evaluation

12. True Experimental Design

13. Evaluation Design - TX Staff Consent; General Staff Survey Families meet eligibility criteria and are consented into the study; Caregiver Survey Random Group Assignment Meeting participants consented into the study; Fidelity Survey Follow Up Survey Needs-Services Assessment; Case- Specific Questionnaire Right now FBSS worker makes FGC referral FGC 1-2 months post-FGC FBSS Case closure/transfer to CVS

14. Evaluation Design - SD Staff Consent; General Staff Survey Families meet eligibility criteria and are consented into the study; Caregiver Survey Random Group Assignment Meeting participants consented into the study; Fidelity Survey Follow Up Survey Needs-Services Assessment; Case- Specific Questionnaire Right now IFA worker makes FGC or FGC/TDM referral FGC/TDM; IFA and Ongoing worker attend 1-2 months post-FGC Ongoing case closure/transfer to out-of-home services

15. Evaluation Design - CO Staff Consent; General Staff Survey Families meet eligibility criteria and are consented into the study; Caregiver Survey Group Assignment – Propensity Score Match Meeting participants consented into the study; Fidelity Survey Follow Up Survey Needs-Services Assessment; Case- Specific Questionnaire Right now Intake or FAR worker makes 1 st FSRT or FUM referral 2 nd FUM/FGC 1-2 months post- FUM/FGC Ongoing case closure/transfer to out-of-home services

16. NPLH Instrumentation 1.General Staff Survey 2.Caregiver Survey 3.Meeting Fidelity Survey a)Participant b)Facilitator/Coordinator 4.Case-Specific Questionnaire 5.Follow-Up Survey a)Caregiver – Intervention b)Caregiver – Control c)Meeting Participant

17. Surveys that are administered TO you General Staff Survey –All staff who have a role in the evaluation (case workers, coordinators/facilitators, supervisors) Coordinator-Facilitator Survey –Coordinators/Facilitators only Case-Specific Questionnaire –Case-workers only

18. General Staff Survey Survey ComponentRationale Demographics-Understand who works in CPS and who uses family meetings Job Satisfaction-Understand how family meeting practice impacts job satisfaction Job Skills-Identify training and technical assistance needs Organizational Culture and Climate -Understand the context in which family meetings are occurring Services-Understand what services are available in the community -Identify barriers to service provision Values-Learn about the underlying values of staff using family meetings Beliefs about Family Meetings -Identify barriers and facilitators to family meeting implementation -Understand what works and what doesn’t in family meeting practice

19. General Staff Survey

20. Coordinator-Facilitator Fidelity Survey Survey ComponentRationale Contact-Prepared- Attended Checklist -To understand who is invited to and who attends family meetings About the Meeting-To understand meeting logistics, purpose, characteristics Pre-Meeting-To understand meeting preparation activities At-Meeting-To understand meeting characteristics and activities

21. Coordinator-Facilitator Fidelity Survey

22. Survey ComponentRationale Length of time on caseContinuity of service; knowledge of family Service Characteristics: family need services received service intensity perceived improvement Family meetings are not the only intervention—the service package is. Understanding whether services received affect outcomes, perceptions or measured with existing data. Understanding whether family meetings influences the type or level of services relative to the families’ service needs. Understanding the type of families (based on service needs) that family meetings are more or less effective with. Informal and Community Supports Understanding the role of different type of social support in influencing outcomes Do family meetings affect the amount or type of support? Primary Caregiver AttitudesDo family meetings affect attitudes of caregivers to working with Child Protective Services? Does caregiver attitude affect outcomes? Case-Specific Questionnaire

24. Surveys that are administered BY you Caregiver Survey –By case workers –At face-to-face meetings with family –At point of referral for a family meeting Participant Fidelity Survey –By coordinators/facilitators –At family meetings

25. What you ARE responsible for 1.Informed Consent Will families agree to be part of the pilot? If so, they have rights which are outlined in the Information Sheet, including the right to change their mind at any time. If not, no penalty will come to them. 2.Distribute Paper and pencil surveys with Caregiver Survey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance Meeting Participant – large self-addressed, stamped envelope for all surveys in the center of the room/table

26. What you are NOT responsible for Explaining the content of surveys to families Helping families/meeting participants fill out surveys People with questions can call the number provided on the Information Sheet (have extra copies available for them to keep)

28. Caregiver Survey

29. Participant Fidelity Survey Survey ComponentRationale Demographics-To understand who attends family meetings Pre-Meeting-To understand family meeting preparation activities Post-Meeting-To understand family meeting characteristics and activities

30. Participant Fidelity Survey Who? Any meeting participant who agrees to fill it out When? AT the family meeting –Section 1: Before the meeting starts –Section 2: After the meeting ends How? Paper and Pencil –Meeting participants will fill out paper-pencil surveys and place them in a large envelope –The coordinator/facilitator will mail them to the evaluation team

31. Surveys administered by evaluation team How? Paper and pencil; via mail 3 follow-up survey versions with the following components: –Meeting Follow-Up Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey –Caregiver Follow-Up Who? All (intervention and control group) parents –Family Satisfaction Who? All (intervention and control group) parents AND meeting participants

32. NPLH Survey Summary Case workers: –Complete: General Staff Survey (1 time) Case Specific-Questionnaire (for every study case) –Distribute: Caregiver Survey (to every study case) Coordinators/Facilitators: –Complete: General Staff Survey (1 time) Coordinator-Facilitator Fidelity Survey (for every study family meeting) –Distribute: Meeting Participant Fidelity Survey (for every study family meeting)

33. Coordinators/Facilitators Meeting Log [placeholder]

34. THANK YOU! If, throughout the life of this project, you have any questions please contact: Heather Allan, NPLH Project Coordinator Email: heather.allan@childrenscolorado.orgheather.allan@childrenscolorado.org Phone: 720.336.8283