1. No Place Like Home Texas Region 3 Evaluation Training

2. General Staff Survey Please open your browser to: https://www.surveymonkey.com/s/TX3STAFF Read the informational page, and if you agree to take the survey, please click “Next” to begin All of your responses are CONFIDENTIAL It should take approximately 15 minutes to complete

3. NPLH Summary Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services –Process What does practice look like? Tools: focus groups, observations, surveys –Outcomes Child welfare placements (number and type), re- reports, recurrence of maltreatment, etc. Family functioning and protective factors Satisfaction and service provision Tools: administrative data, surveys

4. Your role… Is critical! To provide data and feedback about your experience working in child welfare and with family meetings –As part of a federally-funded study that will be disseminated nationally To recruit families to participate, so that they can provide data and feedback about their experience To participate in the feedback loop – ongoing communication

5. WhoWhat is the value? …the fieldResearch is mixed and the evidence-base of FGDM is moderate—more research is needed Additional research can support the national spread of family meetings or suggest the need for alternative practices …familiesFamilies receive interventions with known effectiveness Opportunity for families to provide input about their experiences with CPS What’s in it for…?

6. WhoWhat is the value? …CPS Staff and Agencies (You!) Accountability. Opportunity to get input from staff who know the most about the families (You!) Knowledge of effectiveness of family meetings can increase investment in the practice Visibility and respect for advancing the field and systematically examining internal practices Increased knowledge about the use and potential of evaluation Results can impact system and practice change What’s in it for…?

7. Human Subjects Overview –History of human subjects research –Historical laws and documents protecting human subjects –Your role in protecting the rights of participants*

8. Human Subjects History Before there were human subjects protections: Nuremberg (1930’s – 1940’s) –Concentration camp prisoners, twins Willowbrook Hepatitis Study (1956) –Mentally retarded, institutionalized children Brooklyn Jewish Chronic Disease Cancer Study (1963) –Poor, eldery San Antonio Contraception Study (1971) –Mostly poor, women Tuskegee Syphilis Study (1932-1971) –Poor, black males

9. Human Subjects History Attempts to protect human subjects: The Nuremberg Code (1947) –Informed consent. Benefits outweigh risks. Voluntary participation. Declaration of Helsinki (1964…2000) –“Concern for the interests of the subject must always prevail over the interests of science & society” National Research Act (1974) –Tuskegee  Institutional Review Board (IRB) approval required if getting U.S. Dept. of Health and Human Services funds The Belmont Report (1979) –“Justice, beneficence, respect for persons”

10. Our Role Our Role in Protecting Participants Rights Give participants all the information that they may need to make an informed decision. Ensure that participants understand the information Give participants an opportunity to consider involvement in the project on an initial and ongoing basis (can stop at any time).

11. News You Can Use What’s important to know: –Risks to you personally are minimal –There are potential benefits to children, families, and to staff in the agency –Your participation is voluntary Privacy and Confidentiality –All survey information is confidential and used only for the study –Survey data is maintained by Kempe Center NPLH evaluation team –Individually identifiable information will NOT be shared outside of the Kempe Center evaluation team

12. Evaluation Terms Evidence-Based Practice Confidentiality vs. Anonymity Experimental research design –Random assignment Bias Fidelity Validity Process vs. Outcome evaluation –How vs. What Qualitative vs. Quantitative methods –Depth vs. Breadth

13. Evaluation Design - TX General Staff Survey Families meet eligibility criteria and notified about the study; Parent Survey Random Group Assignment Meeting participants notified about the study; Fidelity Survey Follow Up Survey Case-Specific Questionnaire Right now FBSS worker makes FGC referral* FGC 1-2 months post-FGC FBSS Case closure/transfer to CVS *Your unit becomes eligible for a monthly incentive based on FGC referrals.

14. NPLH Surveys 1.General Staff Survey 2.Parent Survey 3.Meeting Fidelity Survey a)Participant b)Facilitator/Coordinator 4.Case-Specific Questionnaire 5.Follow-Up Survey a)Parent – Intervention b)Parent – Control c)Meeting Participant

15. Surveys that are distributed TO you General Staff Survey –All staff who have a role in the evaluation (case workers, coordinators/facilitators, supervisors) Coordinator-Facilitator Meeting Fidelity Survey –Coordinators/Facilitators only Case-Specific Questionnaire –Case-workers only

16. General Staff Survey Purpose: –To understand more about: Who utilizes family meetings in CPS How much experience staff have with family meetings How staff perceive family meetings The context in which meetings are happening. –We can use this information to inform our process and outcome evaluations

17. General Staff Survey Who? You! –FBSS Case workers providing in-home services –Family meeting facilitators/coordinators –Supervisors of both When? Now How? Via Survey Monkey

18. Coordinator-Facilitator Fidelity Survey Purpose: –To understand more about: How, why, and when family meetings are used Who is invited to and who attends family meetings Other characteristics of family meetings –Fidelity –Logistics –Meeting fidelity is part of our process evaluation

19. Coordinator-Facilitator Fidelity Survey Who? Meeting coordinator/facilitator When? After the Family Group Conference (FGC) –Once back at your desk/computer (same day or day after) How? Via Survey Monkey

20. Case-Specific Questionnaire Purpose: –To understand more about: What services in-home families receive Are services provided impacted by family meetings? –Services provided is one of our outcome evaluation variables

21. Case-Specific Questionnaire Who? Case workers When? At case closure OR transfer from in-home services to out-of-home services (i.e. from FBSS to CVS) –Transfer does NOT mean voluntary placements –It does mean a formal or legal change in custody How? Via Survey Monkey –We will send you a ‘tickler’ or reminder

22. Surveys that are distributed BY you Parent Survey –By FBSS workers –At face-to-face meetings with family –At point of referral for an FGC Participant Fidelity Survey –By coordinators/facilitators –At FGCs

23. What you ARE responsible for To distribute: 1.Information Sheet Explains project to participants. Consent is not required but participation is voluntary. What is voluntary? Completion of surveys NOT participation in child welfare services. 2.Distribute Paper and pencil surveys with: –Parent Survey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance –Meeting Participant – large self-addressed, stamped envelope for all surveys in the center of the room/table To pre-fill: -Case ID and FBSS Stage ID fields before distributing Caregiver Surveys or Participant Fidelity Surveys to families/meeting participants.

24. What you are NOT responsible for Explaining the content of surveys to families Helping families/meeting participants fill out surveys People with questions can call the number provided on the Information Sheet (have extra copies available for them to keep) Other household members or family supports, etc. can help. In order to maintain confidentiality, it is important that you merely handout surveys and provide no more information than what is included on the information sheets.

25. Parent Survey Purpose: –To understand more about: The characteristics of families receiving in-home child welfare services (e.g. demographics) The protective capacities of parents The parent-child relationship and child behavior –This information can be linked to outcomes

26. Parent Survey Who? Parents or legal guardians When? –Face-to-face meeting –At time of referral for FGC How? –Paper and pencil –Self-addressed and stamped envelope –Distributed by FBSS worker –Parents can call the number on the information sheet to take the survey via phone

27. Participant Fidelity Survey Purpose: –To understand more about: How, why, and when family meetings are used Who is invited to and who attends family meetings Other characteristics of family meetings –Fidelity –Logistics –Meeting fidelity is part of our process evaluation –We want to hear from families and other participants as well as staff.

28. Participant Fidelity Survey Who? Any meeting participant who agrees to fill it out When? AT the family meeting –Section 1: Before the meeting starts –Section 2: After the meeting ends How? Paper and Pencil –Meeting participants will fill out paper-pencil surveys and place them in a large envelope –The coordinator/facilitator will collect them in a safe spot to be mailed to the evaluation team

29. Surveys administered by evaluation team How? Paper and pencil; via mail 3 follow-up survey versions with the following components: –Meeting Follow-Up Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey –Parent Follow-Up Who? All (intervention and control group) parents –Family Satisfaction Who? All (intervention and control group) parents AND meeting participants

30. Coordinators/Facilitators Meeting Log Facilitators: –You will be asked to fill out information an additional spreadsheet (not unlike what you may be doing now) Ex. Date of referral, meeting type, date of meeting, meeting cancellation reason, etc. –This will be posted on a document sharing website –We will send the link to you before data collection begins

31. NPLH Survey Summary Case workers: –Complete: General Staff Survey (1 time) Case Specific-Questionnaire (for every study case) –Distribute: Parent Survey (to every study case) Facilitators: –Complete: General Staff Survey (1 time) Facilitator Fidelity Survey (for every study family meeting) Complete meeting log (for every study family meeting) –Distribute: Meeting Participant Fidelity Survey (for every study family meeting)

32. Where can I find survey materials? Margie Casey, Administrative Assistant 1200 E. Copeland Rd., Suite 400 M/C 013-8 –You can find extra survey materials here Parent surveys with self-addressed, stamped envelopes Extra information sheets Meeting Participant fidelity surveys and envelopes –Facilitators can drop off survey packets here after meetings for bulk mailing at a later date –When you are running low on materials, we will deliver more

33. When can I find them? When do we start? October 29th!

34. THANK YOU! If, throughout the life of this project, you have any questions please contact: Heather Allan, NPLH Project Coordinator Email: heather.allan@childrenscolorado.orgheather.allan@childrenscolorado.org Phone: 720.336.8283