1. Texas Region 3 Evaluation Training
No Place Like Home
Texas Region 3 Evaluation Training

2. General Staff Survey Please open [enter link] on your computers
Read the informational page, and if you agree to take the survey, please click “Next” to begin
It should take approximately 15 minutes to complete
General Staff Survey

3. Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services
Process
What does practice look like?
Tools: focus groups, observations, surveys
Outcomes
Child welfare placements (number and type), re-reports, recurrence of maltreatment, etc.
Family functioning and protective factors
Satisfaction and service provision
Tools: administrative data, surveys
Multiple models of meetings, use and timing of which differs within and across sites
NPLH Summary

4. Is critical!
To provide data and feedback about your experience working in child welfare and with family meetings
As part of a federally-funded study whose will be dissemintated nationally
To recruit families to participate, so that they can provide data and feedback about their experience
To participate in the feedback loop – ongoing communication
Your role…

5. What’s in it for…? …the field
Who
What is the value?
…the field
Research is mixed and the evidence-base of FGDM is moderate—more research is needed
Additional research can support the national spread of family meetings or suggest the need for alternative practices
…families
Families receive interventions with known effectiveness
Opportunity for families to provide input about their experiences with CPS
What’s in it for…?

6. What’s in it for…? …CPS Staff and Agencies (You!) Accountability.
Who
What is the value?
…CPS Staff and Agencies (You!)
Accountability.
Opportunity to get input from staff who know the most about the families (You!)
Knowledge of effectiveness of family meetings can increase investment in the practice
Visibility and respect for advancing the field and systematically examining internal practices
Increased knowledge about the use and potential of evaluation
Results can impact system and practice change
FB: Break into 2 or 3 slides – too much info here
What’s in it for…?

7. Human Subjects Overview History of human subjects research
Historical laws and documents protecting human subjects
Your role in protecting the rights of participants*
*This study is not considered human subjects research according to the University of Colorado IRB HOWEVER, we still have an ethical obligation to protect the rights and interests of study participants – this is you and the families and meeting participants that you serve.
Human Subjects

8. Human Subjects History
Before there were human subjects protections:
Nuremberg (1930’s – 1940’s)
Concentration camp prisoners, twins
Willowbrook Hepatitis Study (1956)
Mentally retarded, institutionalized children
Brooklyn Jewish Chronic Disease Cancer Study (1963)
Poor, eldery
San Antonio Contraception Study (1971)
Mostly poor, women
Tuskegee Syphilis Study ( )
Poor, black males
Human Subjects History

9. Human Subjects History
Attempts to protect human subjects:
The Nuremberg Code (1947)
Informed consent. Benefits outweigh risks. Voluntary participation.
Declaration of Helsinki (1964…2000)
“Concern for the interests of the subject must always prevail over the interests of science & society”
National Research Act (1974)
Tuskegee  Institutional Review Board (IRB) approval required if getting U.S. Dept. of Health and Human Services funds
The Belmont Report (1979)
“Justice, beneficence, respect for persons”
Human Subjects History

10. Our Role in Protecting Participants Rights
Give participants all the information that they may need to make an informed decision.
Ensure that participants understand the information
Give participants an opportunity to consider involvement in the project on an initial and ongoing basis
Our information sheet serves that purpose. It is written at an 8th grade reading level to ensure literacy (a Spanish version is being worked on currently). Participation in surveys is voluntary – they can choose not to complete them at any time.
Our Role

11. News You Can Use What’s important to know: Privacy and Confidentiality
Risks to you personally are minimal
There are potential benefits to children, families, and to staff in the agency
Your participation is voluntary
Privacy and Confidentiality
All survey information is confidential and used only for the study
Survey data is maintained by Kempe Center NPLH evaluation team
Individually identifiable information will NOT be shared outside of the Kempe Center evaluation team
News You Can Use

12. Evaluation Terms Evidence-Based Practice Confidentiality vs. Anonymity
Bias
Fidelity
Validity
Process vs. Outcome evaluation
How vs. What
Qualitative vs. Quantitative methods
Depth vs. Breadth
FB: (optional) activity – break into small groups and discuss 1 or 2 bullet points and create a definition to share with the group.
Evaluation Terms

13. Evaluation Design - TX General Staff Survey
Families meet eligibility criteria and notified about the study; Caregiver Survey
Random Group Assignment
Meeting participants notified about the study; Fidelity Survey
Follow Up Survey
Case-Specific Questionnaire
Right now
FBSS worker makes FGC referral
FGC
1-2 months post-FGC
FBSS Case closure/transfer to CVS
Evaluation Design - TX

14. Evaluation Design - SD General Staff Survey
Families meet eligibility criteria and are notified about the study; Caregiver Survey
Random Group Assignment
Meeting participants notified about the study; Fidelity Survey
Follow Up Survey
Case-Specific Questionnaire
Right now
IFA worker makes FGC or FGC/TDM referral
FGC/TDM; IFA and Ongoing worker attend
1-2 months post-FGC
Ongoing case closure/transfer to out-of-home services
Evaluation Design - SD

15. General Staff Survey
Families meet eligibility criteria and are notified about the study; Caregiver Survey
Group Assignment – Propensity Score Match
Meeting participants notified about the study; Fidelity Survey
Follow Up Survey
Case-Specific Questionnaire
Right now
Intake or FAR worker makes 1st FSRT or FUM referral
2nd FUM/FGC
1-2 months post-FUM/FGC
Ongoing case closure/transfer to out-of-home services
Evaluation Design - CO

16. NPLH Surveys General Staff Survey Caregiver Survey
Meeting Fidelity Survey
Participant
Facilitator/Coordinator
Case-Specific Questionnaire
Follow-Up Survey
Caregiver – Intervention
Caregiver – Control
Meeting Participant
NPLH Surveys

17. Surveys that are distributed TO you
General Staff Survey
All staff who have a role in the evaluation (case workers, coordinators/facilitators, supervisors)
Coordinator-Facilitator Meeting Fidelity Survey
Coordinators/Facilitators only
Case-Specific Questionnaire
Case-workers only
Surveys that are distributed TO you

18. General Staff Survey Purpose: To understand more about:
Who utilizes family meetings in CPS
How much experience staff have with family meetings
How staff perceive family meetings
The context in which meetings are happening.
We can use this information to inform our process and outcome evaluations
Purpose here is to explain what you’ve already done – we didn’t want to taint your responses by providing this information earlier.
General Staff Survey

19. General Staff Survey Who? You! When? Now How? Via Survey Monkey
Case workers (FBSS, Ongoing…) providing in-home services
Family meeting facilitators/coordinators
Supervisors of both
When? Now
How? Via Survey Monkey
General Staff Survey

20. Coordinator-Facilitator Fidelity Survey
Purpose:
To understand more about:
How, why, and when family meetings are used
Who is invited to and who attends family meetings
Other characteristics of family meetings
Fidelity
Logistics
Meeting fidelity is part of our process evaluation
Coordinator-Facilitator Fidelity Survey

21. Coordinator-Facilitator Fidelity Survey
Who? Meeting coordinator/facilitator
When? After the family meeting (FUM/FGC/TDM)
Once back at your desk/computer (same day or day after)
How? Via Survey Monkey
FB:
Coordinator-Facilitator Fidelity Survey

22. Case-Specific Questionnaire
Purpose:
To understand more about:
What services in-home families receive
Are services provided impacted by family meetings?
Services provided is one of our outcome evaluation variables
Case-Specific Questionnaire

23. Case-Specific Questionnaire
Who? Case workers
When? At case closure OR transfer from in-home services to out-of-home services (foster care, kinship care, etc.)
Transfer does NOT mean voluntary placements
It does mean a formal or legal change in custody
How? Via Survey Monkey
We will send you a ‘tickler’ or reminder
Case-Specific Questionnaire

24. Surveys that are distributed BY you
Caregiver Survey
By case workers
At face-to-face meetings with family
At point of referral for a family meeting
Participant Fidelity Survey
By coordinators/facilitators
At family meetings
Surveys that are distributed BY you

25. What you ARE responsible for
Information Sheet
Explains project to participants.
Consent is not required but participation is voluntary.
What is voluntary? Completion of surveys NOT participation in child welfare services.
Distribute
Paper and pencil surveys with
Caregiver Survey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance
Meeting Participant – large self-addressed, stamped envelope for all surveys in the center of the room/table
FB: FAQ handout page that addresses some issues that might come up here.
FB: Handout information sheet about project.
What you ARE responsible for

26. What you are NOT responsible for
Explaining the content of surveys to families
Helping families/meeting participants fill out surveys
People with questions can call the number provided on the Information Sheet (have extra copies available for them to keep)
Other household members or family supports, etc. can help.
In order to maintain confidentiality, it is important that you merely handout surveys and provide no more information than what is included on the information sheets.
FB: Ask about potential barriers or challenges that staff think they might run into – incorporate into activities. First: Fishbowl activity modeled by trainers. Second: fishbowl with volunteers. Third: Present case scenario “what if” situations about how to respond to family members asking for this kind of help. (Explain why not?)
What you are NOT responsible for

27. Caregiver Survey Purpose: To understand more about:
The characteristics of families receiving in-home child welfare services (e.g. demographics)
The protective capacities of parents
The parent-child relationship and child behavior
This information can be linked to outcomes
Caregiver Survey

28. Caregiver Survey Who? Parents or legal guardians When? How?
Face-to-face meeting
At time of referral for family meeting
How?
Paper and pencil
Self-addressed and stamped envelope
Distributed by IFA/FBSS/FAR/Intake Caseworker
Parents can call the number on the information sheet to take the survey via phone
Caregiver Survey

29. Participant Fidelity Survey
Purpose:
To understand more about:
How, why, and when family meetings are used
Who is invited to and who attends family meetings
Other characteristics of family meetings
Fidelity
Logistics
Meeting fidelity is part of our process evaluation
We want to hear from families and other participants as well as staff.
Participant Fidelity Survey

30. Participant Fidelity Survey
Who? Any meeting participant who agrees to fill it out
When? AT the family meeting
Section 1: Before the meeting starts
Section 2: After the meeting ends
How? Paper and Pencil
Meeting participants will fill out paper-pencil surveys and place them in a large envelope
The coordinator/facilitator will mail them to the evaluation team
FB: Ask facilitators/coordinators for concerns about barriers and challenges to this.
Participant Fidelity Survey

31. Surveys administered by evaluation team
How? Paper and pencil; via mail
3 follow-up survey versions with the following components:
Meeting Follow-Up
Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey
Caregiver Follow-Up
Who? All (intervention and control group) parents
Family Satisfaction
Who? All (intervention and control group) parents AND meeting participants
Surveys administered by evaluation team

32. NPLH Survey Summary Case workers: Coordinators/Facilitators: Complete:
General Staff Survey (1 time)
Case Specific-Questionnaire (for every study case)
Distribute:
Caregiver Survey (to every study case)
Coordinators/Facilitators:
Coordinator-Facilitator Fidelity Survey (for every study family meeting)
Meeting Participant Fidelity Survey (for every study family meeting)
NPLH Survey Summary

33. Coordinators/Facilitators Meeting Log
You will be asked to fill out information an additional spreadsheet (not unlike what you may be doing now)
Ex. Date of referral, meeting type, date of meeting, meeting cancellation reason, etc.
This will be posted on a document sharing website
We will send the link to you before data collection begins
Coordinators/Facilitators Meeting Log

34. THANK YOU! If, throughout the life of this
project, you have any questions
please contact:
Heather Allan, NPLH Project Coordinator
Phone:
Overall fb: the “what” is clear but “how” to do it, is less clear. We can build in interactivities to enhance that.
THANK YOU!