1. Testing the DR Logic Model: Associating Engagement With Services and Outcomes Presented at the 9th Annual Conference on Differential Response in Child Welfare November 12, 2014
Kempe Center for the Prevention & Treatment of Child Abuse & Neglect
Dana Hollinshead
John Fluke &
Lisa Merkel-Holguin
Sungkyunkwan University
Sangwon Kim

2. Overview QIC-DR Cross-Site Evaluation Methods and Key Findings
Safety, Services, and Engagement
Conceptual Framework for Examining Engagement
Analyses Examining:
Factors affecting Parent Emotional Response
Factors affecting Parent Satisfaction

3. QIC-DR Cross-Site Evaluation: Methods and Key Findings

4. Terminology for Differential Response
Alternative Response
Investigation Response

5. Core Research Questions
How is the AR pathway different from the IR pathway in terms of family engagement, caseworker practice, and services provided?
Are children whose families participate in AR as safe as or safer than children whose families participate in IR?
What are the cost implications for child protection agencies that implement DR?

6. Required Elements of R&D Sites for DR Implementation
Target: low to moderate risk screened-in cases
Clear criteria for assigning to AR
Assignment can be changed (AR to IR)
If assigned to AR, families can choose IR
Services not optional for AR families if safety concerns identified
Guiding statute/policy for AR and IR
No findings on maltreatment allegations for AR
Since AR does not identify perpetrators, no caregivers are entered into the State Central Registry

7. Evaluation Methodology
Evaluation Components
Implementation/process
Outcomes
Cost
Multi-site
Coordination across local evaluations
Common measures, instruments
Each site’s data reported separately
Each site implemented random assignment

8. Random Assignment Investigation Pool of Reports Eligible for AR
Child Maltreatment Reports
Pathway Assignment
Pool of Reports Eligible for AR
Random Assignment
Treatment Group D
Control Group
Inappropriate for AR
AR-appropriate
Alternative Response B C
Initial Screening For CPS
Screened out
Accepted Report
(Screened in) A
The Evaluation
Random Assignment

9. Two Track System Characteristics of Clients
Investigation Response
Alternative Response
Suspicious child death or homicide
Sexual abuse
Severe physical harm
Reports involving childcare providers, teachers, etc.
Lack of supervision
Medical neglect
Poor living conditions
Educational neglect
Drugs and alcohol 9

10. Site Characteristics Category Colorado Illinois Ohio
Scope of implementation
5 counties
Statewide
6 counties
Previous DR implementation? No
Yes
CPS structure
County administered
State administered
Percent of screened-in cases substantiated
45%
15%
Not available
Staff
AR caseworker
Private/public paired approach
Site Characteristics

11. Data Sources and Instrumentation
Administrative Data on each case (at 365 days)
Basic Demographics
ReReferral and Placement Indicators
Case Process Dates
Case Report Questionnaire (at case closure)
Includes case, services, and engagement data
Completed by caseworker
Family Survey (at case closure)
Satisfaction
Additional Demographics
Completed by Primary Caregiver
Caseworker Survey (varied by site)
Work Functions and Caseload
Skills and Capacity
Perception of the Protective Services Role
Completed by Caseworker

12. Cross-site Study Sample
Colorado
Illinois
Ohio
Total
(n=1,667) AR
(n=870) IR
(n=797)
(n=4,534)
(n=1,706)
(n=2,828)
(n=846)
(n=543)
(n=303)
Response Rates
Administrative Data
1,667 (100%)
870 (52.2%)
797 (47.8%)
4,534 (100%)
1,706 (37.6%)
2,828 (62.4%)
846
(100%)
543 (64.2%)
303 (35.8%)
Case Report Data
846 (100%)
Family Survey Data
398 (23.9%)
219
(55.0%)
179 (45.0%)
1,132 (25.0%)
518 (45.8%)
614 (54.2%)
319 (37.7%)
228 (71.5%)
91 (28.5%)
Staff Survey*
Caseworkers: 89/143 (62%)
Supervisors: 30/39 (77%)
Overall: 119/182 (65%)
Caseworkers: 200/741 (27%)
Supervisors: 48/171 (28%)
Overall: 248/912 (27.2%)
Overall: 227/378 (60%)

13. Key Findings: Study Population and Pathway Assignment

14. Study Population of Those Eligible for AR: Cases by Assigned Pathway

15. Study Population Characteristics

16. QIC-DR Cross-Site Findings Regarding Services to Families

17. Post-Assessment Re-Referrals by Pathway by Site

18. Removals by Pathway by Site

19. Summary of Safety Differences between AR and IR
Colorado
Illinois*
Ohio
Number of post-assessment re-referrals NS
IR has slightly fewer “0”
AR has slightly more “1” Both AR and IR decrease at “2” and “3 or more”
Number of children removed
Number of removals by pathway change
*p ≤ .05, **p ≤ .01, ***p ≤ NS=Not Statistically Significant
Summary of Safety Differences between AR and IR

20. Safety Findings in Context

21. Re-Referrals Findings in Context
Minnesota 2004
Ohio 2013
New York 2011
QIC-DR cross site 2014
Colorado
Illinois
Ohio
During assessment or investigation phase
No difference between groups
No difference between groups at 6 months post-closure
No difference (descriptive)
AR lower (regression)
AR more
AR higher (regression)
Families with prior CPS involvement
Not yet analyzed
Ineligible for study
Families with NO prior CPS involvement
AR longer time to re-referral

22. Questions

23. Towards an Understanding of Parent Engagement, Service Use, and Satisfaction

24. QIC-DR Cross-Site Findings Regarding Emotional Responses of Families

25. Engagement Constructs
Caseworker perceptions of parental emotional attributes at first and last visit
Parental reports about affect and emotions
Parental reports about satisfaction with treatment by the caseworker and the help they received
Parental reports of likelihood of re-contacting the worker or agency

26. Survey Items Composing Caseworkers’ Reports of Family Emotional Attributes
Positive Emotional Attributes
Cooperative
Receptive to Help
Engaged
Negative Emotional Attributes
Uncooperative
Difficult

27. Caseworker Report of Caregiver’s Positive Emotional Attributes
Item
Colorado
Illinois
Ohio
Caseworker report of family’s positive emotional attributes at initial meeting
Higher for IR**
Higher for IR***
Caseworker report of family’s positive emotional attributes at last meeting
Equivalent increase for IR and AR***
**p ≤ .01, ***p ≤ .001

28. Caseworker Report of Family’s Negative Emotional Attributes
Item
Colorado
Illinois
Ohio
Caseworker report of family’s negative emotional attributes at initial meeting NS
Caseworker report of family’s negative family emotional attributes at last meeting
Equivalent decrease for IR and AR***
Decrease only for IR**
**p ≤ .01, ***p ≤ .001, NS=Not Statistically Significant

29. Survey Items Composing Parents’ Self-Reported Emotional Responses
Parent Report of Positive Affect
Relieved
Respected
Encouraged
Thankful
Hopeful
Comforted
Parent Report of Anger
Angry
Disrespected
Discouraged
Parent Report of Worry
Worried
Stressed
Afraid

30. Parents’ Reports of Positive Affect, Worry, or Anger
Item
Colorado
Illinois
Ohio
Parent report of positive family affect at first meeting NS
Greater positive affect for AR parents***
Greater positive affect for AR parents*
Parent report of worry at first meeting
IR parents more worried***
AR parents more worried***
Parent report of anger at first meeting
IR parents indicated more anger***
*p ≤ .05, ***p ≤ .001, NS = Not Statistically Significant
Parents’ Reports of Positive Affect, Worry, or Anger

31. Parents’ Satisfaction and Likelihood of Re-Contacting
Item
Colorado
Illinois
Ohio
Parent report of satisfaction with treatment by caseworker NS
AR parents more satisfied***
Parent report of satisfaction with the help received from caseworker
Parent report of likelihood of calling caseworker/agency in the future
AR parents more likely*
AR parents more likely***
*p ≤ .05, ***p ≤ .001, NS=Not Statistically Significant
Parents’ Satisfaction and Likelihood of Re-Contacting

32. Emotional Response Findings in Context

33. Parent/Caregiver Satisfaction in their Treatment by CPS caseworker
Ohio 2010
New York 2011
Nevada 2010
QIC-DR Cross-Site 2014
Colorado
Illinois
Ohio
AR higher
No difference between AR and IR

34. Parent/Caregiver Satisfaction with the Services Received
Ohio 2010
Missouri 1997
QIC-DR cross site 2014
(measure was help received from caseworker)
Colorado
Illinois
Ohio
AR higher
No difference between AR and IR

35. Caseworkers’ Perceptions of Positive Family Emotional Responses
Minnesota 2004
Ohio 2010
QIC-DR cross site 2014
Colorado
Illinois
Ohio
AR higher
IR higher

36. QIC-DR Cross-Site Findings Regarding Services to Families

37. Receipt of Services
AR families were more likely to receive at least one service during the initial phase than IR families
In all 3 sites, more AR families than IR families received services (significant at p≤.001)
Percent of Families Receiving One or More Service

38. Timeliness of Services
High percentages of both AR and IR cases that received services, received them within 2 weeks
Illinois: significant difference between AR and IR.
Timeliness of Receipt of Services
Colorado
Illinois***
Ohio
AR (n=332)
IR (n=207)
Total (n=539)
AR (n=980)
IR (n=579)
Total (n=1,559)
AR (n=263)
IR (n=89)
Total (n=352)
Within 2 weeks
69.9% (232)
77.3% (160)
72.7% (392)
89.2% (874)
67.9% (393)
81.3% (1,267)
66.5% (175)
64.1% (57)
65.9% (232)
***p ≤ .0001

39. Types of Services
Among all 3 grantees, AR families were more likely than IR families to receive services to meet material or basic needs
Only IL had other significant differences:
AR more than IR: social support, educational, parenting
IR more than AR: substance abuse

40. Length of AR and IR Cases (Mean # of days)
AR cases significantly longer than IR in all 3 sites (p ≤ .001)

41. Open in Ongoing Services
Fewer than 20 percent of cases in either pathway were opened in ongoing services among the 3 sites
In CO: 16% (AR) compared to 6% (IR) (p=≤.001)
In IL: 6% (AR) compared to 13% (IR) (p=≤.001)
Possible factors: age of children, engagement of family, how long family served in response phase

42. Services Findings in Context

43. Receipt of one or more services by pathway
Minnesota 2004
New York 2011
Nevada 2010
QIC-DR cross site 2014
Colorado
Illinois
Ohio
AR higher

44. Types of services received by pathway
MN 2004
Ohio 2010
Nevada 2011
New York 2011
Ohio 2013
QIC-DR cross site 2014 CO IL OH
Basic needs
AR more
Substance abuse treatment
IR more

45. New Analyses

46. Conceptual Framework

47. Caregiver Emotional Response Study

48. Factors Affecting Caregiver Emotional Response
Research Question:
To what extent might family, casework, and/ or intervention characteristics influence caregivers’ emotional response to the intervention they received?

49. Caregiver Emotional Response Study: Study Population Characteristics (n = 1530)
Factors AR IR
n (%)
Illinois Cases
518 (70.3%)
614 (77.4%)
Pathway Assigned
737 (48.2%)
793 (51.8%)
Families with Only One Adult in Household
315 (42.8%)
281 (35.4%)**
Physical Abuse Alleged
78 (10.6%)
64 (8.1%)
Neglect Alleged
576 (78.2%)
630 (79.4%)
Psychological Maltreatment
13 (1.8%)
8 (1.0%)
Caseworker Identified Family as Having Two or More Needs
345 (46.8%)
243 (30.6%)***
Caregiver Reports One Face to Face Contact with Caseworker
145 (20.1%)
374 (47.9%)***
Caregiver Respondent Was Female
661 (91.6%)
710 (93.1%)
Caregiver Identifies as White, non-Hispanic
466 (64.7%)
451 (59.4%)*
Mean (SD)
Age of Primary Caregiver at Intake
34.18 (9.89)
33.55 (9.52)
Number of Children Associated with the Assessment
1.89 (1.12)
2.03 (1.14)
Age of Youngest Child at Pathway Assignment
6.16 (5.04)
5.69 (4.84)
Casework Scale
11.17 (1.60)
10.66 (1.97)***
* p <.05, **p < .01, *** p < .001

50. Caregivers’ Emotional Response Study: GLM Model Results
Parameter
Caregiver Report of
Positive Affect
Worry
Anger
(n = 1435)
(n = 1380)
(n = 1427) B
Std. Error
95% Confidence Interval
Lower
Upper
Intercept
-0.29***
0.05
-0.39
-0.19
0.79***
0.06
0.67
0.90
0.78***
0.04
0.71
0.85
Alternative Response Received
0.09***
0.02
0.13
-0.11***
-0.15
-0.07
-0.06***
0.01
-0.09
-0.04
Casework Scale Rating by Caregiver
0.05***
-0.04***
-0.05
-0.03
0.00
-0.06
State*Pathway
-0.05*
0.03
-0.10
0.07*
0.04*
0.08
More than Two Needs Identified by Caseworker
0.04**
Caregiver Reports Only One Face-to-Face Contact
-0.10***
-0.04**
-0.01
One Adult in Household
-0.08
Male Caregiver
-0.09*
-0.02
Caregiver Identifies as White, non-Hispanic
.06**
0.10
F(4, 1430) = 52.79***
F(7, 1372) = 21.04***
F(4, 1422) = 98.74***
*p<.05, **p<.01, ***p<.001

51. Caregiver Satisfaction Study

52. Conceptual Framework

53. Caregiver Satisfaction Study: Methods
Built on Prior Work
Re-ran Emotional Response Factor Analysis on Combined Sample
Tested Alternative Models using Structural Equation Modeling

54. Revised Emotion Factors
Positive Emotional Response
Respected
Encouraged
Thankful
Hopeful
Negative Emotional Response
Anger
Disrespected
Discouraged
Stressed

55. Caregiver Satisfaction Study: Correlations Between Variables
Pathway
CW Listened Carefully
CW Understood Needs
CW Considered Opinions
Positive Affect
Negative Affect
Satisfied with Help
Likelihood of Contacting CW
Pathway (Investigative Response)
Caseworker Listened Carefully
.091** 1
Caseworker Understood Needs
.104**
.746**
Caseworker Considered Opinions
.168**
.622**
.628**
.220**
.228**
.268**
.265**
-.215**
-.405**
-.385**
-.395**
.134**
.621**
.696**
.585**
.291**
-.406**
Likelihood of Contacting Caseworker
.189**
.508**
.479**
.369**
-.410**
Note: The correlation between pathway and other variables is a Biserial correlation coefficient. The rest are Pearson correlation coefficients.
** p < .01

56. Caregiver Satisfaction Study: Population Characteristics (n = 1530)
Factors
n (%)
Range
Data Source
Illinois Cases (Colorado)
1132(74%)
0 - 1
Admin
Alternative Response Pathway Assigned (Investigative Response)
737 (48%)
Families with Only One Adult in Household (Families with Two or More Adults)
596 (39%)
Caregiver Identifies as White, non-Hispanic (All Other Identities)
917 (62%)
Caregiver
Mean (SD)
Data Source 
Number of Children Associated with the Assessment
1.96 (1.14)
1 - 8
Contact Intensity
0.24 (0.26)
0 - 3
Caseworker & Admin 
Number of Face-to-Face and Phone Contacts with Caseworker Reported by Caseworker
5.6 (6.3)
0 - 56
Caseworker
Duration of Assessment and Ongoing Services, in Days
66.8 (65.1)

57. Caregiver Satisfaction Study: Population Characteristics, cont’d
Caregiver Satisfaction Study: Population Characteristics, cont’d. (n = 1530)
Mean (SD)
Range
Data Source 
Interaction Style1
How Carefully Caseworker Listened
2.82 (0.22)
1 - 3
Caregiver
How Well Caseworker Understood Family's Needs
2.72 (0.31)
How Often Caseworker Considered Family's Opinions Before Decisions Were Made
2.72 (0.34)
Positive Affect
0.34 (0.12)
0 - 1
Negative Affect
0.18 (0.07)
Satisfaction1
Satisfaction with Help Received from Caseworker
2.68 (0.36)
Likelihood of Calling Caseworker or Agency for Help in Future
2.48 (0.53)
1 Latent Variable

58. Caregiver Satisfaction SEM Results
Chi-square (df) p
CFI
TLI
RMSEA (90% C.I.)
(28)
.000
.982
.956
.047 (.038, .056)
Pathway
Negative
Emotional
Response
Positive
Interaction
Style
Satisfaction
.282(.246)***
-.455(-.296)***
.114(.171)***
-.042(-.084), p=.157
.804(1.048)***
.113(.089)***
.153(.105)***
-.129(-.066)***
.044(.045)*
Corr. between mediators:
PER with NER = -.290(-.021)***

59. Caregiver Studies Implications
Practice and Training
Development of casework skills
Conscious awareness of emotional impact of intervention
Leverage policy to facilitate more optimal emotional response
Time/resources to apply casework skills
Explicit recognition of organizational values
Proactive policies that reflect organizational values
Other Implications

60. Implications (continued)
Future Research
Does engagement impact services usage?
With or without DR
Type of services received influence:
Caregiver emotional response
Satisfaction with services

61. www.DifferentialResponseQIC.org Site reports Cross-Site Report
Evaluation tools
Protecting Children journal
Issue briefs
Literature reviews (2009 and 2011)
Guide for judges and judicial officers
Online state survey and report
Implementation manual
Webinars
Issue briefs

62. Contact Information
Dana Hollinshead:
John Fluke:
Lisa Merkel-Holguin: Lisa.merkel-