Testing the DR Logic Model: Associating Engagement With Services and Outcomes Presented at the 9th Annual Conference on Differential Response in Child Welfare November 12, 2014 Kempe Center for the Prevention & Treatment of Child Abuse & Neglect Dana Hollinshead John Fluke & Lisa Merkel-Holguin Sungkyunkwan University Sangwon Kim

Overview QIC-DR Cross-Site Evaluation Methods and Key Findings Safety, Services, and Engagement Conceptual Framework for Examining Engagement Analyses Examining: Factors affecting Parent Emotional Response Factors affecting Parent Satisfaction

QIC-DR Cross-Site Evaluation: Methods and Key Findings

Terminology for Differential Response Alternative Response Investigation Response

Core Research Questions How is the AR pathway different from the IR pathway in terms of family engagement, caseworker practice, and services provided? Are children whose families participate in AR as safe as or safer than children whose families participate in IR? What are the cost implications for child protection agencies that implement DR?

Required Elements of R&D Sites for DR Implementation Target: low to moderate risk screened-in cases Clear criteria for assigning to AR Assignment can be changed (AR to IR) If assigned to AR, families can choose IR Services not optional for AR families if safety concerns identified Guiding statute/policy for AR and IR No findings on maltreatment allegations for AR Since AR does not identify perpetrators, no caregivers are entered into the State Central Registry

Evaluation Methodology Evaluation Components Implementation/process Outcomes Cost Multi-site Coordination across local evaluations Common measures, instruments Each site’s data reported separately Each site implemented random assignment

Random Assignment Investigation Pool of Reports Eligible for AR Child Maltreatment Reports Pathway Assignment Pool of Reports Eligible for AR Random Assignment Treatment Group D Control Group Inappropriate for AR AR-appropriate Alternative Response B C Initial Screening For CPS Screened out Accepted Report (Screened in) A The Evaluation Random Assignment

Two Track System Characteristics of Clients Investigation Response Alternative Response Suspicious child death or homicide Sexual abuse Severe physical harm Reports involving childcare providers, teachers, etc. Lack of supervision Medical neglect Poor living conditions Educational neglect Drugs and alcohol 9

Site Characteristics Category Colorado Illinois Ohio Scope of implementation 5 counties Statewide 6 counties Previous DR implementation? No Yes CPS structure County administered State administered Percent of screened-in cases substantiated 45% 15% Not available Staff AR caseworker Private/public paired approach Site Characteristics

Data Sources and Instrumentation Administrative Data on each case (at 365 days) Basic Demographics ReReferral and Placement Indicators Case Process Dates Case Report Questionnaire (at case closure) Includes case, services, and engagement data Completed by caseworker Family Survey (at case closure) Satisfaction Additional Demographics Completed by Primary Caregiver Caseworker Survey (varied by site) Work Functions and Caseload Skills and Capacity Perception of the Protective Services Role Completed by Caseworker

Cross-site Study Sample   Colorado Illinois Ohio Total (n=1,667) AR (n=870) IR (n=797) (n=4,534) (n=1,706) (n=2,828) (n=846) (n=543) (n=303) Response Rates Administrative Data 1,667 (100%) 870 (52.2%) 797 (47.8%) 4,534 (100%) 1,706 (37.6%) 2,828 (62.4%) 846 (100%) 543 (64.2%) 303 (35.8%) Case Report Data 846 (100%) Family Survey Data 398 (23.9%) 219 (55.0%) 179 (45.0%) 1,132 (25.0%) 518 (45.8%) 614 (54.2%) 319 (37.7%) 228 (71.5%) 91 (28.5%) Staff Survey* Caseworkers: 89/143 (62%) Supervisors: 30/39 (77%) Overall: 119/182 (65%) Caseworkers: 200/741 (27%) Supervisors: 48/171 (28%) Overall: 248/912 (27.2%) Overall: 227/378 (60%)

Key Findings: Study Population and Pathway Assignment

Study Population of Those Eligible for AR: Cases by Assigned Pathway

Study Population Characteristics

QIC-DR Cross-Site Findings Regarding Services to Families

Post-Assessment Re-Referrals by Pathway by Site

Removals by Pathway by Site

Summary of Safety Differences between AR and IR   Colorado Illinois* Ohio Number of post-assessment re-referrals NS IR has slightly fewer “0” AR has slightly more “1” Both AR and IR decrease at “2” and “3 or more” Number of children removed Number of removals by pathway change *p ≤ .05, **p ≤ .01, ***p ≤ .001 NS=Not Statistically Significant Summary of Safety Differences between AR and IR

Safety Findings in Context

Re-Referrals Findings in Context   Minnesota 2004 Ohio 2013 New York 2011 QIC-DR cross site 2014 Colorado Illinois Ohio During assessment or investigation phase No difference between groups No difference between groups at 6 months post-closure No difference (descriptive) AR lower (regression) AR more AR higher (regression) Families with prior CPS involvement Not yet analyzed Ineligible for study Families with NO prior CPS involvement AR longer time to re-referral

Questions

Towards an Understanding of Parent Engagement, Service Use, and Satisfaction

QIC-DR Cross-Site Findings Regarding Emotional Responses of Families

Engagement Constructs Caseworker perceptions of parental emotional attributes at first and last visit Parental reports about affect and emotions Parental reports about satisfaction with treatment by the caseworker and the help they received Parental reports of likelihood of re-contacting the worker or agency

Survey Items Composing Caseworkers’ Reports of Family Emotional Attributes Positive Emotional Attributes Cooperative Receptive to Help Engaged Negative Emotional Attributes Uncooperative Difficult

Caseworker Report of Caregiver’s Positive Emotional Attributes Item Colorado Illinois Ohio Caseworker report of family’s positive emotional attributes at initial meeting Higher for IR** Higher for IR*** Caseworker report of family’s positive emotional attributes at last meeting Equivalent increase for IR and AR*** **p ≤ .01, ***p ≤ .001

Caseworker Report of Family’s Negative Emotional Attributes Item Colorado Illinois Ohio Caseworker report of family’s negative emotional attributes at initial meeting NS Caseworker report of family’s negative family emotional attributes at last meeting Equivalent decrease for IR and AR*** Decrease only for IR** **p ≤ .01, ***p ≤ .001, NS=Not Statistically Significant

Survey Items Composing Parents’ Self-Reported Emotional Responses Parent Report of Positive Affect Relieved Respected Encouraged Thankful Hopeful Comforted Parent Report of Anger Angry Disrespected Discouraged Parent Report of Worry Worried Stressed Afraid

Parents’ Reports of Positive Affect, Worry, or Anger Item Colorado Illinois Ohio Parent report of positive family affect at first meeting NS Greater positive affect for AR parents*** Greater positive affect for AR parents* Parent report of worry at first meeting IR parents more worried*** AR parents more worried*** Parent report of anger at first meeting IR parents indicated more anger*** *p ≤ .05, ***p ≤ .001, NS = Not Statistically Significant Parents’ Reports of Positive Affect, Worry, or Anger

Parents’ Satisfaction and Likelihood of Re-Contacting Item Colorado Illinois Ohio Parent report of satisfaction with treatment by caseworker NS AR parents more satisfied*** Parent report of satisfaction with the help received from caseworker Parent report of likelihood of calling caseworker/agency in the future AR parents more likely* AR parents more likely*** *p ≤ .05, ***p ≤ .001, NS=Not Statistically Significant Parents’ Satisfaction and Likelihood of Re-Contacting

Emotional Response Findings in Context

Parent/Caregiver Satisfaction in their Treatment by CPS caseworker Ohio 2010 New York 2011 Nevada 2010 QIC-DR Cross-Site 2014   Colorado Illinois Ohio AR higher No difference between AR and IR

Parent/Caregiver Satisfaction with the Services Received Ohio 2010 Missouri 1997 QIC-DR cross site 2014 (measure was help received from caseworker)   Colorado Illinois Ohio AR higher No difference between AR and IR

Caseworkers’ Perceptions of Positive Family Emotional Responses Minnesota 2004 Ohio 2010 QIC-DR cross site 2014   Colorado Illinois Ohio AR higher IR higher

QIC-DR Cross-Site Findings Regarding Services to Families

Receipt of Services AR families were more likely to receive at least one service during the initial phase than IR families In all 3 sites, more AR families than IR families received services (significant at p≤.001) Percent of Families Receiving One or More Service

Timeliness of Services High percentages of both AR and IR cases that received services, received them within 2 weeks Illinois: significant difference between AR and IR. Timeliness of Receipt of Services Colorado Illinois*** Ohio AR (n=332) IR (n=207) Total (n=539) AR (n=980) IR (n=579) Total (n=1,559) AR (n=263) IR (n=89) Total (n=352) Within 2 weeks 69.9% (232) 77.3% (160) 72.7% (392) 89.2% (874) 67.9% (393) 81.3% (1,267) 66.5% (175) 64.1% (57) 65.9% (232) ***p ≤ .0001

Types of Services Among all 3 grantees, AR families were more likely than IR families to receive services to meet material or basic needs Only IL had other significant differences: AR more than IR: social support, educational, parenting IR more than AR: substance abuse

Length of AR and IR Cases (Mean # of days) AR cases significantly longer than IR in all 3 sites (p ≤ .001)

Open in Ongoing Services Fewer than 20 percent of cases in either pathway were opened in ongoing services among the 3 sites In CO: 16% (AR) compared to 6% (IR) (p=≤.001) In IL: 6% (AR) compared to 13% (IR) (p=≤.001) Possible factors: age of children, engagement of family, how long family served in response phase

Services Findings in Context

Receipt of one or more services by pathway Minnesota 2004 New York 2011 Nevada 2010 QIC-DR cross site 2014   Colorado Illinois Ohio AR higher

Types of services received by pathway   MN 2004 Ohio 2010 Nevada 2011 New York 2011 Ohio 2013 QIC-DR cross site 2014 CO IL OH Basic needs AR more Substance abuse treatment IR more

New Analyses

Conceptual Framework

Caregiver Emotional Response Study

Factors Affecting Caregiver Emotional Response Research Question: To what extent might family, casework, and/ or intervention characteristics influence caregivers’ emotional response to the intervention they received?

Caregiver Emotional Response Study: Study Population Characteristics (n = 1530) Factors AR IR n (%) Illinois Cases 518 (70.3%) 614 (77.4%) Pathway Assigned 737 (48.2%) 793 (51.8%) Families with Only One Adult in Household 315 (42.8%) 281 (35.4%)** Physical Abuse Alleged 78 (10.6%) 64 (8.1%) Neglect Alleged 576 (78.2%) 630 (79.4%) Psychological Maltreatment 13 (1.8%) 8 (1.0%) Caseworker Identified Family as Having Two or More Needs 345 (46.8%) 243 (30.6%)*** Caregiver Reports One Face to Face Contact with Caseworker 145 (20.1%) 374 (47.9%)*** Caregiver Respondent Was Female 661 (91.6%) 710 (93.1%) Caregiver Identifies as White, non-Hispanic 466 (64.7%) 451 (59.4%)*   Mean (SD) Age of Primary Caregiver at Intake 34.18 (9.89) 33.55 (9.52) Number of Children Associated with the Assessment 1.89 (1.12) 2.03 (1.14) Age of Youngest Child at Pathway Assignment 6.16 (5.04) 5.69 (4.84) Casework Scale 11.17 (1.60) 10.66 (1.97)*** * p <.05, **p < .01, *** p < .001

Caregivers’ Emotional Response Study: GLM Model Results Parameter Caregiver Report of Positive Affect Worry Anger (n = 1435) (n = 1380) (n = 1427) B Std. Error 95% Confidence Interval Lower Upper Intercept -0.29*** 0.05 -0.39 -0.19 0.79*** 0.06 0.67 0.90 0.78*** 0.04 0.71 0.85 Alternative Response Received 0.09*** 0.02 0.13 -0.11*** -0.15 -0.07 -0.06*** 0.01 -0.09 -0.04 Casework Scale Rating by Caregiver 0.05*** -0.04*** -0.05 -0.03 0.00 -0.06 State*Pathway -0.05* 0.03 -0.10 0.07* 0.04* 0.08 More than Two Needs Identified by Caseworker 0.04**   Caregiver Reports Only One Face-to-Face Contact -0.10*** -0.04** -0.01 One Adult in Household -0.08 Male Caregiver -0.09* -0.02 Caregiver Identifies as White, non-Hispanic .06** 0.10 F(4, 1430) = 52.79*** F(7, 1372) = 21.04*** F(4, 1422) = 98.74*** *p<.05, **p<.01, ***p<.001

Caregiver Satisfaction Study

Conceptual Framework

Caregiver Satisfaction Study: Methods Built on Prior Work Re-ran Emotional Response Factor Analysis on Combined Sample Tested Alternative Models using Structural Equation Modeling

Revised Emotion Factors Positive Emotional Response Respected Encouraged Thankful Hopeful Negative Emotional Response Anger Disrespected Discouraged Stressed

Caregiver Satisfaction Study: Correlations Between Variables   Pathway CW Listened Carefully CW Understood Needs CW Considered Opinions Positive Affect Negative Affect Satisfied with Help Likelihood of Contacting CW Pathway (Investigative Response) Caseworker Listened Carefully .091** 1 Caseworker Understood Needs .104** .746** Caseworker Considered Opinions .168** .622** .628** .220** .228** .268** .265** -.215** -.405** -.385** -.395** .134** .621** .696** .585** .291** -.406** Likelihood of Contacting Caseworker .189** .508** .479** .369** -.410** Note: The correlation between pathway and other variables is a Biserial correlation coefficient. The rest are Pearson correlation coefficients. ** p < .01

Caregiver Satisfaction Study: Population Characteristics (n = 1530) Factors n (%) Range Data Source Illinois Cases (Colorado) 1132(74%) 0 - 1 Admin Alternative Response Pathway Assigned (Investigative Response) 737 (48%) Families with Only One Adult in Household (Families with Two or More Adults) 596 (39%) Caregiver Identifies as White, non-Hispanic (All Other Identities) 917 (62%) Caregiver   Mean (SD) Data Source  Number of Children Associated with the Assessment 1.96 (1.14) 1 - 8 Contact Intensity 0.24 (0.26) 0 - 3 Caseworker & Admin  Number of Face-to-Face and Phone Contacts with Caseworker Reported by Caseworker 5.6 (6.3) 0 - 56 Caseworker Duration of Assessment and Ongoing Services, in Days 66.8 (65.1) 1 - 364

Caregiver Satisfaction Study: Population Characteristics, cont’d Caregiver Satisfaction Study: Population Characteristics, cont’d. (n = 1530)   Mean (SD) Range Data Source  Interaction Style1 How Carefully Caseworker Listened 2.82 (0.22) 1 - 3 Caregiver How Well Caseworker Understood Family's Needs 2.72 (0.31) How Often Caseworker Considered Family's Opinions Before Decisions Were Made 2.72 (0.34) Positive Affect 0.34 (0.12) 0 - 1 Negative Affect 0.18 (0.07) Satisfaction1 Satisfaction with Help Received from Caseworker 2.68 (0.36) Likelihood of Calling Caseworker or Agency for Help in Future 2.48 (0.53) 1 Latent Variable

Caregiver Satisfaction SEM Results Chi-square (df) p CFI TLI RMSEA (90% C.I.) 118.692(28) .000 .982 .956 .047 (.038, .056) Pathway Negative Emotional Response Positive Interaction Style Satisfaction .282(.246)*** -.455(-.296)*** .114(.171)*** -.042(-.084), p=.157 .804(1.048)*** .113(.089)*** .153(.105)*** -.129(-.066)*** .044(.045)* Corr. between mediators: PER with NER = -.290(-.021)***

Caregiver Studies Implications Practice and Training Development of casework skills Conscious awareness of emotional impact of intervention Leverage policy to facilitate more optimal emotional response Time/resources to apply casework skills Explicit recognition of organizational values Proactive policies that reflect organizational values Other Implications

Implications (continued) Future Research Does engagement impact services usage? With or without DR Type of services received influence: Caregiver emotional response Satisfaction with services

www.DifferentialResponseQIC.org Site reports Cross-Site Report Evaluation tools Protecting Children journal Issue briefs Literature reviews (2009 and 2011) Guide for judges and judicial officers Online state survey and report Implementation manual Webinars Issue briefs

Contact Information Dana Hollinshead: Dana.Hollinshead@childrenscolorado.org John Fluke: John.Fluke@ucdenver.edu Lisa Merkel-Holguin: Lisa.merkel- holguin@childrenscolorado.org