Propensity Score Matching to Evaluate the Impact of Substance Abuse Treatment Services for Child Welfare Clients Richard Barth,¹ Claire Gibbons,² & Shenyang Guo¹ Jordan Institute for Families Schools of Social Work (1) and Public Health (2) University of North Carolina Presented to the Annual Grantees Meeting Substance Abuse Policy Research Program Charleston, South Carolina December 2, 2004 PSM analyses were funded by the Robert Wood Johnson Foundation’s Substance Abuse Policy Research Program NSCAW data used to illustrate PSM were collected under funding by the Administration on Children, Youth, and Families of the U.S. Department of Health and Human Services. Findings do not represent the official position or policies of the U.S. DHHS. Results are not quotable in print. Contact

Today’s Presentation Introduction to the research questions Brief explanation of the data sources and reasons to use PSM to answer the research questions. Description of post PSM analyses Discussion of the findings

Background for the Study Next to poverty, substance abuse is considered most central reason for CWS involvement Linda Gordon Heroes of Their Own Lives (1800s) ACF Building Common Ground (1999) SAMSHA and ACF National Resource Center on Child Welfare and Substance Abuse (2003) Largest population of children who receive child abuse reports remain at home, but almost all CWS and SAT research is about placement into out-of- home care

Research Question Whether or not children of caregivers receiving substance abuse services are living in a safe environment? Operationalized as: Does substance abuse treatment for caregivers affect the risk of child maltreatment re-reports… … by caregiver and family? … by caregiver alone?

Counterfactuals and Propensity Score Matching Theory of Counterfactuals The fact is that some people who could use substance abuse treatment (SAT) receive it and some do not. The key assumption of the counterfactual framework is that individuals selected into treatment and nontreatment groups have potential outcomes in both states: the one in which they are observed and the one in which they are not observed (Winship & Morgan, 1999).

Types of Selection Bias Likely at Play in this Study Self selection (clients influencing the service they receive), Initial attendance Quitting treatment Bureaucratic selection (staff in service programs make decisions based on agency guidelines or more implicit decision rules [e.g., the availability of treatment]), Geographic selection (some services are only available in some areas).

Two Types of Selection Bias Especially Germane to SAT and CWS Research Creaming: Best cases are selected into programs because most disorganized clientele cannot manage program participation Souring (triaging): Most difficult cases are triaged into services--sometimes required by law to participate—so that the treatment group may be likely to look worse at the outset and after treatment

PSM in a Nutshell Employs a predicted probability of group membership—e.g., treatment vs. control group-- based on observed predictors, usually obtained from logistic regression to create a counterfactual group Propensity scores may be used for matching and as covariates in secondary analysis Time to first re-report—event history analysis

First National Random Sample Study Of CWS Extended Research Team includes: Research Triangle Institute University of North Carolina Caliber Associates San Diego Children’s Hospital, CASRC CSRD, Pitt Medical Center Duke Medical Center National Data Archive on Child Abuse and Neglect, Cornell 92 Local Child Welfare Agencies Admin. For Children and Families Children and Families

NSCAW Child Sampling is Complex Stratified: 8 “Certainty” States and remainder (28) states 92 PSUs (basically, county agencies) Over Sampled on the basis of: Children/Families Receiving ServicesChildren/Families Receiving Services InfantsInfants Sexually Abused ChildrenSexually Abused Children Not Sampled on the basis of: Substantiated Reports (cases are included whether abuse allegations are substantiated or not)

Creating the Propensity Score: Dependent Variable AOD service receipt Stayed overnight in AOD treatment program Clinic or doctor CWW reports CG received treatment CG reported “currently receiving treatment”

Creating the Propensity Score: Predictors Marital status Education Employment Poverty Case status Child race/ethnicity Child age Caregiver age Trouble paying for basic necessities CG mental health CG arrest/jail time Prior AOD treatment Maltreatment type

Independent Variables (cont’d) Need for treatment Risk assessment CIDI-SF Screen, Dependence CG report of need

Data & Study Sample NSCAW data from two waves: baseline ( ) and the 18-month follow-up. The sample for this study was limited to families where: children lived at home (n=4034) primary caregivers were female (n=3670) the primary caregiver had at least one indicator of a substance abuse problem (n=1472) there was a non-missing value of AOD service receipt and a non-missing value of time to first re-report (n=1074) Of these children, 276 (26%) formed the “treatment” group, and 798 (74%) comparison group.

Selecting the Best Model 12 matching schemes (using Mahalanobis distance and nearest neighbor matching) were used with different tolerances and with and without including the propensity score as a covariate. Differences in KS survivor functions were all in the same direction; that is, the treated group has a faster rate of re-report than the nontreated group. Did not use Heckman’s difference-in-difference because the results are not amenable to event history analysis Nearest neighbor matching within a caliper of.25 of the standard deviation of the logit Retained the most cases Had the fewest remaining statistical differences between items Had a broad common support area.

Histogram of Predicted Probabilities of Using SAT By Treatment Group Before Matching

Sample Comparison: Demographics* Before matching (n=1074) After matching (n=276) Child racep=.02p=.50 Child agep<.0001p=.96 Caregiver agep=.42p=.98 Educationp=.03p=.47 Employedp<.001p=.45 Povertyp=.44p=.33 Marriedp=.95p=1.0 Open CW casep<.0001p=.89 Type of maltreatmentp<.0001p=.99 *Comparing AOD treatment (yes/no)

Sample Comparison: Caregiver Risks and Need* Before matching (n=1074) After matching (n=276) Trouble paying for basic necessities p=.05p=.90 Mental health problemsp<.0001p=.45 Recent arrest /jail timep<.0001p=.63 Prior AOD treatmentp<.0001p=.29 Need (risk assessment)p<.0001p=.28 Need (CIDI-SF)p=.0008p=.08 Need (caregiver report)p<.0001p=.29 *Comparing AOD treatment (yes/no)

Description of Final Matched Sample Demographics (%) Child race Black30 White54 Hispanic12 Native American4 Child age Married (yes)25 Employed (yes)59

Final Matched Sample (n=276) Demographics (%) Caregiver age < Caregiver education No degree44 High school degree/GED39 Bachelor’s degree or higher17 Mental health problem36 Recent arrest /jail time52 Prior AOD treatment20

Description of Final Matched Sample Demographics (%) Type of maltreatment Physical abuse18 Sexual abuse7 Failure to provide28 Failure to supervise34 Other13 Poverty rate <50%33 50 to < 100% to <150% to <200%8 200% and greater12

Description of Final Matched Sample: Need for AOD Treatment Need (%) Risk assessment47 CIDI-SF (Screening or Dependence) AOD63 Caregiver report of need9

Differences Between Final Matched Sample and Eligible Sample (Generalizability Test)* The final matched sample was significantly more likely than the eligible sample to have: no degree (44% vs. 35%), p=.05 been recently arrested (52% vs. 42%), p=.005 a need for treatment according to the risk assessment (47% vs. 38%), p=.01 The final matched sample was significantly less likely to have a need for treatment according to their self report on the CIDI-SF (63% vs. 73%), p=.001 *Compares 1472 in total population of female caregivers with substance abuse problem with final matched sample of 276 in treatment group and PSM-matched nontreatment group

Research Question Whether or not children of caregivers receiving substance abuse services are living in a safe environment? Operationalized as: Does substance abuse treatment for caregivers affect the risk of child maltreatment re-reports… … by caregiver and family? … by caregiver alone?

Analysis: Kaplan Meier and Cox Regression Dependent variable(s) Time to first re-report (18 month follow-up) by: Caregiver or relative Caregiver alone Independent variables for Cox Regression AOD service receipt Child age CG age Prior child welfare services Open child welfare case Family cumulative risk Urban/rural

Findings EHA of the “best” PSM resample shows that the group difference on survivor functions are statistically significant with recipients of SAT having a greater hazard of re-reports than the nontreatment comparison group (roughly 20% vs. 10% during the 18-months). Time to re-report by caregivers or relatives (p <.02) Time to re-report by caregivers (p <.05)

Percent Children Remaining With No Re-report on Caregivers Substance Abuse Sample (n=1,074) Resampled with PSM (n=276) The proportion of children re-reported within 18-months was higher for SAT after the PSM correction (19 % vs. 10%) but not before (18% vs. 18%)

Estimated Hazard Ratios: Cox Regression based on the Resample Whether we use a narrow definition of who was source of new child abuse allegation or not: SA treatment group had twice the hazard Prior CWS involvement was related, but current Open CWS cases were not related

Limitations of PSM^ Large samples are required The sample size did drop, but is still substantial (unweighted n=276) Group overlap must be substantial before matching There were significant differences between the groups prior to PSM About half of the SAT cases were excluded from the final models Hidden bias may remain because matching only controls for observed variables (to the extent that they are perfectly measured) We based our variable selection on a thorough review of predictors of service use Although we have 14 indicators of the need for substance abuse services, this information is still not perfectly measured ^ Shadish, Cook, & Campbell, 2002

Conclusions Caregivers with allegations of child abuse and neglect against them, and with substance abuse problems, often remain at home following the CWS investigation. Children whose female caregivers receive substance abuse treatment are more likely to receive a subsequent child maltreatment report in the following 18-months: Whether the source of the report includes all relatives or is restricted to the caregiver, and Whether or not the selection bias into SAT is controlled for using PSM methods.

Possible Mechanisms The higher rate of re-reports may be attributable to some of the following mechanisms, each of which was tested and found not to differ between the SAT and nontreatment group: The type of maltreatment at re-report The average CIDI-SF score for alcohol and drug use, across the two waves Children’s behavior on the CBCL (ages 2 and up) Children’s reports (ages 11 and up) with regard to harsh or severe parenting

Alternate Explanations for Findings PSM did not match for important unobserved covariates Services may not have been used in full Substance abuse services may interfere with parental adequacy Focus is on parent’s recovery not child’s welfare Time and effort for SAT can be burdensome to parent Services may result in greater surveillance which results in more observed behaviors that might place children at risk, thus more reports But there were also more placements into foster care (p <.05).

Implications Children whose caregivers receive SAT may be safer if the early re-reports were for failure to participate in substance abuse treatment and prevented abuse or neglect Children whose caregivers receive SAT may be less safe if the re-reports followed abuse or neglect Caregivers who are involved with SAT may increase their likelihood of remaining involved with CWS, at least for 18-months Much more needs to be known about the impact of SAT on child welfare caregivers and children This study suggests that it is not beneficial for reducing maltreatment reports and preserving families

Partial References Gregoire, K. A., & Schultz, D. J. (2001). Substance-abusing child welfare parents: Treatment and child placement outcomes. Child Welfare, 80(4), Leathers, S. J. (2002). Parental visiting and family reunification: Could inclusive practice make a difference? Child Welfare, 81(4), Rosenbaum, P. R., & Rubin, D. B. (1985a). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. American Statistician, 39, Semidei, J., Radel, L. F., & Nolan, C. (2001). Substance abuse and child welfare: Clear linkages and promising responses. Child Welfare, 80(2), Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi- experimental designs for generalized causal inference. Boston: Houghton Mifflin. Sosin, M. R. (2002). Outcomes and sample selection: The case of a homelessness and substance abuse intervention. British Journal of Mathematical and Statistical Psychology, 55, U.S. Department of Health and Human Services. (1999, April). Blending perspectives and building common ground. A report to Congress on substance abuse and child protection. Retrieved May 23, 2000,

PSM and Project Resources This Presentation Substance Abuse and Child Welfare Services: Research Update and Needs (2003) Introduction to PSM: A New Device for Program Evaluation (2004) Substance Abuse among Caregivers Involved with Child Welfare Services: Prevalence and Identification by Child Welfare Workers (2004) Substance Abuse Needs and Services for Families Involved in the Child Welfare System (brief project description and link to other project materials) Substance Abuse Needs and Services for Families Involved in the Child Welfare System

Thank you very much