HSRI Julie Murphy and Linda Newton-Curtis AEA, 2011 A Discussion on Measuring the Black Box of Services in Human Services Evaluation HSRI Julie Murphy and Linda Newton-Curtis AEA, 2011

Black Box of Services OUTCOMES INTERVENTION Goal of Evaluation: To examine impact of a particular intervention. Process/Implementation Study Outcomes Study Describe what it is that is offered to clients that is different from the status quo and what impact does this have on outcomes? In CW, this means gaining a comprehensive picture of services and supports that clients receive may be difficult when families refer to outside resources. In child welfare describing what families receive can be difficult given the scope of the intervention - FTM: intervention is easy to define and measure: number of meetings, timing, who attends, etc. -DR, KN: intervention is focused on helping families with complex needs access a variety of services they need, offered by the agency or out in the community Discussion around defining and measuring the ‘black box’ of external services considered a key contributor to the success of family outcomes.

Efforts to Collect Services Data Family Interviews Case-level caseworker survey Context: two project where intervention involved facilitating families in addressing barriers by linkages to services and supports Family Interviews: PO Kinship: representativeness (62 caregivers), asked what assistance did they receive, what was helpful, what would have been helpful. Case-Level Caseworker Survey: PO Kinship & DR: Completed by staff familiar with the case. Reliant of SACWIS and other data systems, but data is entered in a variety of locations, under different members of the familily, not documented. Reliant on worker memory. Kinship CR-by service type, date and $ amt ; DR CR- Did you help members of this family receive any of the following services Family Survey: DR: Did your caseworker help you get any of the following services KN: Family Resource Scale: how well each need is met on a regular basis (never, always, sometimes) Case Record Review: PO MH/SA: to gather detailed case-level info on MHSA services to a representative sample of cases, what and how quickly services were provided. Very labor intensive, services may not have been noted in documentation, especially if outside the cw agency DR Cost Study- monitor as services provided, quanity or amount, tallied on a monthly basis Family Surveys Case Record Reviews Cost Study

Challenges Scope of Intervention Definition of Services Level of Measurement Quality of Data Scope of Intervention: tight (FTM) vs. broad (DR, Kinship) model (systemic intervention to facilitate families to receive assistance in community). Definition of Services: What is considered a service? Case management? Foster care, a gift card? Level of measurement: -Scope: how big do you spread the net? paid for by agency, provided by agency staff, voluntary services, referred to? -Dosage, quality/engagement of services, knowledge of need, list of included services, scale vs. yes/no Quality of Data: representativeness, level of effort, lack of documentation/knowledge of service receipt, qualitative(helpful for program improvement) vs. quantitative (relationship to outcomes)

Discussion Questions Introduction and share experiences Identification of key services: Discuss how to: Select and prioritize key services to be measured Elicit feedback from programmatic staff to define the list of key services Defining and categorization of services. Measurement: Discuss how to determine: What to measure (e.g. service needed, referred, received)? Dosage? Engagement of client in service? Quality of delivery? Data collection: what data collection methods are used? Administrative data sources, secondary data sources, case audits? Creating incentives for local staff to provide this data. First introduce self and talk about your experiences with collecting services data themselves