1. HSRI Julie Murphy and Linda Newton-Curtis AEA, 2011
A Discussion on Measuring the Black Box of Services in Human Services Evaluation
HSRI
Julie Murphy and Linda Newton-Curtis
AEA, 2011

2. Black Box of Services OUTCOMES INTERVENTION
Goal of Evaluation: To examine impact of a particular intervention.
Process/Implementation Study Outcomes Study
Describe what it is that is offered to clients that is different from the status quo and what impact does this have on outcomes?
In CW, this means gaining a comprehensive picture of services and supports that clients receive may be difficult when families refer to outside resources.
In child welfare describing what families receive can be difficult given the scope of the intervention
- FTM: intervention is easy to define and measure: number of meetings, timing, who attends, etc.
-DR, KN: intervention is focused on helping families with complex needs access a variety of services they need, offered by the agency or out in the community
Discussion around defining and measuring the ‘black box’ of external services considered a key contributor to the success of family outcomes.

3. Efforts to Collect Services Data
Family Interviews
Case-level caseworker survey
Context: two project where intervention involved facilitating families in addressing barriers by linkages to services and supports
Family Interviews: PO Kinship: representativeness (62 caregivers), asked what assistance did they receive, what was helpful, what would have been helpful.
Case-Level Caseworker Survey: PO Kinship & DR: Completed by staff familiar with the case. Reliant of SACWIS and other data systems, but data is entered in a variety of locations, under different members of the familily, not documented. Reliant on worker memory. Kinship CR-by service type, date and $ amt ; DR CR- Did you help members of this family receive any of the following services
Family Survey:
DR: Did your caseworker help you get any of the following services
KN: Family Resource Scale: how well each need is met on a regular basis (never, always, sometimes)
Case Record Review: PO MH/SA: to gather detailed case-level info on MHSA services to a representative sample of cases, what and how quickly services were provided. Very labor intensive, services may not have been noted in documentation, especially if outside the cw agency
DR Cost Study- monitor as services provided, quanity or amount, tallied on a monthly basis
Family Surveys
Case Record Reviews
Cost Study

4. Challenges Scope of Intervention Definition of Services
Level of Measurement
Quality of Data
Scope of Intervention: tight (FTM) vs. broad (DR, Kinship) model (systemic intervention to facilitate families to receive assistance in community).
Definition of Services: What is considered a service? Case management? Foster care, a gift card?
Level of measurement:
-Scope: how big do you spread the net? paid for by agency, provided by agency staff, voluntary services, referred to?
-Dosage, quality/engagement of services, knowledge of need, list of included services, scale vs. yes/no
Quality of Data: representativeness, level of effort, lack of documentation/knowledge of service receipt, qualitative(helpful for program improvement) vs. quantitative (relationship to outcomes)

5. Discussion Questions Introduction and share experiences
Identification of key services: Discuss how to:
Select and prioritize key services to be measured
Elicit feedback from programmatic staff to define the list of key services
Defining and categorization of services.
Measurement: Discuss how to determine:
What to measure (e.g. service needed, referred, received)?
Dosage? Engagement of client in service? Quality of delivery?
Data collection: what data collection methods are used? Administrative data sources, secondary data sources, case audits? Creating incentives for local staff to provide this data.
First introduce self and talk about your experiences with collecting services data themselves