Amy Corron, United Way of Greater Houston *Roger Durand, University of Houston-Clear Lake Emily Gesing, United Way of Greater Houston Julie Johnson, Communities in Schools, Houston Kevin Kebede, Alief YMCA Jennifer Key, Alief Independent School District Linda Lykos, YMCA of Greater Houston Cheryl McCallum, Children’s Museum of Houston. *Presenting for the group An Evaluation of "Dose-Response Relationships" in an Out-of-School-Time Program

About this presentation…. In this presentation, we will report the results of an evaluation of “dose- response relationships” (participation levels and program outcomes) in an out-of-school-time program. We will present findings from a large, multi-faceted program known as “Houston’s Kids.” We will also discuss the program design and management implications of our findings.

The Houston’s Kids Program Developed in the aftermath of Hurricanes Katrina and Rita to help meet the needs of at-risk children and youth The collaborating partners: The Alief Independent School District; The Children’s Museum of Houston; Communities in Schools; the United Way of Greater Houston; the YMCA of Greater Houston; Alief YMCA.

The Houston’s Kids Program - 2 Program elements Based on Search Institute’s “Developmental Assets” Program goals and success standards

The Outcomes Evaluation Designed to assess the impact of the program on changes in math and reading test scores, school attendance, conduct grades, developmental assets and values. True panels of data that tracked changes in the same individuals over time were used (a prospective, longitudinal design). Comparison or “control subjects” were employed

Dose and Response: Program Participation and Changes in Math Scores Over the Program Program Participation Math Change

Dose and Response: Program Participation and Changes in Math Scores – Best Fitting Curve (Natural) Log linear relationship =.27

What does the best-fitting dose- response curve tell us? The impact of participating in the program on changes in math test scores attenuated over the program year. A “diminishing marginal effect” – increasing but at a decreasing rate of change Not a surprising result

Does –Response: Program Participation and School Absences – First Grading Period Best-fitting curve: linear = -.20 Program Participation School Absences

Dose-Response: Program Participation and School Absences – Grading Periods 2 to 4 Program Participation School Absences Best fitting curve: linear = 0.

What do the program participation and school absences curves tell us? A “step” function Participation in the Houston’s Kids Program initially reduced school absences: the greater the program participation, the fewer the absences. After initial impact, continued participation was found to have no impact on absences. (No marginal effect.)

Dose-Response: Program Participation and Changes in Assets

What does the previous table tell us? Assets shown in table increased monotonically (improved) with increased program participation: the more the participation in the program, the greater the positive change (improvement) in Search Assets. But note the period of change

Key findings….. Changes in school (math) test scores, absences, and developmental assets did not occur in the same way with regard to program participation. Changes in test scores, absences, and assets generally do not occur uniformly over the course of the program. Attenuated participation effects on program outcomes.

Implications for evaluation users…. These findings suggest that the amount of “out-of -school” program time that could or should be devoted to obtaining specific outcomes likely will vary by subject (e.g., math, importance of school, assets, etc). Attenuated participation effects suggest the need for modifying or redesigning content – or even “re-inventing” the program’s logic -- over the course of the program in order to enhance intended outcomes.