1. A Method for and Results from Evaluating a Multi-site After-School Program:
Combining a Regression Discontinuity Design with Contextual Effects Modeling

2. Authors: Angelica Adams, Communities in Schools of Houston
Heidi Brasher, YMCA of Greater Houston
Amy Corron, United Way of Greater Houston
Emily Faron, United Way of Greater Houston
Karla Kessler, Alief Independent School District
Cheryl McCallum, Children’s Museum of Houston
Roger Durand, Presenter and Evaluator, Durand Research and Marketing Associates, LLC

3. Purposes
1. To explain and demonstrate a new method for estimating separate and joint effects of site and individual participants’ characteristics on program outcomes.
2. To discuss evaluation findings resulting from applying the method in a multi-site, after-school setting

4. The Houston’s Kids After-School Program
Implemented in the aftermath of Hurricane Katrina
A collaboration of –
The United Way of Greater Houston
The Children’s Museum of Houston
The YMCA of Greater Houston
The Alief Independent School District
Communities in Schools of Houston
Program purpose: To address the needs of displaced and at-risk youth
Currently operates on six school campuses

5. Evaluation Questions
How are the outcomes of an after-school program influenced by active, regular participation over an extended time period?
Does the environment or “context” of the day-school affect individuals’ after-school program outcomes?

6. Data and Methods -- 1
Data on the characteristics and outcomes of regular (30 days or more per year) after-school program participants (N=111) were gathered over a two academic-year period.
Data were from a true “panel” of participants
Additional data were gathered on the context of the six day-school sites on which the program operated. Contextual data included the proportion of students at the school who were economically disadvantaged; at-risk; English language proficient; residentially mobile; and who successfully passed the state’s mandatory reading test.

7. Data and Methods - 2
Regression discontinuity methods (with a one-year program “hard” cut point)
First and second program-year outcome levels and slopes estimated for individuals’ outcomes were treated as dependent variables.
Contextual variables were treated as independent predictors of individuals’ outcomes

8. Principal Findings
Rates of improvement in math test scores among after-school participants were found inversely related to higher rates of day- school residential mobility and a greater proportion of students with limited English proficiency. Rates of improvement in math test scores among after-school participants were found positively (directly related) to higher rates of the day-school “pass proportion” on state-mandated reading tests. After-school participants in programs on day-school campuses with higher proportions of economically disadvantaged and at- risk students declined more in their levels of math test scores compared to students on other campuses.

9. Discussion – Part 1
The day-school campus context, especially economic disadvantages and the level of at- risk behavior, seems to make a difference to rates of improvement or decline in math achievement among individual, after-school program participants.
Are the findings consistent with the conclusions of the original (1966) “Coleman Report”?

10. Discussion – Part 2
The results suggest a “summer learning loss,” at least in level of math achievement, among after-school participants that is related to the day-school context.
The major lesson: future evaluations of after-school programming need to consider the day-school context. It seems hardly surprising that modifying the “student mix” at a school campus may pay rich after-school learning dividends.

11. Resources
Contact for this paper: Roger Durand,
References:
Scott-Little C and others, "Evaluations of After-School Programs: A Meta- Evaluation of Methodologies and Narrative Synthesis of Findings" American Journal of Evaluation, December 2002
Louie J, Rhoads C, and Mark J (2016), Challenges to using the regression discontinuity design in educational evaluations, American Journal of Evaluation, pp
Konishi C and others (2017), Investigating associations between school climate and bullying in secondary schools: Multilevel contextual effects modeling, School Psychology International, of+schools).