A Method for and Results from Evaluating a Multi-site After-School Program: Combining a Regression Discontinuity Design with Contextual Effects Modeling

Authors: Angelica Adams, Communities in Schools of Houston Heidi Brasher, YMCA of Greater Houston Amy Corron, United Way of Greater Houston Emily Faron, United Way of Greater Houston Karla Kessler, Alief Independent School District Cheryl McCallum, Children’s Museum of Houston Roger Durand, Presenter and Evaluator, Durand Research and Marketing Associates, LLC

Purposes 1. To explain and demonstrate a new method for estimating separate and joint effects of site and individual participants’ characteristics on program outcomes. 2. To discuss evaluation findings resulting from applying the method in a multi-site, after-school setting

The Houston’s Kids After-School Program Implemented in the aftermath of Hurricane Katrina A collaboration of – The United Way of Greater Houston The Children’s Museum of Houston The YMCA of Greater Houston The Alief Independent School District Communities in Schools of Houston Program purpose: To address the needs of displaced and at-risk youth Currently operates on six school campuses

Evaluation Questions How are the outcomes of an after-school program influenced by active, regular participation over an extended time period? Does the environment or “context” of the day-school affect individuals’ after-school program outcomes?

Data and Methods -- 1 Data on the characteristics and outcomes of regular (30 days or more per year) after-school program participants (N=111) were gathered over a two academic-year period. Data were from a true “panel” of participants Additional data were gathered on the context of the six day-school sites on which the program operated. Contextual data included the proportion of students at the school who were economically disadvantaged; at-risk; English language proficient; residentially mobile; and who successfully passed the state’s mandatory reading test.

Data and Methods - 2 Regression discontinuity methods (with a one-year program “hard” cut point) First and second program-year outcome levels and slopes estimated for individuals’ outcomes were treated as dependent variables. Contextual variables were treated as independent predictors of individuals’ outcomes

Principal Findings Rates of improvement in math test scores among after-school participants were found inversely related to higher rates of day- school residential mobility and a greater proportion of students with limited English proficiency. Rates of improvement in math test scores among after-school participants were found positively (directly related) to higher rates of the day-school “pass proportion” on state-mandated reading tests. After-school participants in programs on day-school campuses with higher proportions of economically disadvantaged and at- risk students declined more in their levels of math test scores compared to students on other campuses.

Discussion – Part 1 The day-school campus context, especially economic disadvantages and the level of at- risk behavior, seems to make a difference to rates of improvement or decline in math achievement among individual, after-school program participants. Are the findings consistent with the conclusions of the original (1966) “Coleman Report”?

Discussion – Part 2 The results suggest a “summer learning loss,” at least in level of math achievement, among after-school participants that is related to the day-school context. The major lesson: future evaluations of after-school programming need to consider the day-school context. It seems hardly surprising that modifying the “student mix” at a school campus may pay rich after-school learning dividends.

Resources Contact for this paper: Roger Durand, RogerDurand21@Hotmail.com References: Scott-Little C and others, "Evaluations of After-School Programs: A Meta- Evaluation of Methodologies and Narrative Synthesis of Findings" American Journal of Evaluation, December 2002 Louie J, Rhoads C, and Mark J (2016), Challenges to using the regression discontinuity design in educational evaluations, American Journal of Evaluation, pp. 381-407. Konishi C and others (2017), Investigating associations between school climate and bullying in secondary schools: Multilevel contextual effects modeling, School Psychology International, http://journals.sagepub.com/action/doSearch?AllField=contextual+models +of+schools).