A Method for and Results from Evaluating a Multi-site After-School Program: Combining a Regression Discontinuity Design with Contextual Effects Modeling.

Slides:



Advertisements
Similar presentations
VALUE – ADDED 101 Ken Bernacki and Denise Brewster.
Advertisements

Correlational Designs
Catherine Cross Maple, Ph.D. Deputy Secretary Learning and Accountability
Making all research results publically available: the cry of systematic reviewers.
Collaborative Closing the Gap Action Plans: School Counselors, School Social Workers and School Psychologists Working to Close the Gaps.
The Bucks County Montessori Charter School PSSA Results, Local District Comparisons, and Year to Year Progressions.
Father Involvement and Child Well-Being: 2006 Survey of Income and Program Participation (SIPP) Child Well-Being Topical Module 1 By Jane Lawler Dye Fertility.
 ESL program is one that “provides instruction in the English language and other courses of study using teaching techniques for acquiring English, and...
% behaviour incidents cumulative % pupils Concentration of behaviour incidents: Percentage of pupils accounting for percentage of incidents 0.15% of pupils.
L 1 Chapter 12 Correlational Designs EDUC 640 Dr. William M. Bauer.
1 Data Linkage for Educational Research Royal Statistical Society March 19th 2007 Andrew Jenkins and Rosalind Levačić Institute of Education, University.
Research Study by Michela DeBari.  In many districts, at the middle school level, foreign language classes, often have advanced students, who have had.
 Neptune City Feasibility Study: An Analysis of the Educational, Fiscal and Organizational Options Available to Neptune City Board of Education R & R.
New Directions in Pupil Learning Assessment: Emerging Models from the Teachers for a New Era Initiative California State University, Northridge Beverly.
Ensuring Progress in the General Education Curriculum ED 222 Spring 2010.
University of Colorado at Boulder National Center for Research on Evaluation, Standards, and Student Testing Challenges for States and Schools in the No.
Stephen Nkansah-Amankra, PhD, MPH, MA 1, Abdoulaye Diedhiou, MD, PHD, H.L.K. Agbanu, MPhil, Curtis Harrod, MPH, Ashish Dhawan, MD, MSPH 1 University of.
Engaging Youth in Process and Outcomes Evaluations of an Out-of-School-Time Program Amy Corron, United Way of Greater Houston Roger Durand, University.
Transformational Leadership Group of Metropolitan Nashville Public Schools Performance of Disadvantaged Youth November 23, 2010 Brenda Steele, Co-Chair.
Measuring the Impact of an Out-of- School Program on Young Children Amy Corron, United Way of Greater Houston *Roger Durand, University of Houston-Clear.
Can investing in teachers raise outcomes in disadvantaged schools? Anahita Assadi, Caroline Hall, Martin Lundin och Kristina Sibbmark IFAU.
Amy Corron, United Way of Greater Houston *Roger Durand, University of Houston-Clear Lake Emily Gesing, United Way of Greater Houston Julie Johnson, Communities.
SSH integration in climate action CARISMA project Noriko Fujiwara Centre for European Policy Studies Expert meeting on SSH in H2020 Brussels, 5 July, 2016.
Amy Corron, United Way of Greater Houston Roger Durand, University of Houston and Durand Research and Marketing Associates, LLC** Julie Johnson, Communities.
Action Research for School Leaders by Dr. Paul A. Rodríguez.
Phillip Decker University of Houston-Clear Lake Roger Durand**
Participants & Procedure
Data Driven Decisions for School Improvement
Fundamentals of Educational Research
Louisiana's Title I, Part C, Migrant Education Program (MEP)
Effective factors that increase academic achievement among low SES youth Alex Anley November 30, 2016.
Manuel Angel Oscós-Sánchez MD Sandra K. Burge PhD
Collaborative Projects
American Evaluation Association Conference November 10, 2010
How to read a paper D. Singh-Ranger.
Butte Falls Charter School Open House & Annual Title I Meeting
Hannah E. Frank, Lisa Saldana, Philip C. Kendall, Holle Schaper
Evaluation of a Multi-Site, Out-of-School-Time Program: Contextual, Individual, and Combined Influences on Outcomes Amy Corron, United Way of Greater Houston.
Immersion Programs in China: A Case Study
Differential Outcomes of An After-School Program:
Effective educational strategies of resilient schools
Understanding Your Child’s Report Card
Student Homelessness in NYC
Audit Quality Of Outsourced Information Technology Controls
January 2018 Institute for Children, Poverty and Homelessness
Building Customer Relationships Through Effective Marketing
House Bill 22 Overview ESC PEIMS Coordinator Summer Training | August 1, 2017 Texas Education Agency | Academics | Performance Reporting.
Evaluation of the Wisconsin Educator Effectiveness System Pilot: Results of the Teacher Practice Rating System Pilot Curtis Jones, UW Milwaukee Steve.
Afterschool Programs That Follow Evidence-based Practices to Promote Social and Emotional Development Are Effective Roger P. Weissberg, University of Illinois.
IEPI – Participate | Collaborate | Innovate
The Role a Charter School Plays in its Charter Authorizer’s Submission of the Consolidated Federal Programs Application Joey Willett, Unit of Federal Programs.
School’s Cool Makes a Difference!
Let’s make education fairer: Disadvantage, school intakes and outcomes
Problems, Purpose and Questions
Campus Improvement Planning
Who are we? What’s our story?
Taylor ISD Title I Parent Meeting
T.R.U.E. Survey: INPUT ON SCHOOL RECONFIGURATIONS March 2019
State of the School Title I Meeting Folwell School, Performing Arts Magnet October 9, /8/2019.
Specifying a Purpose and Research Questions or Hypotheses
Andrew Jenkins and Rosalind Levačić
Engaging Migrant Parents in Meaningful Parent Involvement
Both of these experiments are investigating the effects of music on learning, specifically whether music can improve learning. Each experiment has been.
State Mandates Teachers Edition
Title 1 Annual Parent Meeting
Disproportionate Impact Study
beating the Barriers at HCTC:
Title I Document Training, Revision, Input Meeting
ELLs in Texas: What Teachers Need to Know
Significant Disproportionality Fiscal Webinar
Presentation transcript:

A Method for and Results from Evaluating a Multi-site After-School Program: Combining a Regression Discontinuity Design with Contextual Effects Modeling

Authors: Angelica Adams, Communities in Schools of Houston Heidi Brasher, YMCA of Greater Houston Amy Corron, United Way of Greater Houston Emily Faron, United Way of Greater Houston Karla Kessler, Alief Independent School District Cheryl McCallum, Children’s Museum of Houston Roger Durand, Presenter and Evaluator, Durand Research and Marketing Associates, LLC

Purposes 1. To explain and demonstrate a new method for estimating separate and joint effects of site and individual participants’ characteristics on program outcomes. 2. To discuss evaluation findings resulting from applying the method in a multi-site, after-school setting

The Houston’s Kids After-School Program Implemented in the aftermath of Hurricane Katrina A collaboration of – The United Way of Greater Houston The Children’s Museum of Houston The YMCA of Greater Houston The Alief Independent School District Communities in Schools of Houston Program purpose: To address the needs of displaced and at-risk youth Currently operates on six school campuses

Evaluation Questions How are the outcomes of an after-school program influenced by active, regular participation over an extended time period? Does the environment or “context” of the day-school affect individuals’ after-school program outcomes?

Data and Methods -- 1 Data on the characteristics and outcomes of regular (30 days or more per year) after-school program participants (N=111) were gathered over a two academic-year period. Data were from a true “panel” of participants Additional data were gathered on the context of the six day-school sites on which the program operated. Contextual data included the proportion of students at the school who were economically disadvantaged; at-risk; English language proficient; residentially mobile; and who successfully passed the state’s mandatory reading test.

Data and Methods - 2 Regression discontinuity methods (with a one-year program “hard” cut point) First and second program-year outcome levels and slopes estimated for individuals’ outcomes were treated as dependent variables. Contextual variables were treated as independent predictors of individuals’ outcomes

Principal Findings Rates of improvement in math test scores among after-school participants were found inversely related to higher rates of day- school residential mobility and a greater proportion of students with limited English proficiency. Rates of improvement in math test scores among after-school participants were found positively (directly related) to higher rates of the day-school “pass proportion” on state-mandated reading tests. After-school participants in programs on day-school campuses with higher proportions of economically disadvantaged and at- risk students declined more in their levels of math test scores compared to students on other campuses.

Discussion – Part 1 The day-school campus context, especially economic disadvantages and the level of at- risk behavior, seems to make a difference to rates of improvement or decline in math achievement among individual, after-school program participants. Are the findings consistent with the conclusions of the original (1966) “Coleman Report”?

Discussion – Part 2 The results suggest a “summer learning loss,” at least in level of math achievement, among after-school participants that is related to the day-school context. The major lesson: future evaluations of after-school programming need to consider the day-school context. It seems hardly surprising that modifying the “student mix” at a school campus may pay rich after-school learning dividends.

Resources Contact for this paper: Roger Durand, RogerDurand21@Hotmail.com References: Scott-Little C and others, "Evaluations of After-School Programs: A Meta- Evaluation of Methodologies and Narrative Synthesis of Findings" American Journal of Evaluation, December 2002 Louie J, Rhoads C, and Mark J (2016), Challenges to using the regression discontinuity design in educational evaluations, American Journal of Evaluation, pp. 381-407. Konishi C and others (2017), Investigating associations between school climate and bullying in secondary schools: Multilevel contextual effects modeling, School Psychology International, http://journals.sagepub.com/action/doSearch?AllField=contextual+models +of+schools).