1. Measuring the Impact of an Out-of- School Program on Young Children Amy Corron, United Way of Greater Houston *Roger Durand, University of Houston-Clear Lake Emily Gesing, United Way of Greater Houston Julie Johnson, Communities in Schools, Houston Kevin Kebede, Alief YMCA Jennifer Key, Alief Independent School District Linda Lykos, YMCA of Greater Houston Cheryl McCallum, Children’s Museum of Houston. *Presenting for the group

2. Background Measuring the impacts of out-of-school-time programs on young children is known to be quite difficult. Problems encountered typically include a lack of reliability in measures, little construct validity, and biased results attributable to a lack of understanding and response among young children. This is especially true of children in grades k- 4, an important group in our program, known as “Houston’s Kids.”

3. About our presentation… In this presentation we will share our experiences, observations and findings in measuring the impact of the Houston’s Kids program on young children in grades k-4.

4. The Houston’s Kids Program Developed in the aftermath of Hurricanes Katrina and Rita to help meet the needs of at-risk children and youth The collaborating partners: The Alief Independent School District; The Children’s Museum of Houston; Communities in Schools; the United Way of Greater Houston; the YMCA of Greater Houston; Alief YMCA.

5. The Houston’s Kids Program -- 2 Program elements Based on Search Institute’s “Developmental Assets” Program goals and success standards

6. The Outcomes Evaluation Designed to assess the impact of the program on changes in math and reading test scores, school attendance, conduct grades, developmental assets and values. True panels of data that tracked changes in the same individuals over time were used. Comparison or “control subjects” were employed

7. Measuring Impacts among Young Children School district reading test scores, math test scores, school absences, conduct grades Program participation (attendance) Surveys of parents Summaries of focus groups conducted by counselors Baseline and follow-up “sticker surveys”

8. Measurement Problems Missing data Student dropouts (panel attrition) Different conduct grading scales No true baseline for school district data Program counselors turnover and difficulties in observing groups Program participation – intermittent attendance and drop outs Lack of correspondence between school test scores and parents’ reports Reading and math test unreliability and invalidity????

9. What Worked Well…(What We were Confident in Measuring) Baseline and follow-up “sticker surveys” Establishing the relationship between program participation and school absences

10. The Biggest Lessons….(and ‘take- aways”) The importance of estimating true change from response uncertainty (random error) in analyzing panel data on young children. Measurement error and mis-estimating program impacts. An ideal vehicle: P.F.Lazarsfeld and others, latent probability models. Reference: Lazarsfeld, P.F. and Henry, N., Latent Structure Analysis. Boston: Houghten- Mifflin, 1968.