Meryle Weinstein, Emilyn Ruble Whitesell and Amy Ellen Schwartz New York University Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy October 3, 2012 Can Formal–Informal Collaborations Improve Science Literacy in Urban Middle Schools? The Impact of Urban Advantage Meryle Weinstein Emilyn Ruble Amy Ellen Schwartz

What is Urban Advantage? Collaboration between New York City Department of Education and 8 New York City informal science education institutions Led by American Museum of Natural History Provides professional development to middle school science teachers and opportunities to students to engage in authentic science practice Workshops for science teachers and school administrators Science materials/equipment for schools, teachers, & students Vouchers for class field trips, family field trips and visits Launched in with 31 schools and in had 137 Funded by NY City Council and DOE

Why Urban Advantage? Increased calls for collaboration between formal and informal institutions, particularly around science Growing aversion to science among students, particularly by the time they reach middle school Middle school is time to grab students attention – “gateway” for high school science courses Strong science instruction has been found to impact science persistence in high school

Our goal is to determine if the Urban Advantage program is effective. Does Urban Advantage lead to increased student achievement? UA students outperform non-UA students on 8 th grade Intermediate Level Science Test Modest impact Magnitude increases over time Students who attend UA schools are more likely to take and pass a Science Regents in 8 th or 9 th grade than students at non-UA schools

We make use of a rich longitudinal student level database. Student-Level Data NYCDOE Administrative Data, – Socio-demographic characteristics, educational needs, and test scores

We use multiple approaches to estimate the impact of UA. Compare the observable differences between students at UA schools and students at non-UA schools Use quasi-experimental techniques to estimate UA’s total effect on student achievement. Use a “difference-in-difference” approach.

Raw performance data suggests UA is effective Student Weighted Mean Achievement, 8 th Grade Intermediate Level Science (ILS) Test – Percent Proficient

Differences between UA and non-UA schools prior to joining UA are largely insignificant UANon-UAUANon-UAUANon-UA N of Schools Total Enrollment (434)(468)(426)(425)(647)(373) % Black (28.1) (29.3)(28.9)(34.0)(29.3) % Hispanic (22.9)(25.4)(27.5)(26.0)(23.2)(26.4) % Asian/Other (19.6)(12.1)(12.0)(13.3)(14.5)(14.1) % White (18.2)(19.6)(22.1)(18.0)(19.1)(18.0) % ELL (7.8)(10.6)(9.9)(11.1)(4.2)(11.7) % Free Lunch (21.8)(23.5)(23.1)(23.3)(31.5)(25.8) % Prof. ELA (16.6)(20.5)(20.4)(21.3)(18.9)(21.6) % Prof. Math (17.4)(20.6)(23.2)(22.2)(18.7)(23.5) % Prof. Science (20.9)(24.8)(23.0)(24.0)(27.7)(22.7) Standard deviations are in parentheses Bold indicates differences are statistically significant at.05 level or less % Proficient is the percent scoring in levels 3 or 4

Basic Model Y ijt = β 1i jt + β 2 PreUA ijt β 3 Post UA ijt + β 4 ST ijt + θ j + ε ijt  Y = individual student outcome  PreUA = indicator variable for whether school joined UA in next year  PostUA = indicator variable for whether school has joined UA  ST = vector of student characteristics  θ = school fixed effect   = random error term

Controlling for student characteristics, students at UA schools outperform those at non-UA schools. Overall, students at UA schools perform.041 sd higher than those at non-UA schools in science Little change is seen on ELA or math or 8 th grade students After their first year of joining UA, students at UA schools perform, on average,.056 sd higher than those at non-UA schools in science

Demographic Characteristics 8 th Grade Sample TotalUANon-UA % Black % White % Asian % Hispanic % Female % Poor % Spec. Ed % LEP % UA N N Schools

OLS Regression with School Fixed Effects, 2004 to 2010 ScienceMathELA Model 1Model 2 β /se Yr Prior UA (0.018)(0.021)(0.024)(0.017) UA in Any Year0.041* (0.016) Yr Ent. UA (0.024)(0.027)(0.021) Yr Post UA0.056* (0.028)(0.031)(0.023) Black-0.397*** ***-0.375*** (0.017) (0.022)(0.023) Hispanic-0.226*** ***-0.275*** (0.015) (0.020)(0.021) Asian0.162*** 0.407***0.064** (0.019) (0.027)(0.022) Female-0.072*** 0.027***0.194*** (0.005) (0.004) School FEYES R-Square N * p<0.05, ** p<0.01, *** p<0.001 Year, LEP, and Special Education dummies not shown Robust clustered standard errors in parentheses

Robustness Checks Controlled for prior achievement Magnitudes are smaller but still significant when controlling for prior math or reading scores Lagged Math Scores, Post UA Yrs β =.037, p <.10 Lagged Reading Scores, Post UA Yrs β =.045, p <.10 Caveat: Sample size decreases by 50,000 No statistically significant findings for percent proficient

Descriptive Statistics, High School Sample Full Sample UA Not-UA N=252,129N=79,090N= 173,039 Mean UA (0.46) % Black (0.46) (0.45) (0.48) % Hispanic (0.48) (0.49) (0.49) % Asian (0.37) (0.40) (0.36) % White (0.34) 14.7 (0.35) (0.33) % Poor (0.33) (0.38) (0.36) Attended a STEM High School 9.83 (0.30) 9.72 (0.30) 9.87 (0.30) Took Living Environment in 8 th or 9 th Grade (0.49) (0.47) (0.50) Took Earth Science in 8 th or 9 th Grade (0.31) (0.31) (0.31)

Linear Probability Coefficients, High School Outcomes Model 3Model 4 β /s.e Attending a STEM School 0.014*** (0.003) 0.008* (0.004) Attending a Partial STEM SchoolNS Taking Living Environment Regents in 8 th or 9 th Grade 0.255*** (0.012) 0.246*** (0.012) Passing Living Environment RegentsNS Passing Living Environment Regents with 65 or higher 0.040*** (0.006) 0.032*** (0.006) Passing Living Environment Regents with 85 or higher 0.062*** (0.005) 0.054*** (0.005) Taking Earth Science Regents in 8 th or 9 th Grade 0.039*** (0.007) 0.033*** (0.007) Passing Earth Science Regents 0.029*** (0.0006) 0.012* (0.0006) Passing Earth Science Regents with 65 or higher 0.059*** (0.007) 0.037*** (0.008) Passing Earth Science Regents with 85 or higher 0.062*** (0.005) 0.054*** (0.005) School Fixed EffectsYES * p<0.05, ** p<0.01, *** p<0.001 Robust clustered standard errors in parentheses Control variables not shown are: Black, Hispanic, Asian, Female, Poor, Special Education, LEP, and for Model 4 lagged_zmath.

Conclusions Student performance increases with the implementation of UA and the magnitude of the difference increases over time. Little change on ELA or math for 8th grade students, suggesting the effect is not merely reflecting coincident overall school improvement UA also contributes to post-8 th grade outcomes. Biggest impact is on the likelihood of taking the Living Environment Regents in 8 th or 9 th grade.

Policy Implications First estimates of the impact of a science program on academic achievement Inquiry as a method to approach science instruction is not emphasized in schools but more common in informal science institutions Benefits of collaboration between formal and informal science institutions Importance of strong partnerships between these institutions and between the institutions and the school district in which they work Need for improved data linking teachers and students Future research: Inside the black box