1/18/2019 1:17:10 AM1/18/2019 1:17:10 AM Discussion of “Strategies for Studying Educational Effectiveness” Mark Dynarski Society for Research on Educational Effectiveness Annual Meetings Landsdowne, VA December 2006 Roadmap/overview of what we’re going to talk about today. Begin with impact estimates for elementary schools In 1st report presented impact estimates for original set of 7 elementary grantees. Since then added 5 more elem. grantees. Today we’ll present impact estimates for full set of 12 elementary grantees in year 1, which is the first year after random assignment. Next we’ll present year 2 middle school outcome differences.

Themes 1/18/2019 1:17:10 AM1/18/2019 1:17:10 AM Alternatives to study effectiveness in situations where experiments may not be feasible or desirable Issues Focus on experiments should not be lost Prospective versus retrospective studies Costs and practicality A few notes before turning to elementary findings: Random assignment successful at aligning T and C groups. Next slide presents sample info. All estimates regression adjusted to improve precision of estimates. Focus on findings significant at 5% or better. Brief overview of findings here followed by more detail in following slides. Overall, impact findings changed little when we added the 5 new sites. Found moderate program attendance -- students attended ~ 2 days/week on average. Program affected students’ location and supervision after school. The program: Reduced care by parents and siblings Increased care by other adults Had no effect on self care. Multiple definitions used – all yielded same results. Reduced % of students in own homes after school Increased % of students at school after school On academic outcomes, found that: No improvement in grades or reading test scores Teachers were less likely to report that students often completed their homework Impact and T/C means similar to those from yr 1 but this year the result is significant. Additional sample reduced variability of estimate. Students reported feeling safer after school. Few impacts on developmental outcomes. Program increased % of students reporting that they helped others after school. No effect on other developmental outcomes such as working on teams, sticking to beliefs when others disagree, getting along with others. No effect on behavior outcomes such as suspensions, being sent to office, being disciplined in class. Some measures of parental involvement increased for program participants relative to control students. Also found parents of participants were more likely to be in the labor force.

Regression Discontinuities 1/18/2019 1:17:10 AM1/18/2019 1:17:10 AM + Being able to allocate services according to need is attractive - Less power and stronger validity conditions than experiments, does not support subgroup estimation as well as experiments Retrospective: if experiment is not feasible, is a score available? Can it be gamed? Prospective: If thinking about an RD study, the experiment may be a better choice (use risk strata) A few notes before turning to elementary findings: Random assignment successful at aligning T and C groups. Next slide presents sample info. All estimates regression adjusted to improve precision of estimates. Focus on findings significant at 5% or better. Brief overview of findings here followed by more detail in following slides. Overall, impact findings changed little when we added the 5 new sites. Found moderate program attendance -- students attended ~ 2 days/week on average. Program affected students’ location and supervision after school. The program: Reduced care by parents and siblings Increased care by other adults Had no effect on self care. Multiple definitions used – all yielded same results. Reduced % of students in own homes after school Increased % of students at school after school On academic outcomes, found that: No improvement in grades or reading test scores Teachers were less likely to report that students often completed their homework Impact and T/C means similar to those from yr 1 but this year the result is significant. Additional sample reduced variability of estimate. Students reported feeling safer after school. Few impacts on developmental outcomes. Program increased % of students reporting that they helped others after school. No effect on other developmental outcomes such as working on teams, sticking to beliefs when others disagree, getting along with others. No effect on behavior outcomes such as suspensions, being sent to office, being disciplined in class. Some measures of parental involvement increased for program participants relative to control students. Also found parents of participants were more likely to be in the labor force.

Propensity Scoring: Observations 1/18/2019 1:17:10 AM1/18/2019 1:17:10 AM It’s an interesting issue to debate It’s useful to have hundreds of undergrads to randomly assign If you are on the other side of a statistical argument with Don Rubin, you should reconsider A few notes before turning to elementary findings: Random assignment successful at aligning T and C groups. Next slide presents sample info. All estimates regression adjusted to improve precision of estimates. Focus on findings significant at 5% or better. Brief overview of findings here followed by more detail in following slides. Overall, impact findings changed little when we added the 5 new sites. Found moderate program attendance -- students attended ~ 2 days/week on average. Program affected students’ location and supervision after school. The program: Reduced care by parents and siblings Increased care by other adults Had no effect on self care. Multiple definitions used – all yielded same results. Reduced % of students in own homes after school Increased % of students at school after school On academic outcomes, found that: No improvement in grades or reading test scores Teachers were less likely to report that students often completed their homework Impact and T/C means similar to those from yr 1 but this year the result is significant. Additional sample reduced variability of estimate. Students reported feeling safer after school. Few impacts on developmental outcomes. Program increased % of students reporting that they helped others after school. No effect on other developmental outcomes such as working on teams, sticking to beliefs when others disagree, getting along with others. No effect on behavior outcomes such as suspensions, being sent to office, being disciplined in class. Some measures of parental involvement increased for program participants relative to control students. Also found parents of participants were more likely to be in the labor force.

Propensity scoring: Shadish and Clark 1/18/2019 1:17:10 AM1/18/2019 1:17:10 AM PS is based on a theorem and it has to work if assumptions are met Small bias (10%); unclear how scoring would perform with large bias Prospective: can be more expensive than an experiment (Rubin recomends potential comparison group 3 to 4 times larger than matched group) Retrospective: probably do not have data to model selection A few notes before turning to elementary findings: Random assignment successful at aligning T and C groups. Next slide presents sample info. All estimates regression adjusted to improve precision of estimates. Focus on findings significant at 5% or better. Brief overview of findings here followed by more detail in following slides. Overall, impact findings changed little when we added the 5 new sites. Found moderate program attendance -- students attended ~ 2 days/week on average. Program affected students’ location and supervision after school. The program: Reduced care by parents and siblings Increased care by other adults Had no effect on self care. Multiple definitions used – all yielded same results. Reduced % of students in own homes after school Increased % of students at school after school On academic outcomes, found that: No improvement in grades or reading test scores Teachers were less likely to report that students often completed their homework Impact and T/C means similar to those from yr 1 but this year the result is significant. Additional sample reduced variability of estimate. Students reported feeling safer after school. Few impacts on developmental outcomes. Program increased % of students reporting that they helped others after school. No effect on other developmental outcomes such as working on teams, sticking to beliefs when others disagree, getting along with others. No effect on behavior outcomes such as suspensions, being sent to office, being disciplined in class. Some measures of parental involvement increased for program participants relative to control students. Also found parents of participants were more likely to be in the labor force.

Using the methods to enhance experiments 1/18/2019 1:17:10 AM1/18/2019 1:17:10 AM Propensity scoring or HLM for dosage HLM for variable counterfactuals RD for testing model variations within a treatment group A few notes before turning to elementary findings: Random assignment successful at aligning T and C groups. Next slide presents sample info. All estimates regression adjusted to improve precision of estimates. Focus on findings significant at 5% or better. Brief overview of findings here followed by more detail in following slides. Overall, impact findings changed little when we added the 5 new sites. Found moderate program attendance -- students attended ~ 2 days/week on average. Program affected students’ location and supervision after school. The program: Reduced care by parents and siblings Increased care by other adults Had no effect on self care. Multiple definitions used – all yielded same results. Reduced % of students in own homes after school Increased % of students at school after school On academic outcomes, found that: No improvement in grades or reading test scores Teachers were less likely to report that students often completed their homework Impact and T/C means similar to those from yr 1 but this year the result is significant. Additional sample reduced variability of estimate. Students reported feeling safer after school. Few impacts on developmental outcomes. Program increased % of students reporting that they helped others after school. No effect on other developmental outcomes such as working on teams, sticking to beliefs when others disagree, getting along with others. No effect on behavior outcomes such as suspensions, being sent to office, being disciplined in class. Some measures of parental involvement increased for program participants relative to control students. Also found parents of participants were more likely to be in the labor force.