1. Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning David Figlio, Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research

2. Background Two major trends affecting higher education – Declining state and local appropriations  more fiscal challenges, increasing demands for “efficiency” (new NRC panel) – Rapid improvements in technology As a consequence, millions of students are now taking classes online

3. What do online classes look like? Two major types: – Innovative, highly interactive classes aimed at exploiting special nature of the internet and modern technology – Traditional lectures presented in an online format While door #1 is the type of class advocated by learning scientists (NB: I’m working with colleagues to develop this type of class in high school science), door #2 is what most universities are doing.

4. What is the causal question of interest? Do students learn more in a traditional lecture when it is broadcast on the internet than when it is presented in a standard lecture hall? In principle, the results could be positive or negative: – Pro: increased flexibility; no need to rely on others’ notes; ability to annotate/rewatch lectures – Con: increased barriers to student-faculty interaction; reduced ability to ask just in time questions; incentives to defer work

5. What is the causal question of interest? (part 2) Are there heterogeneous effects of internet- based traditional lectures on students? Why interesting? – Some groups may face additional communication barriers (e.g., language minority students) – Some groups may have lower self-regulation skills (e.g., college-aged men; relatively lower- achievers)

6. What is the evidence to date? Almost nothing Mainly small-scale case studies; unsurprisingly, almost all studies are based on type 1 of internet- based class rather than type 2 of internet-based class Somewhat larger studies have poor treatment- control contrast Considerable need for experimental evidence on both types of internet class, but especially the broadcast-traditional-lecture type  this study

7. What is the ideal experiment? Treatment and control students should be taught in tandem – Same instructor, same exams, same lectures, same supplementary material, same readings Pure randomization to live vs. internet treatment – No opt-out from experiment No opportunities for contamination of treatment and control – Live students cannot view internet lectures; internet students cannot attend live lectures Ample opportunities for detecting heterogeneous effects and improving external validity – Dozens of experiments in different courses at different institutions, with within-course randomization – Cluster-randomized design; clustering on subgroups to ensure sufficient sample size to detect subgroup-specific effects

8. Fidelity to ideal experiment Treatment and control students should be taught in tandem – Same instructor, same exams, same lectures, same supplementary material, same readings Pure randomization to live vs. internet treatment – No opt-out from experiment (instead: randomization of volunteers) No opportunities for contamination of treatment and control – Live students cannot view internet lectures; internet students cannot attend live lectures (live students may potentially watch with friends) Ample opportunities for detecting heterogeneous effects and improving external validity (nope: just one class, no group cluster) – Dozens of experiments in different courses at different institutions, with within-course randomization – Cluster-randomized design; clustering on subgroups to ensure sufficient sample size to detect subgroup-specific effects

9. Threats to internal and external validity (1) Volunteers, rather than pure randomization – How representative are volunteers of the potential study population at the institution? [external validity]  Table 1 – Knowledge that this is an experiment might lead to differential attrition of live vs. internet [internal validity]  Table 2 contrasts; bounding exercise in Table 3

10. Threats to internal and external validity (2) Fidelity of randomization and lack of treatment contamination – Do people drop from experiment post-randomization? [internal validity]  as 15 students assigned to “live” dropped from the experiment, Table 3 compares results treating defectors as “live” versus dropping from study – Do “live” students view internet version and do “internet” students attend live lecture? [internal validity] Door guards checked IDs so we know that no “internet” students attended live lectures Live students did not have online access, but could have accessed via friends’ log-ins  Figure 1 shows that “live” students attended substantially more lectures than “live+internet” non-volunteers

11. Threats to internal and external validity Intro economics might be special The university in question might be unusual The particular instructor may translate well/poorly to the internet platform  Lots of hand-wringing  And also a call for more experiments in other subjects and settings and with a design aimed at detecting heterogeneous treatment effects

12. Threats to internal and external validity (1) Volunteers, rather than pure randomization – How representative are volunteers of the potential study population at the institution? [external validity]  some differences; volunteers had higher GPAs and lower SAT scores and mom was less likely to be a college grad. – Knowledge that this is an experiment might lead to differential attrition of live vs. internet [internal validity]  no evidence of differential attrition: 6 live, 10 online attriters, and no differences between them. Bounding exercise (giving attriters scores of 0 or 100 on missed exam) shows tight bounds when considering attrition.

13. Threats to internal and external validity (2) Fidelity of randomization and lack of treatment contamination – Do people drop from experiment post-randomization? [internal validity]  no differences in results when we treat defecting volunteers as “live” versus when we drop them from the study – Do “live” students view internet version and do “internet” students attend live lecture? [internal validity]  can’t know for certain about live students viewing lectures online, but it looks like live students definitely attend more live lectures than those with the choice. Note that there is a “professional” private note- taking and tutoring service that is very popular and available to all students regardless of live/online. Other studies at the institution on cramming indicate that many internet course- takers don’t view all (or most, or sometimes any) of the lectures either

14. Threats to internal and external validity Intro economics might be special The university in question might be unusual The particular instructor may translate well/poorly to the internet platform  This is important, and there’s nothing we can do about this except to call for more experiments

15. Results Small insignificant positive estimated effects of live-only vs. internet-only instruction; statistically significant (mostly due to larger coefficients rather than smaller standard errors) when conditioning on covariates Positive estimated effects are largest for Hispanic students (sig) and Asian students (not sig); male students (not sig) and students with relatively low SAT scores (not sig)  larger sample sizes and cluster randomization might have helped detect differences here

16. Conclusions There might be efficiencies to exploit with internet-based traditional lectures, but there is no free lunch The results of this experiment should be interpreted as a first piece of evidence; responsible universities should be slow to implement this policy change despite the momentum and push for this. We need many more – including larger -- experiments!