1. Technology’s Edge: The Educational Benefits of Computer-Aided Instruction Lisa Barrow Federal Reserve Bank of Chicago Lisa Markman Princeton University Cecilia Rouse Princeton University and NBER

2. Student achievement is critical for individuals and society U.S. math skills have been increasing (NAEP), but proficiency levels remain low. Math skills may explain a large portion of wage inequality (Grogger, 1996; Murnane, Willet, & Levy, 1995)

3. School districts are turning to advances in computers to… –Reduce administrative burden; –Compensate for poor teacher content knowledge (especially in districts that report difficulty recruiting and retaining teachers, particularly in math and science); –Allow more individualized student attention; students can progress at own pace.

4. Evidence on effectiveness of CAI is surprisingly weak –Poorly defined computer use. –Little use of randomized controlled study design.

5. In a 2001 review of the literature, Larry Cuban (2001, p. 179) concludes, “When it comes to higher teacher and student productivity and a transformation of teaching and learning … there is little ambiguity. Both must be tagged as failures. Computers have been oversold and underused, at least for now.”

6. Computer-Aided Instruction (CAI) Treatment: Typically used in large urban districts; Software and hardware package designed to deliver one-on-one instruction; Software described as “meeting National Council of Teachers of Mathematics standards”; Software may be configured to state standards; Includes classroom management tools.

7. The Program 5 components per lesson: –Pretest –Review –Lesson –Cumulative review –Comprehensive test Required to achieve certain degree of mastery before advancing; Teachers can monitor student progress.

8. The design of this experiment… Randomly assign 8 th and 9 th grade classes in three large urban districts to be taught using computer-aided algebra and pre-algebra instruction. Assess the impact on statewide tests and tests designed to target algebra and pre-algebra skills. Note that the computer use was well- defined.

9. Research Design: Within-school random assignment to CAI or traditional instruction at the classroom level in three districts Research design addresses two forms of selection bias: Non-random assignment of students to CAI; Non-random assignment of teachers to CAI.

10. Table 1: Characteristics of the Districts in Our Study Top 100 Districts 3 Districts Combined District 1District 2District 3 # Students112,80762,507~68,000~22,000~97,000 % Female48.849.449.748.849.3 % Native American0.60.50.1 1.0 % Asian7.13.11.90.84.4 % African American28.169.593.640.359.4 % Hispanic34.116.21.154.318.0

11. Implementation of Random Assignment Schools provided us with class schedules of pre-algebra and algebra classes; Given option of eliminating particular teachers or classes from the experiment; Randomization information provided to schools after students’ classes were scheduled.

12. TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A Randomized Evaluation of a Computerized Math Curriculum Current School Schedule

13. TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A TeacherPeriod 1Period 2Period 3Period 4 ARegular LAB B Regular C D LAB Randomized Evaluation of a Computerized Math Curriculum Current School Schedule Below is a sample schedule that would be returned to each school after random assignment

14. Table 2a: Schools and Students in Study – District 1 Relevant Schools Schools in Study Students in Study Number of Students29,6038,148973 % Grade 819.316.840.4 % Grade 918.018.347.2 % Grade 1015.117.84.4 % Female50.549.052.0 % African American94.297.287.8 % Hispanic1.00.8 % White2.60.40.1 % Native American<0.1 0.0 % Asian2.21.61.8 % Missing demographic data 9.6

15. Table 2b: Schools and Students in Study – District 2 Relevant Schools Schools in Study Students in Study Number of Students5,2704,476412 % Grade 82.30.0 % Grade 938.040.052.7 % Grade 1022.023.231.8 % Female48.448.246.7 % African American43.642.047.1 % Hispanic50.151.244.7 % White5.55.96.6 % Native American0.20.10.2 % Asian0.70.80.5 % Missing demographic data 0.2

16. Table 2c: Schools and Students in Study – District 3 Relevant Schools Schools in Study Students in Study Number of Students27,5723,540200 % Grade 81.40.03.5 % Grade 935.640.091.5 % Grade 1023.325.13.0 % Female49.947.647.7 % African American61.192.594.5 % Hispanic15.21.20.5 % White18.34.01.5 % Native American1.10.40.0 % Asian4.51.93.0 % Missing demographic data 0.5

17. Numbers of Schools Classes, Teachers, and Randomization Pools Analysis Sample CombinedDistrict 1District 2District 3 Number of schools171043 Number of randomization pools 60311910 Number of classes141744423 Number of teachers5736147 Number of students1,585973412200

18. Outcomes Algebra and pre-algebra tests by Northwest Evaluation Association (NWEA) to be consistent with state and district standards; State-wide administered math tests; District benchmark tests in pre- algebra.

19. Table 3: Randomization of Treatment and Control Using Full Sample Random Assignment Traditional Instruction CAI p-value of diff. Full Sample Baseline algebra test score 24.7 0.494 Female47.247.10.637 African American80.083.20.561 Hispanic15.913.50.195 Class Size25.825.70.860 # of Observations1,1331,145

20. Table 3 (cont.): Randomization of Treatment and Control Using Analysis Sample Random Assignment Traditional Instruction CAI p-value of diff. Analysis Sample Baseline algebra test score 24.724.80.304 Female51.148.90.148 African American81.984.00.060 Hispanic13.812.10.061 Class Size25.826.20.549 # of Observations785800

21. Intent-to-Treat Estimates of the Effect of CAI on Algebra Achievement (with and without Teacher Fixed Effects)

22. Advantages/Disadvantages of the Intent-to-Treat Effect Represents the gains a policy maker might reasonably expect to observe. Does not necessarily represent the effectiveness of the program.

23. Amount of Time in the Computer Lab by the Random Assignment of the Student’s Class

24. Intent-to-Treat and Treatment on the Treated Estimates of the Effect of CAI (with and without teacher fixed effects)

25. Intent-to-Treat Estimates in District 1 Using Different Tests

26. Intent-to-Treat Estimates in Districts 2 and 3 Using Different Tests

27. We might expect to see an advantage of CAI in… Classes where curriculum best suited to students; Larger classes; Classes with more disruptive students; Classes with heterogeneous students.

28. Differential Intent to Treat Effects of the Computerized Instruction on Pre-Algebra and Algebra Achievement by Class Type

29. Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Baseline Test Score Quartile

30. Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Individual Attendance Rates

31. Differential Intent-to-Treat Effects by Class Characteristic: Attendance, Class Size, and Class Baseline Test Score S.D.

32. Overall we find: On average, students in classes taught using CAI scored higher on algebra tests than students in traditionally-taught classes. The effects appear larger for students in larger classes (especially large, heterogenous classes), those with worse attendance rates, and those in classes with lower average attendance rates.

33. Cost-Benefit Analysis Cost of CAI # Classes Total students Class size # periods CAI labs needed Cost/ student School A2273033.283.0$218 School B1232126.881.5$245 District 174173623.589.3$279 Cost of reducing class size to 13 # Classes Total students Class size # periods # new teachers Cost/ student School A2273033.265.7$329 School B1232126.862.1$278 District 174173623.569.9$241