Technology’s Edge: The Educational Benefits of Computer-Aided Instruction Lisa Barrow Federal Reserve Bank of Chicago Lisa Markman Princeton University Cecilia Rouse Princeton University and NBER

Student achievement is critical for individuals and society U.S. math skills have been increasing (NAEP), but proficiency levels remain low. Math skills may explain a large portion of wage inequality (Grogger, 1996; Murnane, Willet, & Levy, 1995)

School districts are turning to advances in computers to… –Reduce administrative burden; –Compensate for poor teacher content knowledge (especially in districts that report difficulty recruiting and retaining teachers, particularly in math and science); –Allow more individualized student attention; students can progress at own pace.

Evidence on effectiveness of CAI is surprisingly weak –Poorly defined computer use. –Little use of randomized controlled study design.

In a 2001 review of the literature, Larry Cuban (2001, p. 179) concludes, “When it comes to higher teacher and student productivity and a transformation of teaching and learning … there is little ambiguity. Both must be tagged as failures. Computers have been oversold and underused, at least for now.”

Computer-Aided Instruction (CAI) Treatment: Typically used in large urban districts; Software and hardware package designed to deliver one-on-one instruction; Software described as “meeting National Council of Teachers of Mathematics standards”; Software may be configured to state standards; Includes classroom management tools.

The Program 5 components per lesson: –Pretest –Review –Lesson –Cumulative review –Comprehensive test Required to achieve certain degree of mastery before advancing; Teachers can monitor student progress.

The design of this experiment… Randomly assign 8 th and 9 th grade classes in three large urban districts to be taught using computer-aided algebra and pre-algebra instruction. Assess the impact on statewide tests and tests designed to target algebra and pre-algebra skills. Note that the computer use was well- defined.

Research Design: Within-school random assignment to CAI or traditional instruction at the classroom level in three districts Research design addresses two forms of selection bias: Non-random assignment of students to CAI; Non-random assignment of teachers to CAI.

Table 1: Characteristics of the Districts in Our Study Top 100 Districts 3 Districts Combined District 1District 2District 3 # Students112,80762,507~68,000~22,000~97,000 % Female % Native American % Asian % African American % Hispanic

Implementation of Random Assignment Schools provided us with class schedules of pre-algebra and algebra classes; Given option of eliminating particular teachers or classes from the experiment; Randomization information provided to schools after students’ classes were scheduled.

TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A Randomized Evaluation of a Computerized Math Curriculum Current School Schedule

TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A TeacherPeriod 1Period 2Period 3Period 4 ARegular LAB B Regular C D LAB Randomized Evaluation of a Computerized Math Curriculum Current School Schedule Below is a sample schedule that would be returned to each school after random assignment

Table 2a: Schools and Students in Study – District 1 Relevant Schools Schools in Study Students in Study Number of Students29,6038, % Grade % Grade % Grade % Female % African American % Hispanic % White % Native American< % Asian % Missing demographic data 9.6

Table 2b: Schools and Students in Study – District 2 Relevant Schools Schools in Study Students in Study Number of Students5,2704, % Grade % Grade % Grade % Female % African American % Hispanic % White % Native American % Asian % Missing demographic data 0.2

Table 2c: Schools and Students in Study – District 3 Relevant Schools Schools in Study Students in Study Number of Students27,5723, % Grade % Grade % Grade % Female % African American % Hispanic % White % Native American % Asian % Missing demographic data 0.5

Numbers of Schools Classes, Teachers, and Randomization Pools Analysis Sample CombinedDistrict 1District 2District 3 Number of schools Number of randomization pools Number of classes Number of teachers Number of students1,

Outcomes Algebra and pre-algebra tests by Northwest Evaluation Association (NWEA) to be consistent with state and district standards; State-wide administered math tests; District benchmark tests in pre- algebra.

Table 3: Randomization of Treatment and Control Using Full Sample Random Assignment Traditional Instruction CAI p-value of diff. Full Sample Baseline algebra test score Female African American Hispanic Class Size # of Observations1,1331,145

Table 3 (cont.): Randomization of Treatment and Control Using Analysis Sample Random Assignment Traditional Instruction CAI p-value of diff. Analysis Sample Baseline algebra test score Female African American Hispanic Class Size # of Observations785800

Intent-to-Treat Estimates of the Effect of CAI on Algebra Achievement (with and without Teacher Fixed Effects)

Advantages/Disadvantages of the Intent-to-Treat Effect Represents the gains a policy maker might reasonably expect to observe. Does not necessarily represent the effectiveness of the program.

Amount of Time in the Computer Lab by the Random Assignment of the Student’s Class

Intent-to-Treat and Treatment on the Treated Estimates of the Effect of CAI (with and without teacher fixed effects)

Intent-to-Treat Estimates in District 1 Using Different Tests

Intent-to-Treat Estimates in Districts 2 and 3 Using Different Tests

We might expect to see an advantage of CAI in… Classes where curriculum best suited to students; Larger classes; Classes with more disruptive students; Classes with heterogeneous students.

Differential Intent to Treat Effects of the Computerized Instruction on Pre-Algebra and Algebra Achievement by Class Type

Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Baseline Test Score Quartile

Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Individual Attendance Rates

Differential Intent-to-Treat Effects by Class Characteristic: Attendance, Class Size, and Class Baseline Test Score S.D.

Overall we find: On average, students in classes taught using CAI scored higher on algebra tests than students in traditionally-taught classes. The effects appear larger for students in larger classes (especially large, heterogenous classes), those with worse attendance rates, and those in classes with lower average attendance rates.

Cost-Benefit Analysis Cost of CAI # Classes Total students Class size # periods CAI labs needed Cost/ student School A $218 School B $245 District $279 Cost of reducing class size to 13 # Classes Total students Class size # periods # new teachers Cost/ student School A $329 School B $278 District $241