Technology’s Edge: The Educational Benefits of Computer-Aided Instruction Lisa Barrow Federal Reserve Bank of Chicago Lisa Markman Princeton University.

Slides:



Advertisements
Similar presentations
Assessing Student Performance
Advertisements

Mastery Learning Compare and contrast direct instruction and mastery learning. Developed by W. Huitt (1998)
Pennsylvania’s Continuous Improvement Process. Understanding AYP How much do you know about AYP?
Teacher Effectiveness in Urban Schools Richard Buddin & Gema Zamarro IES Research Conference, June 2010.
VALUE – ADDED 101 Ken Bernacki and Denise Brewster.
MSP Program Evaluation
Watertown Public Schools Assessment Report 2009 Ann Koufman-Frederick & WPS Administrative Council School Committee Meeting December 7, 2009 Part I MCAS,
Central Middle School and Prime Time Closing the Achievement Gap Through Collaboration By Principal Pablo Sanchez and Jon Goldman, Prime Time Coordinator.
Measures of Student Achievement Appoquinimink School District November 2007.
1 Standards, Curriculum, and Research Mathematically Connected Communities (MC 2 ) Adapted from a PowerPoint by Barbara A. Austin, Ph.D.
Mark DeCandia Kentucky NAEP State Coordinator
performance INDICATORs performance APPRAISAL RUBRIC
Providing Leadership in Reading First Schools: Essential Elements Dr. Joseph K. Torgesen Florida Center for Reading Research Miami Reading First Principals,
Evaluation of Math-Science Partnership Projects (or how to find out if you’re really getting your money’s worth)
Common Questions What tests are students asked to take? What are students learning? How’s my school doing? Who makes decisions about Wyoming Education?
Challenges in Developing a University Admissions Test & a National Assessment A Presentation at the Conference On University & Test Development in Central.
Conducting Randomized Evaluations in Schools Lisa Markman Associate Director Education Research Section Princeton University.
Brown, Q., Mongan, W., Garbarine, E., Kusic, D., Fromm, E., & Fontecchio, A. (2008). Computer Aided Instruction as a Vehicle for Problem Solving: Scratch.
Vouchers in Milwaukee: What Have We Learned From the Nation’s Oldest and Largest Program? Deven Carlson University of Oklahoma.
Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research Fidelity of Implementation and Program Impact.
(High/Middle School) HSTW/MMGW Site Presentation ( Month Date, 2006) Promising Practices Next Steps Major Challenges Technical Review Visit (TRV)
Meryle Weinstein, Emilyn Ruble Whitesell and Amy Ellen Schwartz New York University Improving Education through Accountability and Evaluation: Lessons.
Evaluating the Impact of the Advancement Via Individual Determination Program on Ninth-Grade Students' Learning and Study Skills Jenny Nagaoka, Melissa.
Metropolitan School District of Metropolitan School District of Pike Township Indianapolis, Indiana Pike Township Indianapolis, Indiana A K-12 Coaching.
Seaford School District Annual Parent Meeting 1. Title I Funding and Programs Parent Meeting Agenda Title I Program Presentation Document Review Parent.
Project CLASS “Children Learning Academic Success Skills” This work was supported by IES Grant# R305H to David Rabiner Computerized Attention Training.
1 Using Data to Improve Student Achievement & to Close the Achievement Gap Tips & Tools for Data Analysis Spring 2007.
1 Standards, Curriculum, and Research Mathematically Connected Communities (MC 2 ) Adapted from a PowerPoint by Barbara A. Austin, Ph.D.
End of Year Report_ DataSet 1 Lodi Unified School District Year-End Benchmark Assessment Results (Student Achievement Monitoring)
Linking a Comprehensive Professional Development Literacy Program to Student Achievement Edmonds School District WERA December 4, 2008.
How Much Value is Added? An Evaluation Plan for the Achievement Challenge Pilot Project.
Integrating Success The Transition of All Students From High School to College November 2007 Iowa Educational Research & Evaluation Association Annual.
Implications of Mediated Instruction to Remote Learning in Mathematics Joy L. Matthews-López Educational Testing Service Sergio R. López-Permouth David.
Louisiana Math and Science Partnership State Coordinator Meeting Washington, DC June 2008.
Mark DeCandia Kentucky NAEP State Coordinator
Factors Influencing Preparation in Mathematics for Selective Admission to College in High Schools with Low College-going Rates Faith G. Paul Student Affairs.
Education, Evidence, and Policy Cecilia Elena Rouse, Ph.D. Member of the Council of Economic Advisers June 8, 2009.
The Evaluation of Mathematics and Science Partnerships Program A Quasi Experimental Design Study Abdallah Bendada, Title II Director
The PLC Team Learning Process Review Step One: Identify essential (key) learning standards that all students must learn in each content area during each.
Changes in Professional licensure Teacher evaluation system Training at Coastal Carolina University.
Evaluating Impacts of MSP Grants Ellen Bobronnikov January 6, 2009 Common Issues and Potential Solutions.
Comments on: The Evaluation of an Early Intervention Policy in Poor Schools Germano Mwabu June 9-10, 2008 Quebec City, Canada.
Effectiveness of Reading and Math Software Products Findings From the First Student Cohort Mark Dynarski May 2007.
School Improvement Plan Central Elementary Vanessa S. McAllister, Principal Margaret Lewis, Assistant Principal RaeAnn Whiteside, Literacy Coach.
The ACE Strategy Week 2: Administer and Evaluate Pre-test Cluster Cycle 2 Goal: By the end of the cycle, 78% of students in grades 3-5 will increase their.
Massachusetts Comprehensive Assessment System (MCAS) /22/2010.
Mobile County School District SPED 6-9 End of Year Consultative Session
Effectiveness of Selected Supplemental Reading Comprehension Interventions: Impacts on a First Cohort of Fifth-Grade Students June 8, 2009 IES Annual Research.
GEORGIA’S CRITERION-REFERENCED COMPETENCY TESTS (CRCT) Questions and Answers for Parents of Georgia Students February 11, 2009 Presented by: MCES.
EDP 520 – R ESEARCH IN E DUCATION By Obed Nartey.
Santa Fe Public Schools Our Common Commitments Improving our lives through education.
Classroom Network Technology as a Support for Systemic Mathematics Reform: Examining the Effects of Texas Instruments’ MathForward Program on Student Achievement.
RSU #38 Response to Intervention
Teacher SLTs
Mobile County School District At-Risk/SPED 2-5
Gloria Ladson-Billings University of Wisconsin
Teacher SLTs
Standards Based Grading
January 17, 2017 Board Workshop
Classroom Assessment Validity And Bias in Assessment.
Using Formative Assessment to Improve Student Achievement
Teacher Evaluation “SLO 101”
Average is Officially Over!
Otis J. Brock, III Elementary School
Teacher SLTs
Brahm Fleisch Research supported by the Zenex Foundation October 2017
Jayhawkville Central High School
Otis J. Brock, III Elementary School
Split-Block Class Schedule at Yorktown High School
Teacher SLTs
Presentation transcript:

Technology’s Edge: The Educational Benefits of Computer-Aided Instruction Lisa Barrow Federal Reserve Bank of Chicago Lisa Markman Princeton University Cecilia Rouse Princeton University and NBER

Student achievement is critical for individuals and society U.S. math skills have been increasing (NAEP), but proficiency levels remain low. Math skills may explain a large portion of wage inequality (Grogger, 1996; Murnane, Willet, & Levy, 1995)

School districts are turning to advances in computers to… –Reduce administrative burden; –Compensate for poor teacher content knowledge (especially in districts that report difficulty recruiting and retaining teachers, particularly in math and science); –Allow more individualized student attention; students can progress at own pace.

Evidence on effectiveness of CAI is surprisingly weak –Poorly defined computer use. –Little use of randomized controlled study design.

In a 2001 review of the literature, Larry Cuban (2001, p. 179) concludes, “When it comes to higher teacher and student productivity and a transformation of teaching and learning … there is little ambiguity. Both must be tagged as failures. Computers have been oversold and underused, at least for now.”

Computer-Aided Instruction (CAI) Treatment: Typically used in large urban districts; Software and hardware package designed to deliver one-on-one instruction; Software described as “meeting National Council of Teachers of Mathematics standards”; Software may be configured to state standards; Includes classroom management tools.

The Program 5 components per lesson: –Pretest –Review –Lesson –Cumulative review –Comprehensive test Required to achieve certain degree of mastery before advancing; Teachers can monitor student progress.

The design of this experiment… Randomly assign 8 th and 9 th grade classes in three large urban districts to be taught using computer-aided algebra and pre-algebra instruction. Assess the impact on statewide tests and tests designed to target algebra and pre-algebra skills. Note that the computer use was well- defined.

Research Design: Within-school random assignment to CAI or traditional instruction at the classroom level in three districts Research design addresses two forms of selection bias: Non-random assignment of students to CAI; Non-random assignment of teachers to CAI.

Table 1: Characteristics of the Districts in Our Study Top 100 Districts 3 Districts Combined District 1District 2District 3 # Students112,80762,507~68,000~22,000~97,000 % Female % Native American % Asian % African American % Hispanic

Implementation of Random Assignment Schools provided us with class schedules of pre-algebra and algebra classes; Given option of eliminating particular teachers or classes from the experiment; Randomization information provided to schools after students’ classes were scheduled.

TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A Randomized Evaluation of a Computerized Math Curriculum Current School Schedule

TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A TeacherPeriod 1Period 2Period 3Period 4 ARegular LAB B Regular C D LAB Randomized Evaluation of a Computerized Math Curriculum Current School Schedule Below is a sample schedule that would be returned to each school after random assignment

Table 2a: Schools and Students in Study – District 1 Relevant Schools Schools in Study Students in Study Number of Students29,6038, % Grade % Grade % Grade % Female % African American % Hispanic % White % Native American< % Asian % Missing demographic data 9.6

Table 2b: Schools and Students in Study – District 2 Relevant Schools Schools in Study Students in Study Number of Students5,2704, % Grade % Grade % Grade % Female % African American % Hispanic % White % Native American % Asian % Missing demographic data 0.2

Table 2c: Schools and Students in Study – District 3 Relevant Schools Schools in Study Students in Study Number of Students27,5723, % Grade % Grade % Grade % Female % African American % Hispanic % White % Native American % Asian % Missing demographic data 0.5

Numbers of Schools Classes, Teachers, and Randomization Pools Analysis Sample CombinedDistrict 1District 2District 3 Number of schools Number of randomization pools Number of classes Number of teachers Number of students1,

Outcomes Algebra and pre-algebra tests by Northwest Evaluation Association (NWEA) to be consistent with state and district standards; State-wide administered math tests; District benchmark tests in pre- algebra.

Table 3: Randomization of Treatment and Control Using Full Sample Random Assignment Traditional Instruction CAI p-value of diff. Full Sample Baseline algebra test score Female African American Hispanic Class Size # of Observations1,1331,145

Table 3 (cont.): Randomization of Treatment and Control Using Analysis Sample Random Assignment Traditional Instruction CAI p-value of diff. Analysis Sample Baseline algebra test score Female African American Hispanic Class Size # of Observations785800

Intent-to-Treat Estimates of the Effect of CAI on Algebra Achievement (with and without Teacher Fixed Effects)

Advantages/Disadvantages of the Intent-to-Treat Effect Represents the gains a policy maker might reasonably expect to observe. Does not necessarily represent the effectiveness of the program.

Amount of Time in the Computer Lab by the Random Assignment of the Student’s Class

Intent-to-Treat and Treatment on the Treated Estimates of the Effect of CAI (with and without teacher fixed effects)

Intent-to-Treat Estimates in District 1 Using Different Tests

Intent-to-Treat Estimates in Districts 2 and 3 Using Different Tests

We might expect to see an advantage of CAI in… Classes where curriculum best suited to students; Larger classes; Classes with more disruptive students; Classes with heterogeneous students.

Differential Intent to Treat Effects of the Computerized Instruction on Pre-Algebra and Algebra Achievement by Class Type

Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Baseline Test Score Quartile

Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Individual Attendance Rates

Differential Intent-to-Treat Effects by Class Characteristic: Attendance, Class Size, and Class Baseline Test Score S.D.

Overall we find: On average, students in classes taught using CAI scored higher on algebra tests than students in traditionally-taught classes. The effects appear larger for students in larger classes (especially large, heterogenous classes), those with worse attendance rates, and those in classes with lower average attendance rates.

Cost-Benefit Analysis Cost of CAI # Classes Total students Class size # periods CAI labs needed Cost/ student School A $218 School B $245 District $279 Cost of reducing class size to 13 # Classes Total students Class size # periods # new teachers Cost/ student School A $329 School B $278 District $241