Effectiveness of Reading and Math Software Products Findings From the First Student Cohort Mark Dynarski May 2007

2 Study Synopsis Design –Nine reading and six math software products in 132 volunteer schools –Treatment teachers could use products, control teachers could not Implementation –Companies provided training and other types of support –Study purchased various upgrades and hardware components Key Findings –Test scores at the end of the school year were not statistically different –Few relationships between effects and implementation factors Design –Nine reading and six math software products in 132 volunteer schools –Treatment teachers could use products, control teachers could not Implementation –Companies provided training and other types of support –Study purchased various upgrades and hardware components Key Findings –Test scores at the end of the school year were not statistically different –Few relationships between effects and implementation factors

3 Study Size

4 Implementation Framework Teacher training [O, R] Amount of use [I, R] Technical difficulties and teacher support [I] Teacher training [O, R] Amount of use [I, R] Technical difficulties and teacher support [I] Student and teacher roles [O] –Teaching approach [O] –On-task behavior [O] –Consistency with suggested use of products [I] Student and teacher roles [O] –Teaching approach [O] –On-task behavior [O] –Consistency with suggested use of products [I] Key: O indicates observations, R records, I interviews

5 General Implementation Findings Nearly all teachers received training and believed it prepared them to use products Difficulties using hardware mostly were minor Total use of software products higher in treatment classrooms When products are used –teachers more likely to be “facilitators” and students more likely to work on their own –more students on task in math classrooms Nearly all teachers received training and believed it prepared them to use products Difficulties using hardware mostly were minor Total use of software products higher in treatment classrooms When products are used –teachers more likely to be “facilitators” and students more likely to work on their own –more students on task in math classrooms

6 Difference in Technology Use in Treatment and Control Classrooms: First Grade

7 Effects on Classroom Practices Percent Difference: Teacher as Facilitator 0% 25% 50% 75% 100% First Grade Fourth Grade Sixth Grade Algebra Note: * Significantly different from zero at the 0.05 level * * **

8 Effects on Classroom Practices Percent Difference: Students On Task Note: * Significantly different from zero at the 0.05 level 0% 10% 20% 30% * X First Grade Fourth Grade Sixth Grade Algebra

9 Estimating Effects Outcome: spring test score 3-level model (students, classrooms, schools) 3-level model extended to estimate effects of conditions and practices (implementation) –interactions of treatment effect and classroom and school characteristics Outcome: spring test score 3-level model (students, classrooms, schools) 3-level model extended to estimate effects of conditions and practices (implementation) –interactions of treatment effect and classroom and school characteristics

10 Effect Sizes By School: First Grade

11 Test Scores: First Grade SAT - 9 Reading Score Overall Score Effect Size Note: * Significantly different from zero at the 0.05 level Sounds and Letters Sentence Reading Word Reading

12 Test Scores: First Grade Test of Word Reading Efficiency Score Overall Score Phonemic Decoding Efficiency Sight Word Efficiency Effect Size Note: None of the effect sizes is significantly different from zero at the 0.05 level

13 Interactions: First Grade Larger effects –More experienced teachers –Smaller student-teacher ratio Larger effects –More experienced teachers –Smaller student-teacher ratio No relationship –Product usage –Problems getting access –Technical difficulties –Computer specialist in school –Professional development last year on using technology –Poverty, urban area, African-American students, Hispanic, special education students

14 Test Scores: Fourth Grade SAT-10 Reading Score Overall Score VocabularyWord Study Skills Comprehension Note: None of the effect sizes is significantly different from zero at the 0.05 level X Effect Size

15 Interactions: Fourth Grade Larger effects –Product usage Larger effects –Product usage No relationship –Problems getting access –Technical difficulties –Computer specialist in school –Professional development last year on using technology –Poverty, urban area

16 Test Scores: Sixth Grade SAT-10 Math Score Overall Score Procedures Problem Solving Note: None of the effect sizes is significantly different from zero at the 0.05 level Effect Size

17 Interactions: Sixth Grade No statistically significant relationships

18 Test Scores: Algebra ETS Algebra Exam Overall ScoreConceptsProcessesSkills Note: None of the effect sizes is significantly different from zero at the 0.05 level Effect Size

19 Interactions: Algebra Smaller effects when teachers had technical difficulties

20 Study Tradeoffs 15 reading and math products studied –Many products and types of technology not in the study Precision to detect small effect sizes –Average effect reported Experimental design –Teachers have not used these products in current classrooms 15 reading and math products studied –Many products and types of technology not in the study Precision to detect small effect sizes –Average effect reported Experimental design –Teachers have not used these products in current classrooms

21 Second Year Study 10 products, data from 77 schools, treatment and control teachers with new cohort of students Will study relationship between teacher experience using products and effects Effects reported for products 10 products, data from 77 schools, treatment and control teachers with new cohort of students Will study relationship between teacher experience using products and effects Effects reported for products