1. The Pennsylvania State University (PSU) Rutgers, The State University of New Jersey, Center for Effective School Practices (CESP) Caliber, an ICF International Company Metiri Group ANALYTICA Regional Educational Laboratory Effects of Odyssey® Math Software on the Mathematics Achievement of Selected Fourth Grade Students in the Mid- Atlantic Region: A Multi-Site Cluster Randomized Trial Dr. Kay Wijekumar The Pennsylvania State University Collaborators: Dr. John Hitchcock (Co-PI) Drs. Turner, Lei, and Peck

2. Outline Study Background Intervention Description Study Design Implementation Results Study Limitations

3. Study Background Need to find innovative and effective approaches to improve math achievement 4 th Grade is a critical point in the elementary school curriculum at which the United States is losing ground to other countries Odyssey Math had some research support and the software was widely used in the Mid- Atlantic region

4. Odyssey Math Source: CompassLearning Odyssey Math .

5. Odyssey Math Source: CompassLearning Odyssey Math .

6. Odyssey Math Source: CompassLearning Odyssey Math .

7. Odyssey Math Source: CompassLearning Odyssey Math .

8. Professional Development Professional development “day” number a Type of settingMonth and durationNumber of attendeesContent b 1Large group instruction in computer labs at universities in Altoona and Scranton, Pennsylvania, and Rutgers, New Jersey August 2007  Day 1: 5 hours  Day 2: 3 hours  37 Intervention teachers and 4 administrators  2–4 members of the study team  Student launch pad  Overview of curriculum, tests, and assessments Makeup ”day”In-school “day”Compressed to 1 full day  23 intervention teachers  1 member of the study team  Student launch pad  Overview of curriculum, tests, and assessments 2In-school, one-on-one coaching October–November 2007 1–2 hours 60 intervention teachersStartup, management, logistics 3Large group instruction in computer labs at universities in Altoona, Beaver, and Scranton, Pennsylvania; and New Brunswick, New Jersey January 2008 6 hours  60 intervention teachers  2–3 members of the study team Incorporating Odyssey Math in lesson plans 4In-school, one-on-one coaching February 2008 1–2 hours 60 intervention teachersDeveloping assessments and reports 5In-school, one-on-one coaching March 2008 1–2 hours 60 intervention teachersScaffolding assignments and tailoring to individual students

9. Theory of Action Students -Assessment Tools -On-screen manipulatives -Tiered, guided feedback Teachers - 6 Day PD Training -Identify Learning Objects -Assess if objectives are met -Direct Students based on what they learn -Teacher knowledge & practice (e.g., use of software assessments to support guided feedback); not directly measured Teacher Characteristics Ability to teach mathematics but otherwise offer multiple subjects Schools and districts: Mid-Atlantic Region; not already using Odyssey® Math; has the needed technology platform Student Characteristics Make and female, low and med/high achievers, 4th grade Intervention Context -Increased student motivation -Use of graduated learning strategies; these are no t directly measured Intervention used with fidelity; assessed via observation checklists & software logs Mathematics Achievement Key Features of Intervention -Teacher PD to identify math learning objectives and student progress toward meeting them -Curricula uses software with on-screen manipulatives and guided feedback

10. Study Design – Research Questions Confirmatory 1.Do Odyssey® Math 4th grade classrooms outperform control classrooms on the math subtest of the TerraNova CTBS Basic Battery? Exploratory 1.What is the effect of Odyssey® Math on the math performance of males and female students? 2.What is the effect of Odyssey® Math on the math performance of low and medium/high achieving students?

11. Study Design Effectiveness Trial Approximately 128 classrooms in 32 schools Every school has intervention AND control classroom(s) Different characteristics (e.g., urban vs. rural, experience with computerized curricula, etc.) Pretest & Posttest of Terranova Math Subtest

12. Implementation - 1 Approximately 122 teachers 124 classrooms in 32 schools Pretest TerraNova Math Subtest administered to – 2640 students One teacher opted to do paper lessons One School had to be dropped because they were already using Odyssey Math (prior to pretesting)

13. Implementation - 2 ATS Lab Extension Specialists Field Research Coordinators Pre-tests Fidelity Observations Post-tests (with follow up)

14. Math Curricular Used in Participating Schools Regular Curriculum Number of classrooms Everyday Math (Everyday Math, 2009) 10 Scott Foresman (Pearson, 2009)7 Saxon Math (Saxon, 2009)5 Houghton Mifflin (Houghton Mifflin, 2009a) 1 Harcourt Brace (Harcourt School, 2009) 5 Investigations (Investigations, 2009) 1 McGraw Hill (MacMillan McGraw Hill, 2009) 1 Math Central (Houghton Mifflin, 2009b) 1 Mathematics-Path to Success (Gin, 2001) 1

15. Implementation – Developer Communications Planning Meetings Design and choice of professional development packages Planning and implementation of large- group professional development sessions Regular Communications Review of coaching session status Review of Odyssey Math usage statistics

16. Implementation – Data Management Web-based Secure Full spectrum of services ( data entry, reports, download/upload of files) Central repository Ease of management

17. Results - mean differences on math achievement at pre- and posttest Outcome measure Intervention group mean Control group mean Estimated difference a p-value 95 percent confidence interval Effect size b Pretest score621.46621.350.11(2.51).964 –4.81, 5.03na Posttest score unadjusted for class pretest mean647.41646.600.81(2.36).734 –3.82, 5.44.02 Posttest score adjusted for class pretest mean648.29647.500.78(1.27).543 –1.71, 3.27.02

18. Discussion - Limitations Effectiveness vs. efficacy trials This study used a volunteer sample Time on task varied This study only concentrated on the 4 th Grade and the Math portion of the suite This study used the software as a partial substitute to the regular curriculum

19. Questions? Please use the subject – Odyssey Math in your emails: Kay Wijekumar kxw190@psu.edu Collaborators: Dr. John Hitchcock (Co-PI) Drs. Turner, Lei, and Peck