NAEP and TIMSS Mathematics 2015 Samantha burg, ph.d. STEPHEN PROVASNIK, Ph.D. National Center for Education Statistics Washington, DC Conference board of the mathematical sciences May 5, 2017

National Assessment of Educational Progress (NAEP) Congressionally mandated project administered by the National Center for Education Statistics (NCES) Common measure of student achievement across the country in a variety of subject areas since 1969 Gold standard of U.S. large-scale assessments

Trends in International Mathematics and Science Study (TIMSS) Sponsored by the International Association for the Evaluation of Educational Achievement and managed in the U.S. by NCES Common measure of student achievement and school practices across the world since 1995 TIMSS assesses mathematics and science achievement at 4th- and 8th-grade TIMSS Advanced assesses advanced mathematics and physics achievement in the final year of secondary school

NAEP and TIMSS Similarities At 4th, 8th, and 12th grades (including TIMSS Advanced) Curriculum-based assessments Very similar frameworks Virtually identical IRT scoring methodology Include student, teacher, and principal background questionnaires Differences Mathematics frameworks have a few differences Different weight given to the same topic areas NAEP questions have greater complexity than TIMSS Achievement levels determined differently Math frameworks: TIMSS has a wider range and NAEP has harder items, on average Achievement levels: NAEP’s are aspirational and TIMSS are fixed

NAEP 2015 Mathematics Highlights

Mathematics assessment administered January-March 2015 NAEP 2015 Mathematics assessment administered January-March 2015 National samples: 279,000 fourth-graders, 273,000 eighth-graders, and 31,900 twelfth-graders Results available for: Nation (grades 4, 8, and 12) 50 states, the District of Columbia, and the Department of Defense school system (grades 4 and 8 only) 21 participating districts (grades 4 and 8 only) Performance reported as: Average scale scores 0-500 for grades 4 and 8 mathematics 0-300 for grade 12 mathematics Achievement levels (Basic, Proficient, Advanced)

Fourth-grade average score lower than 2013, higher than 1990 NAEP 2015 MATH Fourth-grade average score lower than 2013, higher than 1990 GRADE 4

Eighth-grade average score lower than 2013, higher than 1990 NAEP 2015 MATH Eighth-grade average score lower than 2013, higher than 1990 GRADE 8

Twelfth-grade average score lower than 2013 NAEP 2015 MATH Twelfth-grade average score lower than 2013 GRADE 12

NAEP 2015 MATH Percent at or above Proficient lower than 2013, higher than 1990: Grades 4 and 8 GRADE 4 GRADE 8

NAEP 2015 MATH Percent at or above Proficient lower than 2013, higher than 1990: Grade 12 GRADE 12

NAEP 2015 MATH Grade 4 scores increased in 3 states/jurisdictions and decreased in 16 compared to 2013 NOTE: DD = Department of Defense Education Activity (DoDEA).

Grade 8 scores decreased in 22 states compared to 2013 NAEP 2015 MATH Grade 8 scores decreased in 22 states compared to 2013 NOTE: DD = Department of Defense Education Activity (DoDEA).

TIMSS and TIMSS Advanced 2015 Highlights

TIMSS and TIMSS Advanced 1995 1999* 2003 2007 2011 2015 TIMSS TIMSS Advanced TIMSS TIMSS TIMSS TIMSS TIMSS TIMSS Advanced TIMSS : Mathematics and science assessment at grades 4 and 8 Both developed by the TIMSS & PIRLS International Study Center (ISC) at Boston College, under contract to the International Association for the Evaluation of Educational Achievement (IEA). TIMSS Advanced also in 2008, but the United States did not participate. TIMSS Advanced : Advanced mathematics and physics assessment in the final year of secondary school * No grade 4 assessment.

TIMSS 2015 Participation at 4th grade Participation at 8th grade 55 education systems More than 310,000 students Participation at 8th grade 44 education systems More than 260,000 students Results for Armenia, which did participate in TIMSS 2015 at grades 4 and 8, are not shown because their data are not comparable for trend analysis. NOTE: Education systems that administered TIMSS at a grade other than the target grade are not included in the count of education systems.

TIMSS Advanced 2015 Participation in the final year of secondary school 9 education systems France, Italy, Lebanon, Norway, Portugal, the Russian Federation, Slovenia, Sweden, and the United States More than 56,000 students In U.S., 12th-grade students who took or were taking advanced courses in mathematics and physics (e.g., AP courses, IB courses) More than 56,000 students (32,000 in advanced mathematics and 24,000 in physics) participated. Nearly 5,000 teachers and 3,000 schools completed questionnaires.

GRADE 4 MATH Fourth-grade average score not measurably different from 2011, higher than 1995 1000 * * 90th percentile * * * * 75th percentile * * * Average score * * * 25th percentile * * 10th percentile * Score is different from 2015

GRADE 4 MATH U.S. mathematics performance in the international context at the 4th grade in 2015 Singapore 618 Hong Kong-CHN 615 Korea, Rep. of 608 Chinese Taipei-CHN 597 Japan 593 Northern Ireland-GBR 570 Russian Federation 564 Norway 549 Ireland 547 England-GBR 546 Belgium (Flemish)-BEL Florida-USA Kazakhstan 544 Portugal 541 United States 539 Denmark Quebec-CAN 536 Lithuania 535 Finland Poland TIMSS Scale Centerpoint 500 Among 54 education systems, average scores in: 10 education systems were higher than the U.S. average 9 education systems were not measurably different from the U.S. average 34 education systems were lower than the U.S. average (not shown) Average higher than U.S. average

GRADE 4 MATH Education systems with score gains between 1995 and 2015: Mathematics 4th grade 1995 2015 Intermediate (475) Low (400) Advanced (625) High (550) 1000 350 400 450 500 550 600 650 Average Score

GRADE 4 MATH Percentage of grade 4 students reaching the international Advanced benchmark in mathematics in 2015 Avg. score Advanced (625) Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning. 618 615 608 597 593 570 564 546 544 539 547 549 529 541 Percent or score higher than U.S. percent or score Percent or score not measurably different from U.S. percent or score Score lower than U.S. score

GRADE 8 MATH Eighth-grade average score higher than 2011 on average, higher than 1995 at all percentiles 1000 * * * 90th percentile * * * * * * 75th percentile * * * * * Average score * * * 25th percentile * * 10th percentile * Score is different from 2015

GRADE 8 MATH U.S. mathematics performance in the international context at the 8th grade in 2015 Singapore 621 Korea, Rep. of 606 Chinese Taipei-CHN 599 Hong Kong-CHN 594 Japan 586 Quebec-CAN 543 Russian Federation 538 Kazakhstan 528 Canada 527 Ireland 523 Ontario-CAN 522 United States 518 England-GBR Slovenia 516 Hungary 514 Dubai-UAE 512 Norway Lithuania 511 Israel TIMSS Scale Centerpoint 500 Among 43 education systems, average scores in: 8 education systems were higher than the U.S. average 10 education systems were not measurably different from the U.S. average 24 education systems were lower than the U.S. average (not shown) Average higher than U.S. average

GRADE 8 MATH Education systems with score gains between 1995 and 2015: Mathematics 8th grade 1995 2015 Intermediate (475) Low (400) Advanced (625) High (550) 1000 350 400 450 500 550 600 650 Average score

GRADE 8 MATH Percentage of grade 8 students reaching the international Advanced benchmark in mathematics in 2015 Avg. score Advanced (625) Students can apply and reason in a variety of problem situations, solve linear equations, and make generalizations. 621 599 606 594 586 528 538 511 514 512 518 543 493 Percent or score higher than U.S. percent or score Percent or score not measurably different from U.S. percent or score Score lower than U.S. score

ADVANCED MATH U.S. advanced mathematics performance in the final year of secondary school in 2015 Among 10 education systems, average scores in: 2 education systems were higher than the U.S. average 2 education systems were not measurably different from the U.S. average 5 education systems were lower than the U.S. average Russian Federation (intensive courses*) 540 Lebanon 532 TIMSS Advanced Scale Centerpoint 500 United States 485 Russian Federation Portugal 482 France 463 Slovenia 460 Norway 459 Sweden 431 Italy 422 * Intensive courses are advanced mathematics courses that involve 6 or more hours per week. Results for students in these courses are reported separately from the results for other students from the Russian Federation taking courses that involve 4.5 hours per week. Average higher than U.S. average Average lower than U.S. average

Advanced Mathematics Coverage Index U.S. coverage index and advanced mathematics performance in the final year of secondary school in 2015 Russian Federation (intensive courses*) 540 Lebanon 532 TIMSS Advanced Scale Centerpoint 500 United States 485 Russian Federation Portugal 482 France 463 Slovenia 460 Norway 459 Sweden 431 Italy 422 Advanced Mathematics Coverage Index * Intensive courses are advanced mathematics courses that involve 6 or more hours per week. Results for students in these courses are reported separately from the results for other students from the Russian Federation taking courses that involve 4.5 hours per week. Average higher than U.S. average Average lower than U.S. average

ADVANCED MATH Average advanced mathematics scores and coverage index of TIMSS Advanced students, by education system: 2015 Average Score Coverage index 1000 * * * * * % * * % % % % % % % % % *Score is different from U.S. score NOTE: The advanced mathematics coverage index is the percentage of the corresponding age cohort covered by the students taking the TIMSS Advanced assessment in advanced mathematics. The corresponding age cohort is determined for education systems individually. In the United States, the corresponding age cohort is 18-year-olds.

TIMSS Advanced mathematics score changes: 1995-2015 Russian Federation (intensive courses)1 1995 2015 (2, 2) Intermediate (475) - UNITED STATES1 (6, 11) Advanced (625) - Country & Coverage index (1995, 2015) (20, 21) - Slovenia1 (75, 34) High (550) - (16, 14) - (14, 25) 1000 350 400 450 500 550 600 650 Average score 1 Change in average score was not measurably different

ADVANCED MATH Percentage of TIMSS Advanced students reaching the international Advanced benchmark in advanced mathematics in 2015 Avg. score Advanced (625) Students demonstrate thorough understanding of concepts, mastery of procedures, and mathematical reasoning skills. They can solve problems in complex contexts in algebra, calculus, geometry, and trigonometry. 540 485 532 460 422 482 431 463 459 Percent or score higher than U.S. percent or score Percent or score not measurably different from U.S. percent or score Percent or score lower than U.S. percent or score

Comparison of Mathematics Results NAEP 1990-2015 TIMSS 1995-2015 (Earliest to Current) NAEP 2007-2015 TIMSS 2007-2015 NAEP 2009-2015 NAEP 2013-2015 TIMSS 2011-2015 (Previous to Current) Grade 4 NAEP TIMSS Grade 8 2015 score is higher 2015 score is not measurably different 2015 score is lower

For more information NAEP http://nces.ed.gov/nationsreportcard/ For more NAEP results: http://nationsreportcard.gov Contact: Samantha Burg, NCES Samantha.burg@ed.gov 202-245-7537 TIMSS http://nces.ed.gov/timss/ For more TIMSS and TIMSS Advanced results: http://nces.ed.gov/timss/timss2015/ Contact: Stephen Provasnik, NCES Stephen.Provasnik@ed.gov 202-245-6442

THANK YOU

Scores decline for some student groups Mathematics Scores decline for some student groups 1 Percentages shown are for grade 4 students in 2015. # Rounds to zero. NOTE: The NAEP mathematics scale ranges from 0-500. Not available

Scores decline for some student groups Mathematics Scores decline for some student groups 1 Percentages shown are for grade 4 students in 2015. NOTE: The NAEP mathematics scale ranges from 0-500. Not available

Change in average mathematics scores by subscale Grade 4 Grade 8 Subscale 2015 '15-'13 Composite 240 -1 * 282 -2* Number properties and operations 243 +1 279 Measurement 238 -1* Geometry 236 -5* 280 Data analysis, statistics, and probability -4* Algebra 288 Significant decrease across subscales in 8th grade RED highlights areas where NVS study found a misalignment between NAEP framework and common core standards 4th grade-Geometry; 8th grade Data analysis and statistics) Numerically, those subscales showed the most significant decrease In both cases, subscales were taught later in common core compared to when NAEP measures it. Item analysis comparison between NAEP and common core will occur when NAEP transition to computer is complete (2017) * Statistically significant (p < .05).

Change in average scores from 2013 to 2015 Mathematics Change in average scores from 2013 to 2015 Score change Grade 4 Grade 8 Both grades Increase District of Columbia Mississippi Department of Defense schools Decrease Arkansas Connecticut Georgia Hawaii Minnesota Montana New Hampshire New York Florida Idaho Kentucky Louisiana Massachusetts Nevada North Carolina Ohio Pennsylvania South Carolina South Dakota Texas Washington West Virginia Colorado Delaware Kansas Maine Maryland North Dakota Rhode Island Vermont

Grade 4 Mathematics 17 state/jurisdictions score below and 21 score above the nation (public) NOTE: DD = Department of Defense Education Activity (DoDEA).

Grade 8 Mathematics 15 state/jurisdictions score below and 23 score above the nation (public) NOTE: DD = Department of Defense Education Activity (DoDEA).

Scores for racial/ethnic groups unchanged compared to 2013 Mathematics Scores for racial/ethnic groups unchanged compared to 2013 † Not applicable. NOTE: The NAEP mathematics scale ranges from 0-300 at grade 12.

Scores for most racial/ethnic groups higher compared to 2005 Mathematics Scores for most racial/ethnic groups higher compared to 2005 † Not applicable. * Significantly different (p < .05) from 2015. NOTE: The NAEP mathematics scale ranges from 0-300 at grade 12.

Score increase for one group since 2013 Mathematics Score increase for one group since 2013 NOTE: The NAEP mathematics scale ranges from 0-300 at grade 12.

Scores decline in all mathematics content areas since 2013 NOTE: The NAEP mathematics scale ranges from 0-300 at grade 12.

U.S. mathematics scores, by gender: 1995-2015 Grade 4 Grade 8 TIMSS Advanced 1000 1000 1000 * * 5 7 3 2 21 30 * * * * * * * 4 8 6 9 7 * Score higher than other gender : Gender score gap (M-F)

Summary of U.S. results from TIMSS and TIMSS Advanced 2015 2015 Science National Headlines Summary of U.S. results from TIMSS and TIMSS Advanced 2015 Trends over two decades (1995-2015) Score gains: math at grades 4 and 8; science at grade 8 No change: advanced math at grade 12; science at grade 4 and physics at grade 12 Trends compared to previous cycle (2011-2015) Score gains: math at grade 8 No change: math at grade 4; science at grades 4 and 8 Performance compared to other education systems in 2015 6 systems outperformed the U.S. students in both subjects at grades 4 and 8: Chinese Taipei, Hong Kong, Japan, Korea, Russian Federation, and Singapore. Gender score gaps in 2015 Gender score gaps favoring males in both subjects at grades 4 and 12; science at grade 8 No gender difference in mathematics at grade 8

Example of grade 4 item at advanced benchmark GRADE 4 MATH Example of grade 4 item at advanced benchmark Content Domain: Number Cognitive Domain: Reasoning Proficiency Level: Advanced Percentage of students answering correctly: U.S. average: 46% International average: 24%

Example of grade 8 item at advanced benchmark GRADE 8 MATH Example of grade 8 item at advanced benchmark Content Domain: Geometry Cognitive Domain: Reasoning Proficiency Level: Advanced Percentage of students answering correctly: U.S. average: 33% International average: 32%

Example of Advanced Mathematics item at Advanced benchmark Content Domain: Calculus Cognitive Domain: Applying Proficiency Level: Advanced Percentage of students answering correctly: U.S. average: 37% International average: 16%

What is on the TIMSS mathematics Grade 4 assessment? GRADE 4 MATH What is on the TIMSS mathematics Grade 4 assessment? Content Domain: Data Display Cognitive Domain: Knowing Proficiency Level: Low Percentage of students answering correctly: U.S. average: 90% International average: 84%

What is on the TIMSS mathematics Grade 8 assessment? GRADE 8 MATH What is on the TIMSS mathematics Grade 8 assessment? Content Domain: Geometry Cognitive Domain: Applying Proficiency Level: High Percentage of students answering correctly: U.S. average: 42% International average: 43%

What is on the TIMSS Advanced mathematics assessment? Content Domain: Algebra Cognitive Domain: Knowing Proficiency Level: Intermediate Percentage of students answering correctly: U.S. average: 61% International average: 50%

GRADE 4 MATH Change in U.S. grade 4 mathematics content domain average scores: 2007-2015 1000 2007 2011 2015 Overall mathematics 529* 541 539 Number 543 546 Geometry 522 535* 525 Data 545 540 * Score is different from 2015

GRADE 8 MATH Change in U.S. grade 8 mathematics content domain average scores: 2007-2015 1000 2007 2011 2015 Overall mathematics 508* 509* 518 Number 514 520 Algebra 507* 512* 525 Geometry 480* 485* 500 Data 533* 527 522 * Score is different from 2015

TIMSS 2015 international mathematics benchmarks GRADE 4 MATH TIMSS 2015 international mathematics benchmarks Grade 4 Advanced (625) Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning. High (550) Students can apply their knowledge and understanding to solve problems. Intermediate (475) Students can apply basic mathematical knowledge in simple situations. Low (400) Students have some basic mathematical knowledge.

TIMSS 2015 international mathematics benchmarks GRADE 8 MATH TIMSS 2015 international mathematics benchmarks Grade 8 Advanced (625) Students apply and reason in a variety of problem situations, solve linear equations, and make generalizations. High (550) Students can apply their understanding in a variety of relatively complex situations. Intermediate (475) Students can apply basic mathematical knowledge in a variety of situations. Low (400) Students have some knowledge of whole numbers and basic graphs.

TIMSS Advanced 2015 international mathematics benchmarks ADVANCED MATH TIMSS Advanced 2015 international mathematics benchmarks Final Year of Secondary School Advanced (625) Students demonstrate thorough understanding of concepts, mastery of procedures, and mathematical reasoning skills. They can solve problems in complex contexts in algebra, calculus, geometry, and trigonometry. High (550) Students can apply a broad range of mathematical concepts and procedures in algebra, calculus, geometry, and trigonometry to analyze and solve multi-step problems set in routine and non-routine contexts. Intermediate (475) Students demonstrate basic knowledge of concepts and procedures in algebra, calculus, and geometry to solve routine problems.