1. NAEP and TIMSS Mathematics 2015
Samantha burg, ph.d.
STEPHEN PROVASNIK, Ph.D.
National Center for Education Statistics
Washington, DC
Conference board of the mathematical sciences
May 5, 2017

2. National Assessment of Educational Progress (NAEP)
Congressionally mandated project administered by the National Center for Education Statistics (NCES)
Common measure of student achievement across the country in a variety of subject areas since 1969
Gold standard of U.S. large-scale assessments

3. Trends in International Mathematics and Science Study (TIMSS)
Sponsored by the International Association for the Evaluation of Educational Achievement and managed in the U.S. by NCES
Common measure of student achievement and school practices across the world since
TIMSS assesses mathematics and science achievement at 4th- and 8th-grade
TIMSS Advanced assesses advanced mathematics and physics achievement in the final year of secondary school

4. NAEP and TIMSS Similarities
At 4th, 8th, and 12th grades (including TIMSS Advanced)
Curriculum-based assessments
Very similar frameworks
Virtually identical IRT scoring methodology
Include student, teacher, and principal background questionnaires
Differences
Mathematics frameworks have a few differences
Different weight given to the same topic areas
NAEP questions have greater complexity than TIMSS
Achievement levels determined differently
Math frameworks: TIMSS has a wider range and NAEP has harder items, on average
Achievement levels: NAEP’s are aspirational and TIMSS are fixed

5. NAEP 2015 Mathematics
Highlights

6. Mathematics assessment administered January-March 2015
NAEP 2015
Mathematics assessment administered January-March 2015
National samples:
279,000 fourth-graders, 273,000 eighth-graders, and 31,900 twelfth-graders
Results available for:
Nation (grades 4, 8, and 12)
50 states, the District of Columbia, and the Department of Defense school system (grades 4 and 8 only)
21 participating districts (grades 4 and 8 only)
Performance reported as:
Average scale scores
0-500 for grades 4 and 8 mathematics
0-300 for grade 12 mathematics
Achievement levels (Basic, Proficient, Advanced)

7. Fourth-grade average score lower than 2013, higher than 1990
NAEP 2015 MATH
Fourth-grade average score lower than 2013, higher than 1990
GRADE 4

8. Eighth-grade average score lower than 2013, higher than 1990
NAEP 2015 MATH
Eighth-grade average score lower than 2013, higher than 1990
GRADE 8

9. Twelfth-grade average score lower than 2013
NAEP 2015 MATH
Twelfth-grade average score lower than 2013
GRADE 12

10. NAEP 2015 MATH
Percent at or above Proficient lower than 2013, higher than 1990: Grades 4 and 8
GRADE 4
GRADE 8

11. NAEP 2015 MATH
Percent at or above Proficient lower than 2013, higher than 1990: Grade 12
GRADE 12

12. NAEP 2015 MATH
Grade 4 scores increased in 3 states/jurisdictions and decreased in 16 compared to 2013
NOTE: DD = Department of Defense Education Activity (DoDEA).

13. Grade 8 scores decreased in 22 states compared to 2013
NAEP 2015 MATH
Grade 8 scores decreased in 22 states compared to 2013
NOTE: DD = Department of Defense Education Activity (DoDEA).

14. TIMSS and TIMSS Advanced 2015 Highlights

15. TIMSS and TIMSS Advanced
1995
1999*
2003
2007
2011
2015
TIMSS
TIMSS Advanced
TIMSS
TIMSS
TIMSS
TIMSS
TIMSS
TIMSS Advanced
TIMSS
: Mathematics and science assessment at grades 4 and 8
Both developed by the TIMSS & PIRLS International Study Center (ISC) at Boston College, under contract to the International Association for the Evaluation of Educational Achievement (IEA).
TIMSS Advanced also in 2008, but the United States did not participate.
TIMSS Advanced
: Advanced mathematics and physics assessment in the final year of secondary school
* No grade 4 assessment.

16. TIMSS 2015 Participation at 4th grade Participation at 8th grade
55 education systems
More than 310,000 students
Participation at 8th grade
44 education systems
More than 260,000 students
Results for Armenia, which did participate in TIMSS 2015 at grades 4 and 8, are not shown because their data are not comparable for trend analysis.
NOTE: Education systems that administered TIMSS at a grade other than the target grade are not included in the count of education systems.

17. TIMSS Advanced 2015
Participation in the final year of secondary school
9 education systems
France, Italy, Lebanon, Norway, Portugal, the Russian Federation, Slovenia, Sweden, and the United States
More than 56,000 students
In U.S., 12th-grade students who took or were taking advanced courses in mathematics and physics (e.g., AP courses, IB courses)
More than 56,000 students (32,000 in advanced mathematics and 24,000 in physics) participated.
Nearly 5,000 teachers and 3,000 schools completed questionnaires.

18. GRADE 4 MATH
Fourth-grade average score not measurably different from 2011, higher than 1995
1000 * *
90th percentile * * * *
75th percentile * * *
Average score * * *
25th percentile * *
10th percentile
* Score is different from 2015

19. GRADE 4 MATH
U.S. mathematics performance in the international context at the 4th grade in 2015
Singapore
618
Hong Kong-CHN
615
Korea, Rep. of
608
Chinese Taipei-CHN
597
Japan
593
Northern Ireland-GBR
570
Russian Federation
564
Norway
549
Ireland
547
England-GBR
546
Belgium (Flemish)-BEL
Florida-USA
Kazakhstan
544
Portugal
541
United States
539
Denmark
Quebec-CAN
536
Lithuania
535
Finland
Poland
TIMSS Scale Centerpoint
500
Among 54 education systems, average scores in:
10 education systems were higher than the U.S. average
9 education systems were not measurably different from the U.S. average
34 education systems were lower than the U.S. average (not shown)
Average higher than U.S. average

20. GRADE 4 MATH
Education systems with score gains between and 2015: Mathematics 4th grade
1995
2015
Intermediate
(475)
Low
(400)
Advanced
(625)
High
(550)
1000
350
400
450
500
550
600
650
Average Score

21. GRADE 4 MATH
Percentage of grade 4 students reaching the international Advanced benchmark in mathematics in 2015
Avg. score
Advanced (625)
Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning.
618
615
608
597
593
570
564
546
544
539
547
549
529
541
Percent or score higher than U.S. percent or score
Percent or score not measurably different from U.S. percent or score
Score lower than U.S. score

22. GRADE 8 MATH
Eighth-grade average score higher than 2011 on average, higher than 1995 at all percentiles
1000 * * *
90th percentile * * * * * *
75th percentile * * * * *
Average score * * *
25th percentile * *
10th percentile
* Score is different from 2015

23. GRADE 8 MATH
U.S. mathematics performance in the international context at the 8th grade in 2015
Singapore
621
Korea, Rep. of
606
Chinese Taipei-CHN
599
Hong Kong-CHN
594
Japan
586
Quebec-CAN
543
Russian Federation
538
Kazakhstan
528
Canada
527
Ireland
523
Ontario-CAN
522
United States
518
England-GBR
Slovenia
516
Hungary
514
Dubai-UAE
512
Norway
Lithuania
511
Israel
TIMSS Scale Centerpoint
500
Among 43 education systems, average scores in:
8 education systems were higher than the U.S. average
10 education systems were not measurably different from the U.S. average
24 education systems were lower than the U.S. average (not shown)
Average higher than U.S. average

24. GRADE 8 MATH
Education systems with score gains between and 2015: Mathematics 8th grade
1995
2015
Intermediate
(475)
Low
(400)
Advanced
(625)
High
(550)
1000
350
400
450
500
550
600
650
Average score

25. GRADE 8 MATH
Percentage of grade 8 students reaching the international Advanced benchmark in mathematics in 2015
Avg. score
Advanced (625)
Students can apply and reason in a variety of problem situations, solve linear equations, and make generalizations.
621
599
606
594
586
528
538
511
514
512
518
543
493
Percent or score higher than U.S. percent or score
Percent or score not measurably different from U.S. percent or score
Score lower than U.S. score

26. ADVANCED MATH
U.S. advanced mathematics performance in the final year of secondary school in 2015
Among 10 education systems, average scores in:
2 education systems were higher than the U.S. average
2 education systems were not measurably different from the U.S. average
5 education systems were lower than the U.S. average
Russian Federation (intensive courses*)
540
Lebanon
532
TIMSS Advanced Scale Centerpoint
500
United States
485
Russian Federation
Portugal
482
France
463
Slovenia
460
Norway
459
Sweden
431
Italy
422
* Intensive courses are advanced mathematics courses that involve 6 or more hours per week. Results for students in these courses are reported separately from the results for other students from the Russian Federation taking courses that involve 4.5 hours per week.
Average higher than U.S. average
Average lower than U.S. average

27. Advanced Mathematics Coverage Index
U.S. coverage index and advanced mathematics performance in the final year of secondary school in 2015
Russian Federation (intensive courses*)
540
Lebanon
532
TIMSS Advanced Scale Centerpoint
500
United States
485
Russian Federation
Portugal
482
France
463
Slovenia
460
Norway
459
Sweden
431
Italy
422
Advanced Mathematics Coverage Index
* Intensive courses are advanced mathematics courses that involve 6 or more hours per week. Results for students in these courses are reported separately from the results for other students from the Russian Federation taking courses that involve 4.5 hours per week.
Average higher than U.S. average
Average lower than U.S. average

28. ADVANCED MATH
Average advanced mathematics scores and coverage index of TIMSS Advanced students, by education system: 2015
Average Score
Coverage index
1000 * * * * * % * * % % % % % % % % %
*Score is different from U.S. score
NOTE: The advanced mathematics coverage index is the percentage of the corresponding age cohort covered by the students taking the TIMSS Advanced assessment in advanced mathematics. The corresponding age cohort is determined for education systems individually. In the United States, the corresponding age cohort is 18-year-olds.

29. TIMSS Advanced mathematics score changes: 1995-2015
Russian Federation (intensive courses)1
1995
2015
(2, 2)
Intermediate
(475) -
UNITED STATES1
(6, 11)
Advanced
(625) -
Country & Coverage index (1995, 2015)
(20, 21) -
Slovenia1
(75, 34)
High
(550) -
(16, 14) -
(14, 25)
1000
350
400
450
500
550
600
650
Average score
1 Change in average score was not measurably different

30. ADVANCED MATH
Percentage of TIMSS Advanced students reaching the international Advanced benchmark in advanced mathematics in 2015
Avg. score
Advanced (625)
Students demonstrate thorough understanding of concepts, mastery of procedures, and mathematical reasoning skills. They can solve problems in complex contexts in algebra, calculus, geometry, and trigonometry.
540
485
532
460
422
482
431
463
459
Percent or score higher than U.S. percent or score
Percent or score not measurably different from U.S. percent or score
Percent or score lower than U.S. percent or score

31. Comparison of Mathematics Results
NAEP
TIMSS
(Earliest to Current)
NAEP
TIMSS
NAEP
NAEP
TIMSS
(Previous to Current)
Grade 4
NAEP
TIMSS
Grade 8
2015 score is higher
2015 score is not measurably different
2015 score is lower

32. For more information NAEP
For more NAEP results:
Contact:
Samantha Burg, NCES
TIMSS For more TIMSS and TIMSS Advanced results: Contact: Stephen Provasnik, NCES

33. THANK YOU

34. Scores decline for some student groups
Mathematics
Scores decline for some student groups
1 Percentages shown are for grade 4 students in 2015.
# Rounds to zero.
NOTE: The NAEP mathematics scale ranges from
Not available

35. Scores decline for some student groups
Mathematics
Scores decline for some student groups
1 Percentages shown are for grade 4 students in 2015.
NOTE: The NAEP mathematics scale ranges from
Not available

36. Change in average mathematics scores by subscale
Grade 4
Grade 8
Subscale
2015
'15-'13
Composite
240 -1 *
282
-2*
Number properties and operations
243 +1
279
Measurement
238
-1*
Geometry
236
-5*
280
Data analysis, statistics, and probability
-4*
Algebra
288
Significant decrease across subscales in 8th grade
RED highlights areas where NVS study found a misalignment between NAEP framework and common core standards 4th grade-Geometry; 8th grade Data analysis and statistics)
Numerically, those subscales showed the most significant decrease
In both cases, subscales were taught later in common core compared to when NAEP measures it.
Item analysis comparison between NAEP and common core will occur when NAEP transition to computer is complete (2017)
* Statistically significant (p < .05).

37. Change in average scores from 2013 to 2015
Mathematics
Change in average scores from 2013 to 2015
Score change
Grade 4
Grade 8
Both grades
Increase
District of Columbia
Mississippi
Department of Defense schools
Decrease
Arkansas
Connecticut
Georgia
Hawaii
Minnesota
Montana
New Hampshire
New York
Florida
Idaho
Kentucky
Louisiana
Massachusetts
Nevada
North Carolina
Ohio
Pennsylvania
South Carolina
South Dakota
Texas
Washington
West Virginia
Colorado
Delaware
Kansas
Maine
Maryland
North Dakota
Rhode Island
Vermont

38. Grade 4 Mathematics
17 state/jurisdictions score below and 21 score above the nation (public)
NOTE: DD = Department of Defense Education Activity (DoDEA).

39. Grade 8 Mathematics
15 state/jurisdictions score below and 23 score above the nation (public)
NOTE: DD = Department of Defense Education Activity (DoDEA).

40. Scores for racial/ethnic groups unchanged compared to 2013
Mathematics
Scores for racial/ethnic groups unchanged compared to 2013
† Not applicable.
NOTE: The NAEP mathematics scale ranges from at grade 12.

41. Scores for most racial/ethnic groups higher compared to 2005
Mathematics
Scores for most racial/ethnic groups higher compared to 2005
† Not applicable.
* Significantly different (p < .05) from 2015.
NOTE: The NAEP mathematics scale ranges from at grade 12.

42. Score increase for one group since 2013
Mathematics
Score increase for one group since 2013
NOTE: The NAEP mathematics scale ranges from at grade 12.

43. Scores decline in all mathematics content areas since 2013
NOTE: The NAEP mathematics scale ranges from at grade 12.

44. U.S. mathematics scores, by gender: 1995-2015
Grade 4
Grade 8
TIMSS Advanced
1000
1000
1000 * * 5 7 3 2 21 30 * * * * * * * 4 8 6 9 7
* Score higher than other gender
: Gender score gap (M-F)

45. Summary of U.S. results from TIMSS and TIMSS Advanced 2015
2015 Science National Headlines
Summary of U.S. results from TIMSS and TIMSS Advanced 2015
Trends over two decades ( )
Score gains: math at grades 4 and 8; science at grade 8
No change: advanced math at grade 12; science at grade 4 and physics at grade 12
Trends compared to previous cycle ( )
Score gains: math at grade 8
No change: math at grade 4; science at grades 4 and 8
Performance compared to other education systems in 2015
6 systems outperformed the U.S. students in both subjects at grades 4 and 8: Chinese Taipei, Hong Kong, Japan, Korea, Russian Federation, and Singapore.
Gender score gaps in 2015
Gender score gaps favoring males in both subjects at grades 4 and 12; science at grade 8
No gender difference in mathematics at grade 8

46. Example of grade 4 item at advanced benchmark
GRADE 4 MATH
Example of grade 4 item at advanced benchmark
Content Domain: Number
Cognitive Domain: Reasoning
Proficiency Level: Advanced
Percentage of students answering correctly:
U.S. average: 46%
International average: 24%

47. Example of grade 8 item at advanced benchmark
GRADE 8 MATH
Example of grade 8 item at advanced benchmark
Content Domain: Geometry
Cognitive Domain: Reasoning
Proficiency Level: Advanced
Percentage of students answering correctly:
U.S. average: 33%
International average: 32%

48. Example of Advanced Mathematics item at Advanced benchmark
Content Domain: Calculus
Cognitive Domain: Applying
Proficiency Level: Advanced
Percentage of students answering correctly:
U.S. average: 37%
International average: 16%

49. What is on the TIMSS mathematics Grade 4 assessment?
GRADE 4 MATH
What is on the TIMSS mathematics Grade 4 assessment?
Content Domain: Data Display
Cognitive Domain: Knowing
Proficiency Level: Low
Percentage of students answering correctly:
U.S. average: 90%
International average: 84%

50. What is on the TIMSS mathematics Grade 8 assessment?
GRADE 8 MATH
What is on the TIMSS mathematics Grade 8 assessment?
Content Domain: Geometry
Cognitive Domain: Applying
Proficiency Level: High
Percentage of students answering correctly:
U.S. average: 42%
International average: 43%

51. What is on the TIMSS Advanced mathematics assessment?
Content Domain: Algebra
Cognitive Domain: Knowing
Proficiency Level: Intermediate
Percentage of students answering correctly:
U.S. average: 61%
International average: 50%

52. GRADE 4 MATH
Change in U.S. grade 4 mathematics content domain average scores:
1000
2007
2011
2015
Overall mathematics
529*
541
539
Number
543
546
Geometry
522
535*
525
Data
545
540
* Score is different from 2015

53. GRADE 8 MATH
Change in U.S. grade 8 mathematics content domain average scores:
1000
2007
2011
2015
Overall mathematics
508*
509*
518
Number
514
520
Algebra
507*
512*
525
Geometry
480*
485*
500
Data
533*
527
522
* Score is different from 2015

54. TIMSS 2015 international mathematics benchmarks
GRADE 4 MATH
TIMSS 2015 international mathematics benchmarks
Grade 4
Advanced (625)
Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning.
High (550)
Students can apply their knowledge and understanding to solve problems.
Intermediate (475)
Students can apply basic mathematical knowledge in simple situations.
Low (400)
Students have some basic mathematical knowledge.

55. TIMSS 2015 international mathematics benchmarks
GRADE 8 MATH
TIMSS 2015 international mathematics benchmarks
Grade 8
Advanced (625)
Students apply and reason in a variety of problem situations, solve linear equations, and make generalizations.
High (550)
Students can apply their understanding in a variety of relatively complex situations.
Intermediate (475)
Students can apply basic mathematical knowledge in a variety of situations.
Low (400)
Students have some knowledge of whole numbers and basic graphs.

56. TIMSS Advanced 2015 international mathematics benchmarks
ADVANCED MATH
TIMSS Advanced 2015 international mathematics benchmarks
Final Year of Secondary School
Advanced (625)
Students demonstrate thorough understanding of concepts, mastery of procedures, and mathematical reasoning skills. They can solve problems in complex contexts in algebra, calculus, geometry, and trigonometry.
High (550)
Students can apply a broad range of mathematical concepts and procedures in algebra, calculus, geometry, and trigonometry to analyze and solve multi-step problems set in routine and non-routine contexts.
Intermediate (475)
Students demonstrate basic knowledge of concepts and procedures in algebra, calculus, and geometry to solve routine problems.