Mathematics and Science Indicators: Deciding on Measures that Matter Val Plisko National Center for Education Statistics 14th Annual Management Information Systems Conference February 27, 2001

2 Current Status of Indicators Affecting Student Learning Current research points to three broad aspects of school quality that affect student learning: –Training and talent of the teaching force –Classroom activities –School culture and atmosphere Indicators in each of these three areas are currently of differing levels of quality From Monitoring School Quality: An Indicators Report (Dec 2000)

3 Determining Factors in Quality of Current Indicators High quality indicators usually result from easily assessed dimensions or a long history of data collection on the dimension Moderate quality indicators generally lack information on an important facet of the indicator, but still provide some value Poor quality indicators generally suffer from the indicator being more complex than the data

4 Current Quality for Student Learning Indicators High Quality Indicators Teacher Assignment Teacher Experience Teacher Academic Skills Class Size Moderate Quality Indicators Professional Development Technology Course Content Discipline Academic Environment Poor Quality Indicators Pedagogy Goals School Leadership Professional Community

5 Data on Teacher Preparation Teacher subject matter preparation in mathematics and science is related to student achievement Higher student performance associated with more experienced teachers than novice teachers Higher student learning has been connected with teachers who attended higher quality undergraduate institutions, as measured by admissions selectivity Teachers with higher scores on standardized tests have been linked to students with higher test scores

6 Data on Class Size Most analyses of class size have found that smaller classes lead to higher student test scores, particularly for primary-grade students who are minorities or who come from economically disadvantaged backgrounds Yet lowering class size may not enhance student learning without changes to instructional practices or if unqualified teachers are used to reduce class size

7 Data on Goals Researchers agree that successful schools begin with identifying and communicating ambitious goals, then implementing and institutionalizing those goals with broad consensus from key stakeholders Data on school goals are generally lacking –Only nationally representative data come from how public and private school principals responded on a questionnaire to a list of seven general goal statements –Data lack specifics on how these goals are identified, communicated, implemented and institutionalized

8 International assessments offer a macro perspective Enable the United States to benchmark student performance against international averages or countries of particular interest (e.g., G-8 countries) Enable us to think outside of the U.S. box for alternatives to the content, methods and context for teaching

9 Average Eighth-Grade Mathematics and Science Achievement, by Nation: 1999

10 Comparisons of Average Mathematics Achievement, by Nation: 1995 and 1999

11 Comparisons of Average Science Achievement, by Nation: 1995 and 1999

12 Mathematics Achievement Relative to International Average, Fourth Grade TIMSS 1995 and Eighth Grade TIMSS-R 1999

13 Science Achievement Relative to International Average, Fourth Grade TIMSS 1995 and Eighth Grade TIMSS-R 1999

14 The materials presented in U.S. mathematics classrooms is at a lower grade level than that presented in German and Japanese classrooms. The mathematical content presented to U.S. eighth- grade students is of a lower quality than that presented to Japanese eighth-grade students. U.S. eighth-grade mathematics teachers’ typical goal is to teach students how to do something; Japanese teachers’ typical goal is to help students understand mathematical concepts. U.S. eighth-grade mathematics lessons appear to be less coherent than Japanese mathematics lessons. U.S. curriculum covers more topics and spends less time on each topic than in other nations. U.S. curriculum retains topics throughout the K-12 learning experience, whereas other nations introduce and then move to more advanced topics. Factors related to mathematics and science achievement: TIMSS and TIMSS-R Findings

15 Percentage of Lessons Rated as Having Low, Medium, and High Quality of Mathematical Content

16 Eighth-Grade Mathematics Teachers’ Academic Preparation: 1999 *Significant difference between U.S. average and international average in this category.

17 Eighth-Grade Science Teachers’ Academic Preparation: 1999 *Significant difference between U.S. average and international average in this category.

18 It is not the implementation of a national curriculum. It is not the amount of instructional time. It is not the amount of homework assigned by teachers. It is not the presence or use of computers in the classroom. It is not average class size. It is not the amount of time spent watching TV or videos. * This is not to say that such factors do not matter nationally. Factors not* related to mathematics and science achievement among nations in TIMSS

19 TIMSS-R State and District Benchmarking Purpose: to provide data on the mathematics and science achievement of 8th-grade students in participating states and districts in comparison to students nationally and in 37 other nations  Benchmark performance against international average  Compare state/district instructional practices of teachers, student attitudes, and curriculum with that of other nations  Allow state/district to assess the rigor and effectiveness of local math and science programs in an international context

20 TIMSS-R Benchmarking Participants States  Connecticut  Idaho  Illinois  Indiana  Maryland  Massachusetts  Michigan  Missouri  North Carolina  Oregon  Pennsylvania  South Carolina  Texas Districts and Consortia  Academy School District #20 (CO)  Chicago Public Schools (IL)  Delaware Science Coalition  First in the World Consortium (IL)  Fremont/Lincoln/West Side Public Schools (NE)  Guilford County Schools (NC)  Jersey City Public Schools (NJ)  Miami-Dade County Public Schools (FL)  Michigan Invitational  Montgomery County Public Schools (MD)  Naperville School District #203 (IL)  Project SMART Consortium (OH)  Rochester City School District (NY)  SW Pennsylvania Regional Mathematics and Science Collaborative

21 Program for International Student Assessment (PISA) Purpose: to measure 15-year-old students’ knowledge, skills, and competencies in reading, mathematics and science. Nations can compare how their students perform on reading, mathematics, and science in relation to students in other nations Nations can compare themselves to other nations on contextual variables related to reading, mathematics, and science education Nations can gauge the impact of policy changes and reform efforts over time

22 PISA Nations Australia Austria Belgium Brazil Canada China Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea, Republic of Latvia Luxembourg Mexico The Netherlands New Zealand Norway Poland Portugal Russian Federation Spain Sweden Switzerland United Kingdom United States

23 Complementing Indicators with Research on What Matters Indicators should be framed by current research on areas that affect school and teacher quality Indicators can also inform research in these areas, but research needed on macrolevel policies that contribute to quality teaching and learning International assessment can help inform the development of indicators that are most valuable to capture

24 Upcoming International Releases April 4, 2001: TIMSS-R Benchmarking Results Late 2001: Mathematics Results from the TIMSS-R Video Study Late 2001: PISA Results Late 2002: Science Results from the TIMSS-R Video Study

25 For More Information Val Plisko T: (202)