© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR International Comparative Assessments: Using TIMSS to illustrate Assessment Types, Purposes, & Goals VIPP Korean Physics Teacher Summer Institute August 9, 2005 Leland S. Cogan Michigan State University

Focusing solely upon cross-national differences in student attainments presupposes that the entire schooling process – educational goals, curriculum, and pedagogy – are invariant across countries. Such a focus precludes the possibility on constructing useful explanations for these differences – explanations necessary to inform policy and practice…These simple facts must be remembered in the design and execution of any serious cross- national studies of education. Characterizing Pedagogical Flow, 1996, p. 137

The SMSO project tried to marry these two approaches, combining discourse-driven qualitative methodology with traditional quantitative instrument validation methods to develop more sensitive and responsive questionnaire instruments. This hybrid approach is more labor intensive than traditional survey methodology, but the SMSO experience suggests its merits are enough to warrant careful consideration by others engaged in similar enterprises. Characterizing Pedagogical Flow, 1996, p. 23

© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR How long do teachers teach the same students?

© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Critical Aspects of Educational Assessment Assessment Goals – Policy Perspective –Instructional Management –Student Selection/Certification –Accountability & Program Evaluation Purposes of Assessments – Student Perspective –Measuring Learning from Instruction –Measuring Aptitude/Competence –Documenting Achievement Assessment Types –Multiple Choice –Short Answers\ –Extended Constructed Response

Intended Curriculum: system-wide policies, plans, & goals Implemented curriculum: goals, strategies, & practices carried out in classrooms Attained curriculum: pupil knowledge, skills, & attitudes Tripartite Model of Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Intended Curriculum: system-wide policies, plans, & goals Implemented curriculum: goals, strategies, & practices carried out in classrooms Attained curriculum: pupil knowledge, skills, & attitudes Tripartite Model of Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Textbooks?

Intended Curriculum: system-wide policies, plans, & goals Implemented curriculum: goals, strategies, & practices carried out in classrooms Attained curriculum: pupil knowledge, skills, & attitudes Tripartite Model of Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Potentially Implemented curriculum: Textbooks published according to a system’s intentions

From School Districts –Topic Trace Maps From Systems –Textbooks From Teachers –Teacher Content Goals From Students –Student Assessment Sources of Data © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Intended Implemented Attained Potentially Implemented

TIMSS Science Framework © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Examines content and performance expectation Content examines: –What topics are intended? –When are topics intended? Performance expectation examines: –What student performances are expected? Procedure is designed to be Low Inference

TIMSS 1995 Science Framework - An Example © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Examining the Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Curricular Variation Across Systems: The Intended Curriculum

© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Number of Science Topics Intended by Standards for Each Grade

High Achieving Countries’ Science Standards © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

High Achieving Countries’ Science Standards © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

High Achieving Countries’ Science Standards © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

High Achieving Countries’ Life Science Standards

© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR High Achieving Countries’ Physical Science Standards

© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR High Achieving Countries’ Earth Science Standards

© 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR High Achieving Countries’ Environmental Science Standards

Standards Compared with Top-Achieving Countries’ Profile © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Examining the Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Curricular Variation Across Systems: The Potentially Implemented Curriculum

TIMSS Coding Procedure © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Code all pages in each text –Pages are divided into units and blocks Include introduction, all narrative, graphs and pictures Exclude table of contents, index, answer section, glossary, appendices Requires an average of hours per textbook

Distribution of Performance Expectations in Population 2 Textbooks for Chemical Properties of Matter © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Examining the Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Curricular Variation Across Systems: The Implemented Curriculum

Instructional Content Constructs © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Exposure Time (OTL) Curricular Complexity or Degree of Challenge Curricular Structure Level of Cognitive Complexity Curricular Coherence Curricular Focus

Measuring Teachers’ Science Instruction © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Prompt:

Measuring Teachers’ Physical Science Instruction © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Measuring Teachers’ Physical Science Instruction © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Average Percent Teaching Time in Science Areas at Each Grade for District 1 © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Average Percent Teaching Time in Science Areas at Each Grade for District 2 © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Average Percent Teaching Time in Nine Broad Science Areas at Each Grade for District 1 © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Average Percent Teaching Time in Nine Broad Science Areas at Each Grade for District 2 © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

International Grade Placement of Curriculum Content Across Districts © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Grade International Grade Placement (IGP)

Examples of the International Grade Placement (IGP) Index © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Examining the Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR Teacher Subject Matter Readiness

Teachers’ Academic Preparation and Instructional Confidence © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Middle School Science Teacher Preparedness - Self Reported © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

High School Science Teacher Preparedness - Self Reported © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Examining the Curriculum © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR What Students Do: The Attained Curriculum

Number of Science Courses Offered in 7 Districts © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Number of Course Patterns for Meeting High School Science Requirement in 7 Districts © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Earth Science 1Earth Science 2 Biology 1Biology 2 Physical Science 1 Physical Science 2 Gen. Life Sci. 1Gen. Life Sci. 2 Anatomy/Physio. 1 Anatomy/Physio. 2 AP Biology 1 AP Biology 2 Chemistry 1 Chemistry 2 CP Biology 1 CP Biology 2 AP Chemistry 1AP Chemistry 2 Physics 1 Physics II CP Chemistry 1CP Chemistry 2 Master Research Master Program Gen. Sci. 1Gen. Sci. 2 Gen. Sci. 3 Gen. Sci. 4 Diagram of Science Course-Taking Sequences in District F

Average Percent Correct on 1995 TIMSS Science Items © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Average Percent Correct on TIMSS 1995 End-of-Secondary Science Literacy Test © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Boxplots of Average Percent Correct on Physical Science Strands for All PROM/SE Elementary Schools at Each Grade © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Boxplots of Average Percent Correct on Physical Science Strands for All PROM/SE Middle Schools at Each Grade © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR

Average Percent Correct on PROM/SE Science Items by Type of High School Science Courses © 2005 MSU PROM/SE Promoting Rigorous Outcomes in Mathematics and Science Education, Supported by NSF Cooperative Agreement EHR