Using Curricular Measures for Description & Analysis Surveys of Enacted Curriculum Using Curricular Measures for Description & Analysis SEC Collaborative Meeting Tampa, FL February 21, 2005 John L. Smithson, Director, Measures of Enacted Curriculum Wisconsin Center for Education Research, University of Wisconsin-Madison johns@education.wisc.edu

The SEC Data - Sets Distinctions On-Line Off-line Data collection/processing/reporting Descriptive Data Limited Reporting Options Easy Access / Indiv. Results Analytic analyses Unlimited reporting options Requires data manipulation

Collaborative or Evaluative Conducting Inquiry Using SEC Data Forms of Inquiry Collaborative or Evaluative Teacher Enrichment School Improvement Professional Lrng. Comm. Program Evaluation Indicator Reporting Program Management Performance Modeling

A short history of SEC research Reform-Up-Close (Porter, Kirst, Osthoff, Smithson, Schneider, 1993) Validation of teacher self-report survey data. Upgrading Mathematics (Gamoran, Porter, Smithson, White, 1997) First content analysis of assessment using content language. Predictive validity of alignment index comparing instruction & assessments Data on Enacted Curriculum (Blank, Porter, Smithson, 2004) Use of SEC data to facilitate school improvement efforts First content analysis of state standards MSP-PD Study (In progress: Blank, Smithson, Porter, Garet, Birman) Use of SEC data for program evaluation

Assessments Curriculum Standards Alignment Relationships in Standards-based Reform Assessments Curriculum Alignment Standards

Alignment Relationships in Standards-based Reform Intersection of what is taught with what is tested. Assessment Instruction Taught, tested, and in the standards Intersection of what is taught with what is in the standards Standards Intersection of what is taught with what is in standards.

A Quantitative Approach to Alignment Porter-Smithson/SEC Alignment Process Content analyses of curriculum documents and reports of practice by content experts using two-dimensional content language. Multiple raters (w/ content & assessment expertise) using independent ratings in combination with team discussions. Content Description [Topic(s) by Cognitive Demand(s)] Yields Alignment Index based on:

Alignment Analyses Prediction Program Evaluation* Planning The analytic power of quantitative alignment measures Prediction Program Evaluation* Planning Performance Modeling * Not teacher evaluation

Alignment Analyses Class Gains Student Gains Alignment Index 0.451 Students perform best when tested on material for which they have been provided the opportunity to learn. Correlation of Alignment Index to Achievement Gains Class Gains Student Gains Alignment Index 0.451 0.259 From: Gamoran, et.al. (1997). Uprgading High School Mathematics Instruction. EEPA v19n4pp325-338.

Explaining variation in student learning gains

Alignment Analyses for School Improvement Using alignment as an outcome measure Alignment Index: Instruction to Standards Mathematics Across 4 Districts Counts Treatment 99 Control 124 Leaders 16 (Measuring change in alignment over time)

Alignment Analyses for Planning Purposes

Mapping Curriculum Materials

SEC - Online (www.seconline.org)

Administrative Functions: Administration Set-up Review Registrants, Completion Rates Administrator Report Generator

SEC Reports Dynamic Sample Selection Dynamic Data Disaggregation Floating Bars Sample and Disaggregate Counts Comparison Charts Individual Results

SEC Reports: Instructional Content Coarse Grain Maps Content Areas by Cognitive Demand

SEC Reports: Instructional Content Fine Grain Maps Topics by Cognitive Demand

Assessments 0.19 Curriculum 0.27 0.18 Standards Alignment as a Systemic Tool Assessments 0.19 Curriculum 0.27 Alignment 0.18 Standards Fine Grain

Calculating Alignment 1-((|x-y|(1-n))/2)