1. Using Curricular Measures for Description & Analysis
Surveys of Enacted Curriculum
Using Curricular Measures
for Description & Analysis
SEC Collaborative Meeting
Tampa, FL
February 21, 2005
John L. Smithson, Director, Measures of Enacted Curriculum
Wisconsin Center for Education Research, University of Wisconsin-Madison

2. The SEC Data - Sets Distinctions On-Line Off-line
Data collection/processing/reporting
Descriptive Data
Limited Reporting Options
Easy Access / Indiv. Results
Analytic analyses
Unlimited reporting options
Requires data manipulation

3. Collaborative or Evaluative
Conducting Inquiry Using SEC Data
Forms of Inquiry
Collaborative or Evaluative
Teacher Enrichment
School Improvement
Professional Lrng. Comm.
Program Evaluation
Indicator Reporting
Program Management
Performance Modeling

4. A short history of SEC research
Reform-Up-Close (Porter, Kirst, Osthoff, Smithson, Schneider, 1993)
Validation of teacher self-report survey data.
Upgrading Mathematics (Gamoran, Porter, Smithson, White, 1997)
First content analysis of assessment using content language.
Predictive validity of alignment index comparing instruction & assessments
Data on Enacted Curriculum (Blank, Porter, Smithson, 2004)
Use of SEC data to facilitate school improvement efforts
First content analysis of state standards
MSP-PD Study (In progress: Blank, Smithson, Porter, Garet, Birman)
Use of SEC data for program evaluation

5. Assessments Curriculum Standards
Alignment Relationships in Standards-based Reform
Assessments
Curriculum
Alignment
Standards

6. Alignment Relationships in Standards-based Reform
Intersection of what is taught with what is tested.
Assessment
Instruction
Taught,
tested, and in
the standards
Intersection of what
is taught with what
is in the standards
Standards
Intersection of what is taught with what is in standards.

7. A Quantitative Approach to Alignment
Porter-Smithson/SEC Alignment Process
Content analyses of curriculum documents and reports of practice
by content experts using two-dimensional content language.
Multiple raters (w/ content & assessment expertise) using
independent ratings in combination with team discussions.
Content Description [Topic(s) by Cognitive Demand(s)]
Yields Alignment Index based on:

8. Alignment Analyses Prediction Program Evaluation* Planning
The analytic power of quantitative alignment measures
Prediction
Program Evaluation*
Planning
Performance Modeling
* Not teacher evaluation

9. Alignment Analyses Class Gains Student Gains Alignment Index 0.451
Students perform best when tested on material for which they have been provided the opportunity to learn.
Correlation of Alignment Index to Achievement Gains
Class Gains
Student Gains
Alignment Index
0.451
0.259
From: Gamoran, et.al. (1997). Uprgading High School Mathematics Instruction. EEPA v19n4pp

10. Explaining variation in student learning gains

11. Alignment Analyses for School Improvement
Using alignment as an outcome measure
Alignment Index:
Instruction to Standards
Mathematics
Across 4 Districts
Counts
Treatment 99
Control 124
Leaders 16
(Measuring change in alignment over time)

12. Alignment Analyses for Planning Purposes

13. Mapping Curriculum Materials

14. SEC - Online (

15. Administrative Functions:
Administration Set-up
Review Registrants, Completion Rates
Administrator Report Generator

17. SEC Reports Dynamic Sample Selection Dynamic Data Disaggregation
Floating Bars
Sample and
Disaggregate
Counts
Comparison
Charts
Individual
Results

18. SEC Reports: Instructional Content
Coarse Grain Maps
Content Areas by
Cognitive Demand

19. SEC Reports: Instructional Content
Fine Grain Maps
Topics by
Cognitive Demand

20. Assessments 0.19 Curriculum 0.27 0.18 Standards
Alignment as a Systemic Tool
Assessments
0.19
Curriculum
0.27
Alignment
0.18
Standards
Fine Grain

21. Calculating Alignment
1-((|x-y|(1-n))/2)