1. Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice

2. Tools  Teacher surveys of instruction Daily logs End-of-semester or end-of-year surveys  Content analyses of Standards Tests Curriculum materials  Alignment indices—e.g, alignment between assessment and standards

3. Content Matrix

4. Instruction Vertical and Horizontal Alignment Achievement Assessment Standards District State

5. Example Matrices to Measure Alignment.30.1 0 0 0.2.1 0.20.1 0.20.1.2.1 00 AssessmentStandards Topics Cognitive Demand Alignment Index =1 - X=Assessment Cell Proportions Y=Standards Cell Proportions ∑ |X-Y| 2

6. Alignment of Assessments with Standards 7th-Grade Math:Goals Study Average Within-State Alignment =.40 Average Between-State Alignment =.39 Average State-Test-to-NCTM Alignment =.39

7. Alignment of Instruction with Assessment 8th-Grade Math:SCASS Study Average Within-State Alignment =.22 Average Between-State Alignment =.23 Average State-to-NAPE Alignment =.39

8. Average Alignment =.69 Alignment of Instruction with Instruction 8th-Grade Math: SCASS Study

9. 7th Grade Standards State E State FNCTM

10. Quality of Data  Response rates  Interrater agreement for content analyses  Validity of teacher self-report Explaining between-teacher variance in alignment to NAEP Predicting student achievement gains [Note: The need for a reform-neutral language]

11. Uses of Tools Describing Instructional Practices  Research Serve as dependent variable in teacher decision- making research Describe the implemented curriculum Measure implementation of new curricula Assess the validity of transcript studies  Practice Inform teacher reflections on their own instructional practices [Note: Should not be used for teacher accountability.]

12. Uses of Tools Describing Instructional Materials  Research Research effects of textbooks on instruction Assess the breadth and depth of content in instructional materials  Practice Build tests Write content standards Develop national, state, or district indicator systems

13. Uses of Indices of Alignment  Research Serve as a control variable Serve as a dependent variable Serve as a descriptive variable  Practice Align state tests to state standards Align instructional materials to standards or course frameworks

14. Increasing Validity and Value content language  Getting the content language right  Using time samples to describe instruction for an entire school year  Replicating the finding that alignment predicts student achievement gains teacher self-report  Identifying contexts in which teacher self-report on the content of instruction is more or less accurate interrater agreement  Improving the level and consistency of interrater agreement in content analyses distributional properties  Understanding the distributional properties of the alignment statistics powerful professional development  Building powerful professional development programs for data-based decision making  Developing a content language for reading

15. Conclusions  Much progress has been made in recognizing the importance of instructional content as a variable in education research.  Some progress has been made in building tools for including content in education research.  There have been several innovative uses of these new tools in both research and practice, and more are on the horizon.  But there is much more work to be done.

17. 7th-Grade Standards--Close View State EState FNCTM Number Sense and Numeration

18. 7th-Grade Standards--Close View State EState FNCTM Data Analysis and Probability

19. Response Rates for Survey Eisenhower Longitudinal Wave 1 75% Wave 2 74% Wave 3 75% Eisenhower Cross-Sectional 72% Reform Up Close 75%

20. Interrater Agreement AssessmentMeanRange Goals Study.51.77 to.34 CCSSO Study.47.60 to.37 Standards Goals Study.48.59 to.33 [Note: In each study, there was one outlier rater.]

21. Eisenhower Longitudinal Study 42% of variance explained by level (elementary, middle, high school) and subject 27% of variance explained by teachers in the same school 0% of variance explained by between school or between years Longitudinal data on instruction alignment to NAEP yielded:

22. Alignment to Predict Achievement Gains

23. Index Intercorrelations