Issues Related to Judging the Alignment of Curriculum Standards and Assessments Norman L. Webb Wisconsin Center for Education Research University of Wisconsin-Madison Annual Meeting of the American Educational Research Association Montreal, April 11, 2005 This work was supported by a subgrant from the U. S. Department of Education (S368A030011) to the State of Oklahoma and a grant from the National Science Foundation, (EHR ) to the University of Wisconsin—Madison. Any opinions, findings, or conclusions are those of the author and do not necessarily reflect the view of the supporting agencies.

Webb Alignment Process Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the Standards/Objectives Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

Reports Standards DOK analysis Degree of alignment by standard Source of Challenge Reviewers’ Notes General Comments Made by Reviewers Reliability Among Reviewers Overall Finding of the Degree of Alignment

Data Tables Acceptable levels on four alignment criteria Source of challenge DOK by item and intraclass correlation Notes DOK level and objective code by item Objectives by item Items for each objective Number of reviewers coding an item by objective

Alignment Criteria  Categorical Concurrence Number of items per standards  Depth-of-Knowledge Consistency Percent below, at, and above  Range-of-Knowledge Correspondence Percent of objectives with at least one item  Balance of Representation Index 0 to 100

What are appropriate acceptable levels for each criterion? Number of items for one standard Distribution of items by complexity Number of standard-objectives with at least one item Distribution of items among objectives under a standard

Alignment Levels Using the Four Criteria Alignment Level Categorical Concurrence Depth of Knowledge Range of Knowledge Balance of Representation Acceptable6 items per standard 50% 70% Weak---40%-49% 60%-69% UnacceptableFewer than 6 items per standard Less than 40% Less than 60%

State A Categorical Concurrence for Grade 3 Science (N=55 items) StandardsHits Cat. Concurr. TitleMeanS.D History/Nature10NO Inquiry YES Unifying Themes7.54YES Subj Matter/Conc YES Design/Applic NO Personal/Social NO Total

State B Depth-of-Knowledge Consistency High School Mathematics (N=51 items) Standards # Hits% Under% At% Above DOK Consistency TitleMMMM I - Patterns, Relationships and Functions NO II - Geometry and Measurement YES III – Data Analysis and Statistics WEAK IV - Number Sense and Numeration YES V - Numerical and Algebraic Operations and Analytical YES VI - Probability and Discrete Mathematics YES Total

State B Range of Knowledge Correspondence High School Mathematics (N=51 items) Standards # Hits# Objs Hit% Objs HitRng. of Know. Title Goals # Objs # Mean I - Patterns, Relationships and Functions NO II - Geometry and Measurement NO III - Data Analysis and Statisti NO IV - Number Sense and Numeration NO V - Numerical and Algebraic Operations and Analytical YES VI - Probability and Discrete Mathematics NO Total

State B Balance of Representation High School Language Arts (3 of 12 standards) (N=116 items) Standards Balance Index Bal. of Represent. % Hits in Std/Ttl Hits Index Title Goals # Objs # MeanS.D.MeanS.D. I. - Meaning and Communication— Reading NO II. - Meaning and Communication— Writing WEAK VIII. - Genre and Craft of Language WEAK Total

What considerations should be given to different item types? How to consider a multiple-point assessment item? What is the trade off between multiple-choice items and open- ended items?

What are issues related to vertical alignment? Appropriate progression of complexity across grades Appropriate progression of content across grades

State A Mathematics DOK Levels for Objectives by Grade

State A Reading Language Arts DOK Levels for Objectives by Grade

Vertical Alignment Questions What level of concurrence is there between objectives for the two grades? To what extent do comparable objectives increase in depth from one grade to the next? To what extent does the range of content increase from one grade to the next? How does the balance of representation change from one grade to the next?

Type of Vertical Relationships Broader Deeper Prerequisite New Identical