TILSA Alignment Tool Dissemination Workshop July 25 and 26, 2005 WYNDHAM Hotel Boston, Massachusetts Funded by the U.S. Department of Education through a contract to the state of Oklahoma and subcontracts to CCSSO, WCER, HumRRO, and Tindal.

Alignment Powerful Tool for Focusing Instruction, Curricula, and Assessment

Agenda July 25, :30 to Noon Speakers: Norman L. Webb Lauress L. Wise Gerald Tindal NoonLunch 1:00 to 5:00 PM Concurrent sessions Session A: Using the WAT Session B: Interpreting reports and coordinating an alignment study

Agenda July 26, :30 to 12:30 PM Concurrent sessions Session A: Using the WAT Session B: Interpreting reports and coordinating an alignment study 12:30 Lunch 1:30 to 3:00 Plenary Technical issues with the CD alignment system General questions and closing

Alignment Issues Vertical Alignment Grade to grade content linkages Lauress Wise Alternate Assessment Alignment Operationalize the process Gerald Tindal Webb Alignment Process Norman Webb, Rob Ely, Meredith Alt, & Brian Vesperman

Workshop Expectations Set up of an alignment study Responsibilities of a group leader Responsibilities of reviewers Coding procedures Special features How to conduct an alignment analysis of standards and assessments with the WAT

Alignment The degree to which expectations and assessments are in agreement and serve in conjunction with one another to guide the system toward students learning what is expected.

Standards CurriculumAssessment

Degree of Alignment Standards Assess- ment Assessment Standards Assessment Assessment Items

Alignment Process Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the Standards/Objectives Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

Specific Criteria Content Focus A. Categorical Concurrence B. Depth-of-Knowledge Consistency C. Range-of-Knowledge Correspondence D. Balance of Representation and Source of Challenge

Depth of Knowledge Level 1 Recall Recall of a fact, information, or procedure. Level 2 Skill/Concept Use information or conceptual knowledge, two or more steps, etc. Level 3 Strategic Thinking Requires reasoning, developing plan or a sequence ofsteps, some complexity, more than one possible answer. Level 4 Extended Thinking Requires an investigation, time to think and process multiple conditions of the problem.

Items by Objectives Report LowMediumHigh root I

What alignment is good enough?

Alignment Levels Using the Four Criteria Alignment Level Categorical Concurrence Depth of Knowledge Range of Knowledge Balance of Representation Acceptable6 items per standard 50% 70% Weak---40%-49% 60%-69% UnacceptableLess than 6 items per standard Less than 40% Less than 60%

Categorical Concurrence State B Grade 8 Mathematics

Depth-of-Knowledge Consistency State B Grade 8 Mathematics

Balance Index State B Grade 8 Mathematics

Coding Process Tips One Primary Objective and up to Two Secondary Objectives (if necessary) Source of Challenge (a correct/incorrect response for the wrong reason) Notes (any insights to share) Consider Full Range of Standards Use generic objectives sparingly

Structure of the Automated Alignment Process Registration Group Leader Reviewers Standards/Goals/Objectives Entry Process Training on Depth-of-Knowledge Levels Phase I Consensus Process on Assigning DOK Levels to Objectives Phase II Coding of Assessment Tasks Phase III Analysis of Coding Phase IV Reporting

Web Sites Alignment Tool http: // Survey of the Enacted Curriculum

EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS CONTENT AREA: GEOMETRY

EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS CONTENT AREA: PROBABILITY AND STATISTICS

New Cubes Your school is planning a casino night to raise funds to construct a wall aquarium in your school. As a mathematics student, you are given the job of developing a dice game for this event. A regular pair of “number dice” consists of two cubes, each with its faces numbered 1 through 6. Often, dice games are played by rolling the two dice and then finding the sum of the two numbers turned upward. 1. Show that, with a regular pair of number dice, the probability of rolling a sum of 7 is greater than the probability of rolling any other sum.

Coordination of an Alignment Institute Identify Content areas Grade levels Number of test forms Number of reviewers Computer facilities Standards and their structure

Coordination of an Alignment Institute Ask if Tests include field test items Items have different point values Alternate assessments will be included English Language Learners will be included

WAT Adoption to State Needs Assessment development (front end alignment) District and local assessments Test to test comparison analysis Curriculum to standard analysis

Grades 9–12 Science Objectives and Depth-of- Knowledge (DOK) Levels for Michigan Alignment Analysis StandardNumber of Objs. Under Standard DOK Levels of Objs. # of Objs by DOK Levels % of Objs by DOK Levels Total

Comparison of Six Science Assessments on Categorical Concurrence StandardCategorical Concurrence MEAPACTPLANSAT Biology SAT Chemistry SAT Physics I YYYYNN II NNNNNN IIIYNNYNN IVY NNNYY VYNNNNN

Comparison of Six Science Assessments on Balance of Representation StandardBalance of Representation MEAPACTPLANSAT Biology SAT Chemistry SAT Physics I YYWYn/a II n/a IIIYn/a W IVY n/a WN VW

Cross-Assessment Summary

Three Analytic Methods Common Framework Expert Consensus Common Criteria

Survey of the Enacted Curriculum Time on Topic Number sense/Properties/ Relationships Memorize Facts/etc. Perform Procedures Demonstrate Understanding of Mathematical Ideas Place value Whole numbers Operations Fractions Decimals Percents

Enacted, Intended, and Assessed Curriculum Enacted—What teachers teach Intended—What standards require Assessed—What state tests

Achieve Matrix Grade 3 Mathematics Data Analysis and Probability Obj. #Text of ObjectiveABContent Centrality Type of Performance Centrality Source of Challenge Organize, describe and make predictions from existing data 10.A.1aOrganize and display data using pictures, tallies, tables, charts, or bar graphs A.1bAnswer questions and make predictions based on given data

Achieve Alignment Criteria Item-Standard Match Content Centrality Performance Centrality Source of Challenge Instrument-Standard Match Level of Challenge Balance Range

Alignment Process Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the Standards/Objectives Part II: Code DOK Levels and Corresponding Objectives of Assessment Items