1. TILSA Alignment Tool Dissemination Workshop July 25 and 26, 2005 WYNDHAM Hotel Boston, Massachusetts Funded by the U.S. Department of Education through a contract to the state of Oklahoma and subcontracts to CCSSO, WCER, HumRRO, and Tindal.

2. Alignment Powerful Tool for Focusing Instruction, Curricula, and Assessment

3. Agenda July 25, 2005 8:30 to Noon Speakers: Norman L. Webb Lauress L. Wise Gerald Tindal NoonLunch 1:00 to 5:00 PM Concurrent sessions Session A: Using the WAT Session B: Interpreting reports and coordinating an alignment study

4. Agenda July 26, 2005 8:30 to 12:30 PM Concurrent sessions Session A: Using the WAT Session B: Interpreting reports and coordinating an alignment study 12:30 Lunch 1:30 to 3:00 Plenary Technical issues with the CD alignment system General questions and closing

5. Alignment Issues Vertical Alignment Grade to grade content linkages Lauress Wise Alternate Assessment Alignment Operationalize the process Gerald Tindal Webb Alignment Process Norman Webb, Rob Ely, Meredith Alt, & Brian Vesperman

6. Workshop Expectations Set up of an alignment study Responsibilities of a group leader Responsibilities of reviewers Coding procedures Special features How to conduct an alignment analysis of standards and assessments with the WAT

7. Alignment The degree to which expectations and assessments are in agreement and serve in conjunction with one another to guide the system toward students learning what is expected.

8. Standards CurriculumAssessment

9. Degree of Alignment Standards Assess- ment Assessment Standards Assessment Assessment Items

10. Alignment Process Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the Standards/Objectives Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

11. Specific Criteria Content Focus A. Categorical Concurrence B. Depth-of-Knowledge Consistency C. Range-of-Knowledge Correspondence D. Balance of Representation and Source of Challenge

12. Depth of Knowledge Level 1 Recall Recall of a fact, information, or procedure. Level 2 Skill/Concept Use information or conceptual knowledge, two or more steps, etc. Level 3 Strategic Thinking Requires reasoning, developing plan or a sequence ofsteps, some complexity, more than one possible answer. Level 4 Extended Thinking Requires an investigation, time to think and process multiple conditions of the problem.

13. Items by Objectives Report LowMediumHigh 05.2020248 root I 1781919 3838 4545 1.1.222277777778881010 1010 1010 1919 1919 1919 1919 1919 3838 3838 3838 3838 3838 3838 3838 4545 4545 1.2.1919 1919 1919 2929 3232 3232 3232 3232 3232 4040 4343 4343 4545 4545 4545 4545 1.3.2929 3636 3838 4545 1.4.253535 3535 3535 3535 4141 1.5.22227881010 1010 2525 2525 2525 2525 2929 3636 4141 4545

14. What alignment is good enough?

15. Alignment Levels Using the Four Criteria Alignment Level Categorical Concurrence Depth of Knowledge Range of Knowledge Balance of Representation Acceptable6 items per standard 50% 70% Weak---40%-49% 60%-69% UnacceptableLess than 6 items per standard Less than 40% Less than 60%

16. Categorical Concurrence State B Grade 8 Mathematics

17. Depth-of-Knowledge Consistency State B Grade 8 Mathematics

19. Balance Index State B Grade 8 Mathematics

24. Coding Process Tips One Primary Objective and up to Two Secondary Objectives (if necessary) Source of Challenge (a correct/incorrect response for the wrong reason) Notes (any insights to share) Consider Full Range of Standards Use generic objectives sparingly

25. Structure of the Automated Alignment Process Registration Group Leader Reviewers Standards/Goals/Objectives Entry Process Training on Depth-of-Knowledge Levels Phase I Consensus Process on Assigning DOK Levels to Objectives Phase II Coding of Assessment Tasks Phase III Analysis of Coding Phase IV Reporting

26. Web Sites http://facstaff.wcer.wisc.edu/normw/ Alignment Tool http: //www.wcer.wisc.edu/WAT/index.aspx Survey of the Enacted Curriculum http://www.SECsurvey.org

27. EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS CONTENT AREA: GEOMETRY

28. EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS CONTENT AREA: PROBABILITY AND STATISTICS

35. New Cubes Your school is planning a casino night to raise funds to construct a wall aquarium in your school. As a mathematics student, you are given the job of developing a dice game for this event. A regular pair of “number dice” consists of two cubes, each with its faces numbered 1 through 6. Often, dice games are played by rolling the two dice and then finding the sum of the two numbers turned upward. 1. Show that, with a regular pair of number dice, the probability of rolling a sum of 7 is greater than the probability of rolling any other sum.

37. Coordination of an Alignment Institute Identify Content areas Grade levels Number of test forms Number of reviewers Computer facilities Standards and their structure

38. Coordination of an Alignment Institute Ask if Tests include field test items Items have different point values Alternate assessments will be included English Language Learners will be included

39. WAT Adoption to State Needs Assessment development (front end alignment) District and local assessments Test to test comparison analysis Curriculum to standard analysis

40. Grades 9–12 Science Objectives and Depth-of- Knowledge (DOK) Levels for Michigan Alignment Analysis StandardNumber of Objs. Under Standard DOK Levels of Objs. # of Objs by DOK Levels % of Objs by DOK Levels Total60 12341234 10 43 6 1 17 72 10 1

41. Comparison of Six Science Assessments on Categorical Concurrence StandardCategorical Concurrence MEAPACTPLANSAT Biology SAT Chemistry SAT Physics I YYYYNN II NNNNNN IIIYNNYNN IVY NNNYY VYNNNNN

42. Comparison of Six Science Assessments on Balance of Representation StandardBalance of Representation MEAPACTPLANSAT Biology SAT Chemistry SAT Physics I YYWYn/a II n/a IIIYn/a W IVY n/a WN VW

43. Cross-Assessment Summary

44. Three Analytic Methods Common Framework Expert Consensus Common Criteria

45. Survey of the Enacted Curriculum Time on Topic Number sense/Properties/ Relationships Memorize Facts/etc. Perform Procedures Demonstrate Understanding of Mathematical Ideas 0 1 2 3Place value0 1 2 3 Whole numbers0 1 2 3 Operations0 1 2 3 Fractions0 1 2 3 Decimals0 1 2 3 Percents0 1 2 3

47. Enacted, Intended, and Assessed Curriculum Enacted—What teachers teach Intended—What standards require Assessed—What state tests

48. Achieve Matrix Grade 3 Mathematics Data Analysis and Probability Obj. #Text of ObjectiveABContent Centrality Type of Performance Centrality Source of Challenge Organize, describe and make predictions from existing data 10.A.1aOrganize and display data using pictures, tallies, tables, charts, or bar graphs 81 10.A.1bAnswer questions and make predictions based on given data. 5 20 41 53 66 69

49. Achieve Alignment Criteria Item-Standard Match Content Centrality Performance Centrality Source of Challenge Instrument-Standard Match Level of Challenge Balance Range

50. Alignment Process Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the Standards/Objectives Part II: Code DOK Levels and Corresponding Objectives of Assessment Items