1 Models for Aligning Assessments to Standards Consortia Conference Call August 23, 2005 Regie Stites SRI International

2 Assessment Limits Some (more) Chinese sayings: “Viewing the heavens through a bamboo tube, Measuring the ocean with a spoon.” “Riding a horse, viewing flowers.” “Frog sitting in a well, viewing the sky.”

3 Accountability Demands An American saying: “What gets tested, gets taught.” When West meets East, we’ve got a world of trouble unless … everyone understands the limits of assessment, and we achieve reasonably good alignment of standards, assessment, and instruction.

4 Two Directions for Alignment Horizontal – align content standards to assessments, match content and depth Vertical – align accountability tests to educational systems (curricula, materials content, instruction, student outcomes, stakeholder opinions)

5 How to Align Tests and Standards: Three Methods 1.Sequential – develop content standards and curriculum, then develop standards- based tests 2.Expert review – use expert judgments to evaluate alignment, then select/reject test or explicate alignment 3.Content analysis – use content criteria to guide formal analysis, then select/reject test or explicate alignment

6 How to Analyze Alignment: Four Expert Models 1.‘Webb’ – procedures and criteria developed by Norman Webb 2.‘SEC’ – Surveys of Enacted Curriculum 3.‘Achieve’ – alignment analysis service provided by Achieve, Inc. 4.‘CBE’ – technical assistance from the Council for Basic Education See ‘Dimensions of Comparison’ on page 6 of Models for Alignment Analysis and Assistance to States, CCSSO (2002)

7 How Alignment Analysis Helps 1.Provides feedback for revision of assessments or standards 2.Provides information to help align instruction with assessments and standards 3.Helps everyone understand the proper limits for interpreting results of accountability tests

8 Metaphors for Alignment Goals Alignment as Congruence – assessment and standards content mirror each other Alignment as a Set of Correspondences – assessment content samples from standards content Alignment as a Bridge –curriculum content connects standards and assessments Alignment as Gravitational Pull – standards, assessments, and instruction all reflect common elements of a reform agenda [from Baker, 2004]

9 Achieve’s Alignment Protocol: Four Dimensions 1.Content centrality – test item match to content of related standard 2.Performance centrality – test item match to the type of performance (cognitive demand) of related standard 3.Challenge – source and level of challenge for sets of items 4.Balance and range – match between emphasis and coverage of sets of items and standards

10 Achieve’s Alignment Protocol: Three Stages Stage 1: Item-by-item analysis 1(a) Confirming the test blueprint 1(b) Content centrality 1(c) Performance centrality Stage 2: Challenge 2(a) Source of challenge (each item) 2(b) Level of challenge (sets of items mapped to each standard) Stage 3: Balance and range (sets of items mapped to each standard) 3(a) Balance (emphasis) 3(b) Range (coverage)

11 Achieve’s Alignment Protocol: Ratings Stage 1: Content and performance centrality ratings = 2 clearly consistent; 1A not specific enough; 1B somewhat consistent; 0 inconsistent Stage 2: Source of challenge rating = 1 appropriate source(s); 0 inappropriate source(s) Stage 2: Level of challenge rating = narrative evaluation Stage 3: Balance rating = narrative evaluation Stage 3: Range rating = fraction of total objectives (indicators) mapped to standard assessed by at least one item [above.67 = good, between.50 and.66 = acceptable]

12 Achieve’s Alignment Protocol: Results Elementary English Language Arts Content centrality (% of items rated 2) State A = 100; State B = 89; State C = 66; State D = 46; State E = 29 Performance centrality (% of items rated 2) State A = 100; State B = 79; State C = 62; State D = 79; State E = 21 Source of challenge (% of items rated ‘appropriate’) State A = 90; State B = 76; State C = 89; State D = 88; State E = 79 Average range scores State A =.27; State B =.75; State C =.73; State D =.52; State E =.31

13 Some Resources Achieve alignment procedures Robert Rothman et al. (May 2002). Benchmarking and alignment of standards and testing. Metaphors for alignment Eva Baker. (December 2004). Aligning curriculum, standards, and assessments: Fulfilling the promise of school reform. Test publishers see more alignment than teachers Chad Buckendahl et al. (2000). Alignment of standardized achievement tests to state content standards: A comparison of publishers’ and teachers’ perspectives. ‘Nine requirements’ for a responsible state assessment system Commission on Instructionally Supportive Assessment. (October 2001). Building tests to support instruction and accountability: A guide for policymakers.

14 Resources Continued Horizontal and Vertical Alignment Betsy Chase and Sasha Zucker, Harcourt Assessments, Inc. Rethinking Issues of Alignment Under No Child Left Behind Sri Ananda, WestEd.

15 Discussion Points What are your goals for alignment? What do you want to learn from an alignment analysis? What will you do with the results of an alignment analysis? –help educators and stakeholders interpret results of accountability tests? –influence instructional content and practice? –shape policy?