Innovative Approaches for Examining Alignment Validity in the Context of Computer Adaptive Testing Patricia Reiss, Ph.D. National Conference on Student Assessment (NCSA) Tuesday, June 21, 2016

Alignment Context: Traditional Approach Agenda Alignment Context: Traditional Approach Alignment Approaches to State Assessments in the Context of Computer Adaptive Testing Alignment of Smarter Balanced Assessments by HumRRO Simulation Evaluated Alignment Connection

Evidence Based on Test Content Evidence Based on Response Processes Validity The Standards of Educational and Psychological Testing (AERA, APA, and NCME, 2014) refer to four types of evidence: Evidence Based on Test Content Evidence Based on Response Processes Evidence Based on Internal Structure Evidence Based on Relations to Other Variables

Evidence Based on Test Content Alignment A comparison of traditional alignment models can be found on the CCSSO website at: http://programs.ccsso.org/projects/Align ment_Analysis/Models/ Webb’s (1997, 2002) alignment model

Alignment The assessments cover the full range of content specified in the State’s academic content standards. measure both the content (what students know) and the process (what students can do) aspects of the academic content standards. reflect the same degree and pattern of emphasis apparent in the academic content standards. reflect the full range of cognitive complexity and level of difficulty of the content standards. Yield results that represent all achievement levels US Department of Education, Standards and Assessment Peer Review Guidance, Revised June 25, 2007.

From: Webb (July 25, 2005) TILSA Alignment Tool Dissemination Workshop

Webb’s Alignment Criteria Categorical Concurrence Depth of Knowledge Consistency Range of Knowledge Correspondence Balance of Representation and Source of Challenge

Alignment Approaches to State Assessments in the Context of Computer Adaptive Testing Various approaches Examination of alignment of blueprint and item pool Examination of alignment of a sample of actually administered tests

Alignment U.S. Department of Education Non-Regulatory Guidance, September 25, 2015.

HumRRO’s Alignment Study of the Smarter Balanced Assessments Alignment Study Report posted at http://www.smarterapp.org/documents/Alignmen tStudyReport.pdf Comprehensive evaluation based on evidence-centered design of the summative assessment system

Connections Examined in the HumRRO Alignment Study of the Smarter Balanced Assessments HumRRO (2016) Smarter Balanced Assessment Consortium Alignment Study Report

Alignment Connections Examined Content Specifications and the CCSS (A) CCSS and the Content Specifications (A) Evidence Statements and Content Specifications (B) Content Specifications and Test Blueprints (C) Evidence Statements and Items (D) Items/Performance Tasks and Content Specifications (G)

Alignment Criteria Content Representation DOK Distribution DOK Consistency Agreement between Reviewers’ and Content Specifications/Developers’ Ratings Agreement among Reviewers’ Ratings

Simulated tests included both CAT and PT Evaluated 3 item pools Alignment Connection (F) Simulation-Based Evaluation of the Smarter Balanced Summative Assessments by CRESST http://www.smarterbalanced.org/wp- content/uploads/2016/05/Simulation-based- Evaluation.pdf Simulations conducted for both ELA/L and Mathematics in grades 3-8 and HS Simulated tests included both CAT and PT Evaluated 3 item pools General Spanish Braille

Simulation Description 1000 simulated administrations Evaluated blueprint match Evaluated item exposure rate Examined how well true proficiency scores were recovered, and Examined precision of the score estimates

Alignment Connection F CRESST Findings CAT engine Good estimation of student proficiency Low item exposure rates CRESST results similar to AIR results For PT – difficulty in fulfilling blueprint requirements related to the number of items in the PT section for each claim .

Questions? Patricia.Reiss@SmarterBalanced.org