Innovative Approaches for Examining Alignment Validity in the Context of Computer Adaptive Testing Patricia Reiss, Ph.D. National Conference on Student Assessment (NCSA) Tuesday, June 21, 2016
Alignment Context: Traditional Approach Agenda Alignment Context: Traditional Approach Alignment Approaches to State Assessments in the Context of Computer Adaptive Testing Alignment of Smarter Balanced Assessments by HumRRO Simulation Evaluated Alignment Connection
Evidence Based on Test Content Evidence Based on Response Processes Validity The Standards of Educational and Psychological Testing (AERA, APA, and NCME, 2014) refer to four types of evidence: Evidence Based on Test Content Evidence Based on Response Processes Evidence Based on Internal Structure Evidence Based on Relations to Other Variables
Evidence Based on Test Content Alignment A comparison of traditional alignment models can be found on the CCSSO website at: http://programs.ccsso.org/projects/Align ment_Analysis/Models/ Webb’s (1997, 2002) alignment model
Alignment The assessments cover the full range of content specified in the State’s academic content standards. measure both the content (what students know) and the process (what students can do) aspects of the academic content standards. reflect the same degree and pattern of emphasis apparent in the academic content standards. reflect the full range of cognitive complexity and level of difficulty of the content standards. Yield results that represent all achievement levels US Department of Education, Standards and Assessment Peer Review Guidance, Revised June 25, 2007.
From: Webb (July 25, 2005) TILSA Alignment Tool Dissemination Workshop
Webb’s Alignment Criteria Categorical Concurrence Depth of Knowledge Consistency Range of Knowledge Correspondence Balance of Representation and Source of Challenge
Alignment Approaches to State Assessments in the Context of Computer Adaptive Testing Various approaches Examination of alignment of blueprint and item pool Examination of alignment of a sample of actually administered tests
Alignment U.S. Department of Education Non-Regulatory Guidance, September 25, 2015.
HumRRO’s Alignment Study of the Smarter Balanced Assessments Alignment Study Report posted at http://www.smarterapp.org/documents/Alignmen tStudyReport.pdf Comprehensive evaluation based on evidence-centered design of the summative assessment system
Connections Examined in the HumRRO Alignment Study of the Smarter Balanced Assessments HumRRO (2016) Smarter Balanced Assessment Consortium Alignment Study Report
Alignment Connections Examined Content Specifications and the CCSS (A) CCSS and the Content Specifications (A) Evidence Statements and Content Specifications (B) Content Specifications and Test Blueprints (C) Evidence Statements and Items (D) Items/Performance Tasks and Content Specifications (G)
Alignment Criteria Content Representation DOK Distribution DOK Consistency Agreement between Reviewers’ and Content Specifications/Developers’ Ratings Agreement among Reviewers’ Ratings
Simulated tests included both CAT and PT Evaluated 3 item pools Alignment Connection (F) Simulation-Based Evaluation of the Smarter Balanced Summative Assessments by CRESST http://www.smarterbalanced.org/wp- content/uploads/2016/05/Simulation-based- Evaluation.pdf Simulations conducted for both ELA/L and Mathematics in grades 3-8 and HS Simulated tests included both CAT and PT Evaluated 3 item pools General Spanish Braille
Simulation Description 1000 simulated administrations Evaluated blueprint match Evaluated item exposure rate Examined how well true proficiency scores were recovered, and Examined precision of the score estimates
Alignment Connection F CRESST Findings CAT engine Good estimation of student proficiency Low item exposure rates CRESST results similar to AIR results For PT – difficulty in fulfilling blueprint requirements related to the number of items in the PT section for each claim .
Questions? Patricia.Reiss@SmarterBalanced.org