National Center on Educational Outcomes What Item Level Data Tell Us About Universal Design: Fantasy, Foolishness, or Fuel for Fire? in Large- Scale Assessments

National Center on Educational Outcomes Presenters: Martha Thurlow, NCEO Liru Zhang, Department of Education, Delaware Karen Barton, Research Triangle Institute, International Brent Garrett, Alliance for Systems Change, MSRRC

National Center on Educational Outcomes Title I Regulations introduced the need for universally designed assessments – [Assessments must be] designed to be accessible and valid with respect to the widest possible range of students, including students with disabilities and students with limited English proficiency. Sec (b)(2)

National Center on Educational Outcomes Proposed Reauthorization of IDEA includes universal design concepts as well – UNIVERSAL DESIGN- The State educational agency (or, in the case of a districtwide assessment, the local educational agency) shall, to the extent possible, use universal design principles in developing and administering any assessments…. Sec 612, Senate Bill 1248

National Center on Educational Outcomes A Quick Definition Universally designed assessments are built from the beginning to be accessible and valid for the widest range of students.

National Center on Educational Outcomes Five categories for consideration of item bias grew out of an international effort (SENDD subgroup of OECD): 1.Bias due to item layout and design 2.Bias due to differential acculturation/lack of experience directly related to disability 3.Bias due to unnecessary or extraneous complexity of item language or length, including bias due to memory load and bias due to verbal load

National Center on Educational Outcomes Five categories for consideration of item bias grew out of an international effort (SENDD subgroup of OECD): 4.Bias due to presentation or response design that differentially affects users of assistive technology or alternate formats 5.Bias due to use of manipulatives that differentially affect students, irrelevant to the construct being measured

National Center on Educational Outcomes Another effort involved reviewing and piecing together information from other fields: Vision Ergonomics Graphic Design Architecture

National Center on Educational Outcomes From this review, NCEO identified a set of elements of universally designed assessments They relate to different points in the item development process – some of these are the responsibility of item writers, some are the responsibility of test contractors; some are the responsibility of both Some have more research behind them than others – so far

National Center on Educational Outcomes Elements of Universally Designed Assessments  Inclusive assessment population  Precisely defined constructs  Items developed and reviewed for bias and accessibility  Amenable to accommodations

National Center on Educational Outcomes Elements of Universally Designed Assessments  Simple, clear, and intuitive instructions and procedures  Maximum readability/ comprehensibility  Maximum legibility: text, graphs, tables, illustrations, and response formats

National Center on Educational Outcomes “Four holds on one of the rock climbing walls are labeled on the diagram below. Matthew first climbs vertically 10 feet from Hold A to Hold B, horizontally 25 feet from Hold to Hold C, and then vertically 15 feet from Hold C to Hold D. How many fewer feet would Matthew have climbed if he had climbed directly from Hold A to Hold D?” Is the use of “hold” as a noun familiar to students? Is the concept of a “rock climbing wall” familiar to most students? Will students be distracted by the odd shapes on the diagram?

National Center on Educational Outcomes Amenable to Accommodations Could this item be presented in an alternate format? Braille? Is the high number of items on the map and long list of cities necessary to respond to this item? “According to this weather page, which place is the warmest on December 28?” If you were flying to Chicago the day this weather page was printed, what information could you learn for your trip from this page?

National Center on Educational Outcomes Legible graphs, tables, illustrations What is that big black rectangle?

National Center on Educational Outcomes Is the border distracting?

National Center on Educational Outcomes Element #5: Simple, clear, and intuitive instructions and procedures Are the swimmers at the bottom of the page distracting?

National Center on Educational Outcomes Abedi research suggested that linguistic complexity of test items was a significant source of measurement error for ELL students (and students with disabilities) Examples of Linguistic Modifications: CensusVideo game Familiarity or frequency of non-math vocabulary (unfamiliar or infrequent words changed): Last year’s class vice president The vice president Length of nominals (long nominals shortened):

National Center on Educational Outcomes Many of the Universal Design elements directly address the characteristics of test items.  Is there a way to tell which items may be a problem without looking at all of the items?  We have procedures that we use to identify potential bias for other subgroups – will these work for students with disabilities?

National Center on Educational Outcomes Universal Design Project at NCEO has suggested three ways to flag items: Statistical analysis of test results (or field test results) Expert review using universal design considerations Think aloud (cognitive labs) with students

National Center on Educational Outcomes Focus Today – Statistical Analysis of Test or Field Test Results Do these approaches work for identifying problematic items? Pros and cons of approaches? Should we instead do as one TAC said – just look at all items?