National Center on Educational Outcomes Universal Design of Assessments Christopher Johnstone, National Center on Educational Outcomes

Goals Define “universally designed” assessments within the current context of large-scale assessments and accountability Identify elements of universally designed assessments Identify relevant research

National Center on Educational Outcomes Universal Design Research? Pieced together from: –Specific accommodations research –Policy statements –Other fields (vision, ergonomics, graphic design, Universal Design of architecture research)

National Center on Educational Outcomes Universally designed assessments : are designed from the beginning to be accessible and valid for the widest range of students provide optimal standard assessment conditions

National Center on Educational Outcomes Who Benefits? Universal design does not apply exclusively to people with disabilities or limited English proficiency It applies to all individuals, with wide ranging characteristics

National Center on Educational Outcomes Think about universal design in architecture and tool design  Curb cuts and ramps  Elevators that talk to you  Door handles rather than knobs  Special pen shapes that are easier to hold

National Center on Educational Outcomes Remember this? OFFICIAL BALLOT, PALM BEACH COUNTY, FLORIDA

National Center on Educational Outcomes Elements of UD Assessments  Inclusive assessment population  Precisely defined constructs  Accessible, non-biased items  Amenable to accommodations

National Center on Educational Outcomes Elements of UD Assessments (continued)  Simple, clear, and intuitive instructions and procedures  Maximum readability and comprehensibility  Maximum legibility

National Center on Educational Outcomes Preliminary Research in Universal Design Sample of 230 students taken from four schools in US Southwest. Two schools were “town” schools (pop. 20,000) and two were “rural” schools. Students chosen from sixth grade teams that had populations of students with disabilities.

National Center on Educational Outcomes Two tests were created, one from sample statewide test items, the other re-designed using UD principles. Each student took both tests. Students randomly assigned to take a particular test first to prevent practice effect. Constructs held constant for each item.

National Center on Educational Outcomes Advisory Board trained in principles of Universal Design and asked to comment / suggest improvements based on their perspectives. Team consisted of three parents of children in special education program (one Navajo, one Latina, one Anglo) and one community member with dyslexia.

National Center on Educational Outcomes Sample Item Ramón is building a doghouse. He wants the roof of the doghouse to be at an angle that is more than 90° but less than 110°. Which angle below could he use for the roof? A.B. C.D.

National Center on Educational Outcomes Revised Item Which angle is more than 90° and less than 110°? A.B. C.D.

National Center on Educational Outcomes What changed?? Design element #2: construct more precisely defined. Design element #3: bias eliminated (dog house, Ramón) Design element #4: “built in accommodations” – un- timed, students circled answer on paper, did not bubble Design element #5: simple instructions and procedures Design element #6: more comprehensible language, underlined key words Design element #7: larger font

National Center on Educational Outcomes Means of two tests were compared and t- tests performed. A difference of 8.16 (1.67 sig.) was found between means, a statistically significant finding. Effect size calculated using Cohen’s d. Effect of design =.61 (or 6/10 Standard Deviation difference) – a “moderate effect”

National Center on Educational Outcomes Students with largest difference between two tests were interviewed to determine difference for them. Students noted that: more direct language made it easier for them to “understand” items and unlimited time helped them to “think better” about items. Students also said they “remembered” content better on UD test.

National Center on Educational Outcomes What have we learned? Design matters!! How a test is designed may effect how a student scores on that test. Items that are better designed appear to aid students that are English Language Learners and with disabilities “show what they know” better. This leads to more valid assessment of traditionally “under-performing” students.

National Center on Educational Outcomes “Simplex Sigillum Veri” “The simple is the seal of the true” Tests that remain true to constructs, are easy to understand, and contain language that is accessible to all will give the truest readings of what students do and do not know. Universal Design does not mean “dumbing down” a test.

National Center on Educational Outcomes More information? Visit: or Search for NCEOhttp://education.umn.edu/nceo Web site includes:  Topic introduction  Frequently Asked Questions  Online and Other Resources including synthesis report and policy directions for Universally Designed assessments.