Guidelines for Accessible Assessment Project: Research Results Laurene Christensen, Ph.D., Research Associate, National Center on Educational Outcomes Vitaliy Shyyan, Ph.D., Research Associate, National Center on Educational Outcomes

Project Goal To develop research-based guidelines for the representation of Common Core State Standards assessment content in audio and sign

Methodology Cognitive labs with elementary, middle, and high school students Students selected based on the following criteria: Print disabilities Low vision English language learners Deaf, hard of hearing, and ASL users Participating students were from states in PARCC, Smarter Balanced, and unaffiliated states Students took a short math assessment with isomorph pairs Research teams observed the student during the assessment and asked follow-up questions

Research Instruments Cognitive lab protocols Demographic/login information Audio/sign item sets Post-test questions Computer-based test items

Demographic/Login Information School # Student # Grade Need (deaf/hard of hearing; reading-based learning disability; English language learner; low vision) Student gender Test login ID Note taker Date

Audio and Sign Item Sets Grades 3-5Grades 6-8Grades 9-12 Audio12 items (6 sets)16 items (8 sets) Sign12 items (6 sets) 9 items (4 sets)

Post-test Questions AudioSign When there were answer options, the read aloud said, “answer” before each answer option (like “Answer A, Answer B”). Was it clear what the answer options in each question were? Did you like having the word “answer” read before the answer option, or would you prefer “Answer Option A, Answer Option B,” or just “A, B,” or nothing at all? For how many years have you received a read aloud accommodation during testing? Do you use a read aloud accommodation in the classroom (outside of testing)? Does someone read aloud to you at home? Do you use read aloud computer programs? Do you believe that using a read aloud accommodation helps you understand test questions better? What did you think about the voice of the read aloud we used today (quality, speed, etc.)? For how many years have you communicated in ASL? Are members of your family deaf or hard of hearing? At what age was the onset of your hearing loss? Do you view ASL videos in the classroom (outside of testing)? When taking this practice test, did you usually read the item first or view the ASL video first? Do you believe that viewing ASL videos helps you understand test questions better?

Grade 3-5 Audio: Set 1, Item 1 For the first question, the places of each house were read aloud.

Grade 3-5 Audio: Set 1, Item 2 For the second question, places of each classroom were not read aloud.

Grade 3-5 Audio: Set 2, Item 1 For the first question, the student told when the student dropped a soybean in the field and how many fields were planted.

Grade 3-5 Audio: Set 2, Item 2 For the second question, the student was not told anything when the student dropped a corn kernel on a field.

Grade 3-5 Sign: Set 3, Item 1 In the first question, the equations were not signed.

Grade 3-5 Sign: Set 3, Item 2 In the second question, the equations were signed.

Grade 3-5 Sign: Set 4, Item 1 In the first question, the signer used the ASL signs for right angle, symmetry, and length.

Grade 3-5 Sign: Set 4, Item 2 In the second question, the signer used ASL signs and finger-spelled those terms.

Grade 6-8 Audio: Set 4, Item 1 For the first question, each room was highlighted as it was read aloud.

Grade 6-8 Audio: Set 4, Item 2 In the second question, the room names were only read aloud when the student clicked on them.

Grade 6-8 Sign: Set 5, Item 1 In the first question, the signer used the ASL sign for estimates.

Grade 6-8 Sign: Set 5, Item 2 In the second question, the signer used ASL signs and fingerspells estimates.

Grade 9-12 Audio: Set 7, Item 1 For the first question, the picture was described.

Grade 9-12 Audio: Set 7, Item 2 For the second question, the picture was not described.

Grade 9-12 Sign: Set 3, Item 1 In the first question, the signer presented the question twice, once at the beginning of the item and again after the money information.

Grade 9-12 Sign: Set 3, Item 2 In the second question, the signer presented the information in the order it appears in English.

Student Demographics: Gender

Student Demographics: Accommodation

Student Demographics: Grade Elementary SchoolMiddle SchoolHigh School 3 rd 4 th 5 th 6 th 7 th 8 th 9 th 10 th 11 th 12 th

Student Demographics: Need Elementary SchoolMiddle SchoolHigh School NeedNumberPercentageNumberPercentageNumberPercentage Low Vision 311%413%1030% Deaf/HH 1657%1445%1648% LD-Reading 518%1032%721% ELL 414%310%0--

Student Preferences: Elementary Audio

Student Preferences: Middle School Audio

Student Preferences: High School Audio

Insights from Students: Audio It’s easier to break down a problem when you are listening. (205) I liked how it actually told me what the equation was. (317) The read aloud accommodation is helpful sometimes because sometimes I can’t really focus on what I’m reading. (210) Yes, [helpful], it tells you the question, and you can better think about the answer and what’s the problem. … [equations read aloud] You know better the problem – you can think about it faster if the computer tells you the problem. (112) [when shapes are described] You don’t have to think that hard. It helped me answer better. (106) It’s easier to break numbers down when you are listening and taking notes. (205) [usefulness of read aloud] Sometimes there is a word, and I’m not sure what it is, so it helps when I am told what the word is. (216) Helpful. Sometimes I can’t read stuff correctly or don’t understand by myself. If they read for me, it’s easy to know. (213) Preferred the one with the numbers read – it is unnatural not to have the numbers read. … Yes, a whole lot [helpful] because I really don’t struggle on words, it tells me what it is, it helps me better when somebody reads to me. (209) I like it when I read in my head. I get more focused. Another voice is distracting. (303)

Insights from Students: Audio Quality Yes, I liked the voice, it was a little slow. (201) The voice is clearer than some I heard before. (109) It sounded like a robot. (303) It sounded pretty much like Siri from my phone. A bit too fast. (102)

Student Preferences: Elementary Sign Item SetItem DescriptionFirst ItemSecond ItemNo Preference 1 Coordinate Grids/Use of Space Number880 Percent50% -- 2 Drag and Drop Number592 Percent31%56%13% 3 Equations Number2102 Percent14%71%14% 4 Finger Spelling Number861 Percent53%40%7% 5 Plurality and Repetition Number770 Percent50% -- 6 Signing Answer Options Number653 Percent43%36%21%

Student Preferences: Middle School Sign Item SetItem DescriptionFirst ItemSecond ItemNo Preference 1 Coordinate Grids/Diamond Presentation Number580 Percent38%62%-- 2 Drag and Drop/Presentation of Number Line Number760 Percent54%46%-- 3 Equations Number381 Percent25%67%8% 4 Floor Diagram/Point of View Number471 Percent33.3%58.3%8.3% 5 Finger Spelling Number580 Percent38%62%--

Student Preferences: High School Sign Item SetItem DescriptionFirst ItemSecond ItemThird ItemNo Preference 1Finger Spelling Number0880 Percent--50% -- 2Equations Number1220 Percent86%14%-- 3Diamond Presentation Number1102 Percent85%--15% 4Political Cartoons Number1320 Percent87%13%--

Insights from Students: Sign Some of the English words I don’t know so I can watch the sign. (262) I liked the one with the picture describe. In deaf culture we do describe photos. (353) Watching videos is helpful – if I don’t understand something, I can get more information. (362) I can understand questions better when they are signed, not finger-spelled. (156) ASL is our language. English is not bad but ASL is my language. … When you read the English, it is hard to understand. But the signs help. (167) We don’t have laptops. But if I had a test on a computer – I’m not sure about videos. You have to learn English. (365)

Questions and Comments