1. Lietta Scott, PhD Arizona Department of Education
How are States using accessibility and accommodation research to improve their tests? Arizona’s Experience
Lietta Scott, PhD
Arizona Department of Education

2. Arizona’s New Online Assessment System
In 2015, Arizona started the move of their Paper-Pencil multiple choice assessments to Online.
We took the opportunity, with AIR to:
Study the effectiveness and validity of an English glossary accommodation for ELs
Examine how various item types translate between the two delivery modes.

3. 2015 Glossary Study - Grades 3 and 7
ELA: 2 reading passages w/associated items
Math: Grade 3 (6 items) – Grade 7 (13 items)
Glossing is different than dictionaries - definitions are context specific
Glossaries included both writing and audio file of the English definition
Rules for glossing based on SBAC’s
(Solano-Flores, 2012)

4. 2015 Glossary Study - Grades 3 and 7
Questions asked:
Are they effective?
Do they help ELs without changing the construct?
How easy is it to implement them?

5. 2015 Glossary Study - Grades 3 and 7
Data:
All students who took our statewide assessment online (40%)
Randomization of students (study/control)
Randomization of items
Included with all other field test items (~130) into 7 field test slots per test

6. 2015 Glossary Study - Grades 3 and 7
Results – Math:
Negative impact (reduced correct responses) for both EL and Non-EL students in both grades
Perhaps glossing words that are irrelevant to getting the item correct is pulling students’ focus from relevant item content.

7. 2015 Glossary Study - Grades 3 and 7
Results – ELA:
Grade 3: Negative impact for both EL and Non- EL students
Grade 7: Positive impact for EL students and No impact for Non-EL students
Perhaps literal interpretation of figurative, metaphorical, or colloquial words was key

8. Arizona’s 2017 Glossary Research
Studied reliability of creating glossary entries
Changed glossing rules
Gloss “culturally bound language” & unusual usages
Didn’t gloss words that were not relevant to solving the particular problem (remove distraction)
Still didn’t gloss words would cue the answer
Studied all grades, both subjects
Added 1 more condition:
English glossary with audio plus Spanish translation of the word with audio (80% of AZ ELs)

9. % of glossed words in common
Glossing is harder than it looks. Without reliable, reproducible glossing, how can it have reliable effects?
Percent of glossed words in common between two individual or teams of raters
Approach to glossing
Training
% of glossed words in common
1 person
1 week, trained with decision tree 10
3 people -
2/3 must agree on gloss
1 week, trained with decision tree and standard reference guide 48
2/3 must agree on selected data*
1 week. All words listed in spreadsheet. Extract information from reference guide and enter into a spreadsheet 59
* Used in data collection

10. 2017 Glossary Study – ELA Group, Condition Grade 3 4 5 6 7 8 9 10 11
Non EL –
Glossary
Non EL – Glossary w/
Spanish translation
EL –
EL – Glossary w/
Negative effect > .95 confidence
Positive effect > .95 confidence

11. 2017 Glossary Study – Mathematics
Group, Condition
Grade 3 4 5 6 7 8 9 10 11
Non EL –
Glossary
Non EL – Glossary w/
Spanish translation
EL –
EL – Glossary w/
Negative effect > .95 confidence
Positive effect > .95 confidence

12. 2017 Glossary Study – Conclusions
Glossaries help Els without affecting non-Els
Need to be done reliably & according to studied guidelines
In ELA may be a distraction in grades 3 & 4
Effects larger in upper grades, especially in Math. Probability of correct response increase:
In lower grades: 1-2 percentage points
In upper grades: 10+ percentage points

13. Item Level Mode Comparability
Ran DIF on items by Mode
Examined the types of items that showed Mode DIF
Explored reasons for Mode DIF by examining how the items display

14. Mode DIF Rare in Both Subjects
ELA items: 1.7% (6 of 350) – All in Grades 3-6
Editing items (drop-down) were harder online
The correct response was visible in the box without clicking. They didn’t make a choice since the sentence was already correct.
Math items: 4.3% (17 of 391) – Across all Grades
Open Equation items were harder than Gridded Response items for all grades
Perhaps due to limiting the number of options available for the student to consider.