1. 1 Bias and Sensitivity Review of Items for the MSP/HSPE/EOC August, 2012 ETS Olympia 1

2. General WCAP Development Cycle Operational Scoring Operational Rangefinding Operational Testing Data Review Pilot Scoring Training and Pilot Scoring Pilot Rangefinding Item Pilot Testing Item Reviews for Content & Bias/Sensitivity by Committee Item Reviews for Content by OSPI Item Writing Training and Item Writing Develop Item and Test Specifications

3. Principles of Universal Design Assessments should be designed to be valid and accessible for use by the widest possible range of students, including students with disabilities and students with limited English proficiency. Results should not be influenced by disability, ethnicity, gender, or English language ability. 3

4. What is Item Bias? Language or content in the item that prevents members of a group from demonstrating their knowledge and skills in a content area; and Systematic error in the measurement process. 4

5. Potential Sources of Bias and Stereotyping Regional and geographic differences in language usage and topic familiarity Linguistic issues (e.g., idiomatic expressions or figurative language) Gender and age stereotypes Ethnic, cultural, and religious stereotypes Socio-economic and occupational stereotypes 5

6. Sensitivity Issues Any reference or language in an item or passage that might cause a student to have an emotional reaction during the test administration and prevent the student from being able to accurately demonstrate knowledge and skills. Examples found in previous bias & sensitivity reviews: Passage about abuse or struggles of an individual or group because of minority status Change of meaning or structure in a culture’s legend or folk tale Competitive context in passages or items Controversial issue unrelated to the content standards Use of patronizing language to describe an individual or group 6

7. How Might Potential Item Bias be Detected? Judgmental Procedure: Review of item for issues related to context, language, or stereotyping prior to field-testing Statistical Procedure: DIF (Differential Item Functioning) analysis following field-tests using item statistics 7

8. Purpose of Bias and Sensitivity Review Review test items for potential sources of bias, sensitive issues, and gender or cultural stereotypes. Apply professional test development standards to ensure items are fair and not insensitive or offensive. 8

9. Read the Bias & Sensitivity Review Guidelines Questions? Comments?

10. Guiding Questions Is there anything controversial, inflammatory, or insensitive in a scenario, reading passage, or item context? Is the language likely to be familiar to most examinees regardless of socioeconomic status, region, language proficiency, cultural background, or gender Is there any apparent patronizing language, stereotypical descriptions, or stereotypical actions/beliefs present? Would an item context or content give students of a particular group, background, or region a distinct advantage or disadvantage? 10

11. Sample Item #1 In the poem “The Bells,” what is the effect of phrases such as the jingling and the tinkling and the moaning and the groaning?  A. They indicate the actual sounds made by the bells.  B. They reflect the beauty of the sounds made by the bells.  C. They provide the required number of syllables in each line.  D. They show that the speaker is gradually losing touch with reality. Bias issue: not accessible to students who are deaf or hard of hearing 11

12. Sample Item #2 Which of the following measures could be the length of a typical Par 4 golf fairway?  A. 400 inches  B. 400 feet  C. 400 yards  D. 400 miles Bias issue: SES and/or prior knowledge 12

13. Sample Item #3 At a carnival, Rolando sees a booth that has a prize wheel. The wheel has 20 sections, and 5 of these sections are labeled ‘Winner.’ If Rolando decides to give the wheel booth a shot, what is the probability that he will win a prize?  A. 0.20  B. 0.25  C. 0.33  D. 0.75 Bias issue: Idiomatic Language “… give the wheel booth a shot…” 13

14. Practice Item Review Read the items Decide whether the content of the items have any of these problems:  Stereotyping or negative representation of any group  Context that is controversial or sensitive  Language or context that may be unfamiliar to a student because of region, language proficiency, socio-economic status, etc.

15. Do you need to review for: Names of the children and adults in math and science items? NO, names are adjusted for gender and ethnic balance once tests are put together Whether the concepts or skills are appropriate for students at the identified grade level? NO, content review committees check that items only measure on-grade level concepts and skills

16. Read the passages, keeping in mind the guiding questions. Make your own judgment. Write your judgment on the review form:  Accept as is (no apparent bias/sensitivity issue)  Edit  Reject due to potential bias or sensitivity issue As a group, discuss each item flagged for revision or rejection. Recorder will take detailed notes. Make a group decision about each item. Record dissenting position, if appropriate How To Proceed 16