BEYOND SKIN DEEP: INVESTIGATING THE “WHO” OF THE SENSITIVITY REVIEW James Grand Juliya Golubovich Ann Marie Ryan Neal Schmitt Society for Industrial & Organizational Psychology Atlanta, GA April 8, 2010

2 Both here and in available sources, we have... – Examined why reviews are important – Investigated what current practices are – Discussed how to develop/approach reviews – Considered when to conduct reviews Answers to two important questions are missing: – Who should conduct reviews? – Do reviews make a difference on test performance? What’s Left to Know About Sensitivity Reviews? << Main Focus << In Progress

3 Current Considerations of the “Who” Most commonly observed procedures/suggestions – Formally train reviewers » Survey of current practices suggests this is rare (33% of reviewers) – Minority review strategy (Camilli, 1993; Office for Minority Education, 1980) » Review panel representative of diverse ethnic, gender, and geographic backgrounds (e.g., ACT, 2006; ETS, 2009) » “Individuals sensitive to differences in cultural groups” (Reckase, 1996)

4 Assumptions – Primary: » Demographic representativeness increases likelihood of identifying material biased or offensive towards members of one’s subgroup – Secondary: » Minorities better calibrated to identify problematic item content as a result of life experiences » Sociopolitical and legal importance of diverse review panel A Closer Look at Minority Review Strategies “Minority review panels...serve important political and social justice goals. For these purposes, it is more appropriate that members of the [sensitivity review] panel represent constituencies pertinent to the use of the test rather than experts on the trait measured by the test.” (cited in Hood & Parker, 1989) >

5 Evidence of Effectiveness of the Minority Review? Currently no evidence exists to directly support whether the demographics of reviewers influence sensitivity judgments – Engelhard, Hansche & Rutledge (1990) » Neither Black nor White reviewers able to correctly identify which items in a test produce DIF at better than chance levels » Furthermore, Black reviewers did not identify more biased items nor were they more confident in their judgments than White reviewers Individual differences likely affect judgments made by sensitivity reviewers, but has not been examined (Engelhard et al., 1990, Engelhard, Davis, & Hansche, 1999; Plake, 1980; Ramsey, 1993)

6 Research Questions 1.Do demographics influence sensitivity review judgments? 2.What individual differences influence sensitivity review judgments?

7 Hypothesis 6 Individuals who are more socially dominant and more strongly endorse status legitimacy beliefs will be less likely to detect insensitivity Hypothesis 5 Individuals with greater cultural intelligence, better perspective taking ability, and greater empathic concern will be more likely to detect insensitivity Hypothesis 4 The effects of gender and ethnicity on the detection of insensitivity will be moderated by gender and ethnic identification Hypothesis 3 Stigma consciousness, perceived attributions to prejudice, and past experience with discrimination will mediate the relationship between reviewer demographics and detection of insensitivity Hypothesis 2 Individuals who are more conscious of stigmas, have a greater tendency to make attributions to prejudice, and have had more experiences with discrimination will be more likely to detect insensitivity Hypothesis 1 Females and non-Whites will be more likely to detect insensitivity than males and Whites Research Model and Hypotheses Stereotype-related Characteristics Gender stigma consciousness Ethnic stigma consciousness Perceived attributions to prejudice Past experience with discrimination Psychological Characteristics Cultural intelligence Perspective taking Empathic concern H1 Gender/ Ethnicity Detection of Insensitivity H2H3 H4 H5 H6 Gender/Ethnic Identification Social dominance orientation Status legitimacy beliefs Stereotype-related Characteristics Gender stigma consciousness Ethnic stigma consciousness Perceived attributions to prejudice Past experience with discrimination Psychological Characteristics Cultural intelligence Perspective taking Empathic concern H1 Gender/ Ethnicity Detection of Insensitivity H2H3 H4 H5 H6 Gender/Ethnic Identification Social dominance orientation Status legitimacy beliefs Demographics Individual Difference Measures

8 Methods n = 303 student reviewers (26% males, 15% non-White) recruited to conduct a sensitivity review on a test of general knowledge – Demographics & individual differences collected online prior to session – Provided brief instructional period on sensitivity reviews prior to review task – Ratings collected on a 108-item test

9 Review Task: Test Breakdown Category#Example Item Stem or Response Options Non-problematic54 The coach worked long and hard into the night for preparing the team’s strategy for the next game. Offensive content7 a. Some whites believing it’s fashionable to be Indian, stretch the truth about their ancestry, claiming, ‘My grandmother was a Cherokee princess.’ Offensive language9 Some religious officials claim that the ancient Egyptian’s history of brutal violence, ritual sacrifices, and worship of non-Christian deities has contributed to the ________ of bloody genocide ravaging Eastern Africa. Emotionally provocative content 11 a. Many single women with children choose not to apply for welfare b. According to a national survey, single childless women choose not to have children because they lack monetary resources. Portrayal of gender/ racial stereotypes 7 Grace Hopper should be an inspiration to female workers everywhere; not only did she prove that a woman could be highly successful in a field dominated by men, and she was able to do so without special treatment or excessive pleas for equality. Unequal referrals to men and women 7 The temperaments of both architects were markedly different; Kevin was reserved and courteous, Joe was ________ and boastful. Vocabulary unfamiliar to a group 7 In order to _________ a mortgage, an individual should periodically pay his or her lender principal and interest. Content unfamiliar to a group 6 In India, seeing an elephant when one is leaving for a journey is considered ______ because an elephant represents Lord Ganesha, the Indian God who _______ obstacles. Problematic Item Categories (54 items)

Calculating Detection of Insensitivity Highly InsensitiveModerately InsensitivePossibly InsensitiveNot Problematic Writing a good twenty-page research paper is more difficult than when you have to write two good ten-page papers. 0 3 Those students who sit through her lectures day after day, having been numbed into thinking that history could never be even remotely interesting. 1 4 The coach worked long and hard into the night for preparing the team’s strategy for the next game. N o n - P r o b l e m a t i c 1 1 Some religious officials claim that the ancient Egyptian’s history of brutal violence, ritual sacrifices, and worship of non-Christian deities has contributed to the ________ of bloody genocide ravaging Eastern Africa. 0 4 The temperaments of both architects were markedly different; Kevin was reserved and courteous, Joe was ________ and boastful a. Some whites believing it’s fashionable to be Indian, stretch the truth about their ancestry, claiming, ‘My grandmother was a Cherokee princess.’ P r o b l e m a t i c Accuracy Score Correct? Mean Item Type Rating RatingItemItem Type Correctly identified if rating = 1-3 Incorrectly identified if rating = 4 Correctly identified if rating = 4 Incorrectly identified if rating = 1-3 Manipulation Check: Significant difference (d = 1.49, p <.001) between problematic item ratings (M = 3.23) & non- problematic item ratings (M = 3.71)

11 Hypothesis 1: – No main effect for Gender in ratings or accuracy – Significant main effects for ethnicity, but... » Non-Whites tended to rate non-problematic items as slightly more insensitive than Whites (d =.19, p <.05) and were less accurate in their judgments overall (d =.36, p <.05) – Significant interaction (p <.05) between Gender & Ethnicity on problematic item ratings Results Detection of Insensitivity Gender Ethnicity ns * Non-white females = White males (Inconsistent with hypothesis)

12 Results Hypothesis 2: – Individuals more aware of ethnic stigmas tended to rate problematic items as more insensitive (β = -.19, p <.01)... –...however, these individuals also tended to rate non- problematic items as more insensitive (β = -.21, p <.01) Stereotype-related Characteristics Gender stigma consciousness Ethnic stigma consciousness Perceived attributions to prejudice Past experience with discrimination Detection of Insensitivity *

13 Hypothesis 3: – Non-Whites tended to exhibit greater awareness of ethnic stigmas, which resulted in a tendency to view both problematic and non-problematic items as more insensitive Results Ethnicity Ethnic Stigma Consciousness Problematic item ratings Non-problematic item ratings β = -.24 β = -.19 β = -.21 * * *

14 Results Hypothesis 4: – Gender/ethnic identification did not moderate the relationship between demographics and sensitivity judgments Gender/ Ethnicity Detection of Insensitivity Gender/Ethnic Identification ns

15 Results Hypothesis 5: Hypothesis 6: – More socially dominant (β =.14, p <.05) and status legitimizing (β =.11, p =.06) individuals tended to rate problematic items as less insensitive – More socially dominant individuals also tended to produce less accurate sensitivity judgments (β = -.12, p <.05) Psychological Characteristics Cultural intelligence Perspective taking Empathic concern Detection of Insensitivity H5 H6 Social dominance orientation Status legitimacy beliefs ns *

16 Summary of Findings No support obtained for minority review strategy – Non-Whites appeared to perceive all items as generally more problematic Stereotype-related and psychological individual difference variables explained meaningful variance in sensitivity ratings – Part of the reason non-Whites tended to be slightly over-sensitive to item content was attributable to their greater awareness of ethnic stigmas – Individuals who believed in the legitimacy of social hierarchies and the need to earn one’s status in society were less sensitive to problematic content

17 Implications Practical Implications – Minority review strategy could potentially increase the rate of false alarms » Problematic as it lengthens development process and time-to-market estimates – Diversity in the sensitivity review panel can still be beneficial to the process (Hood & Parker, 1989), but it should not be the end goal Research Implications – Some individual difference predictors of sensitivity judgments were identified, but many more failed – Continued investigation of psychological characteristics, dispositions, and traits that influence the sensitivity review process

BEYOND SKIN DEEP: INVESTIGATING THE “WHO” OF THE SENSITIVITY REVIEW James Grand Juliya Golubovich Ann Marie Ryan Neal Schmitt Society for Industrial & Organizational Psychology Atlanta, GA April 8, 2010

19 Future Directions What effect does insensitive item content have on actual test performance? – At this point, there is no evidence that speaks to the effect of sensitivity reviews on test taker performance (Engelhard et al., 1999; Ployhart & Holtz, 2008) Can item-level insensitivity produce DIF? What types of individuals are most at risk from performance deficits related to insensitive item content? *We have collected data to begin examining these issues directly, but are not yet in a position to comment on any specific results.*

20 Measures & Statistics Measure α Source Gender Stigma Consciousness.55Pinel, 1999 Ethnic Stigma Consciousness.63Pinel, 1999 Gender Identification.74Luhtanen & Crocker, 1992 Ethnic Identification.82Luhtanen & Crocker, 1992 Perceived Attributions to Prejudice.68Branscombe, Schmitt, & Harvey, 1999 Past Experience with Discrimination.92N/A Cultural Intelligence facets.78 to.82Ang et al., 2007 Perspective Taking.76Davis, 1980 Empathic Concern.82Davis, 1980 Social Dominance Orientation.90Sidanius & Pratto, 1999 Status Legitimacy Beliefs.79Major et al., 2002