1. It’s 2019: Do We Need
“Super” Attention Check Items to Conduct Web-Based Survey Research? The Evolution of MTurk Survey Respondents
Kateryna Sylaska, Ph.D., Carthage College
John D. Mayer, Ph.D., University of New Hampshire
Association for Research in Personality
June 28, 2019

2. Why Do We Need Attention Checks?
Low control over testing conditions in online-surveys (e.g., Johnson, 2005)
Participant distraction and “multi-tasking” (e.g., Chandler et al., 2013)
Participant satisficing to reduce cognitive demand (e.g., Oppenheimer et al., 2009)
To support the integrity of our research (e.g., Curran, 2015; Mead & Craig, 2012)

3. Standard Screening Missing > 50% of survey Speedy Completion
Longstring Responding
Attention Check Items

4. Attention Check Items
For a number of years, very simple attention-check items were sufficient

5. But now things appear to be changing
Attention Check Items
But now things appear to be changing

6. How do we know things are changing?

7. Evidence from Research with the Test of Personal Intelligence
Personal Intelligence (Mayer, 2008; 2014)
Ability to reason about ourselves and others based on personality information
Recognize personality information
Form accurate models of personality
Use models to guide choices and make future plans
Test of Personal Intelligence (TOPI)
Objective, research-based questions

8. TOPI
A person is straightforward and modest. Most likely, she also could be described as:
Valuing ideas and beliefs
Active and full of energy
Sympathetic to others and “tender minded”
Self-conscious and more anxious than average
Research into traits indicate that people who are straightforward and modest are also likely to be more tender-minded and sympathetic to others rather than hard-headed.

9. the good old days ( )

10. Mturk Sample on the TOPI-MINI-12 Data Collected February, 2013 (reported in Mayer et al, 2018, Study 1)
M = 0.25  Expectation if Randomly Responding
“Employees high in PI…”
(attention checks employed)

11. College and Mturk Samples on TOPI-MINI-12 Data Collected January-April 2016 (Sylaska & Mayer, 2019)
College (N = 299 for MINI)
no attention checks
Mturk (N = 468 for MINI)
attention checks

12. College Sample of TOPI-MINI-12 Data Collected 2017-2018 (Sylaska, 2019a)
(attention checks employed)

13. Now

14. Mturk Sample for TOPI-MINI Collected December 2018 (Sylaska & Mayer, 2019b)
What’s wrong with this picture?
The negative skew has disappeared
Nearly half appear to be answering at or near a random level
M = 0.25  Expectation if Randomly Responding

16. Turns out, we weren’t the only ones to notice
Major attention to this issue last August

17. Attempt to Solve the Problem

18. New Sample Paid for 150 participants Removed 25 for speedy completion

19. Covert Attention Check Item
Original Attention Check Item
Covert Attention Check Item

20. Comparing Original and Embedded Attention Checks

21. AFTER eliminating participants based on passing 50%+ attention checks
Evaluating TOPI
Traditional Attentional Checks
Covert Attention Checks
AFTER eliminating participants based on passing 50%+ attention checks

22. Cost Consideration Paid for 150 participants
Removed 25 for speedy completion
Removed 53 for failing traditional and covert attention checks
Final N = 72
48% return on investment
Likely still keeping some inattentive responders
Expected mean for TOPI is closer to .80 (mean for using these criterion is .70)

24. Other Solutions IP Address Collection GPS Coordinate Tracking
Open-Ended Response Comparisons
Embedded Activity Tracking (e.g., TaskMaster)
Dennis et al., 2019; Kennedy et al., 2019; Permut et al., 2019

25. References
Chandler, J., Mueller, P., & Paolacci, G. (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. doi: /s
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, doi: /j.jesp
Dennis, S. A., Goodson, B. M., & Pearson, C. (March 14, 2019). Virtual Private Servers and the limitations of IP-based screening procedures: Lessons from the MTurk quality crisis of doi: /ssrn
Johnson, J. A. (2005). Ascertaining the validity of Web-based personality inventories. Journal of Research in Personality, 39, 103–129. doi: /j.jrp
Kennedy, R., Clifford, S., Burleigh, T., Jewell, R., & Waggoner, P. (October 24, 2018). The shape of and solutions to the MTurk quality crisis. doi: /ssrn
Mayer, J. D. (2008). Personal intelligence. Imagination, Cognition and Personality, 27,
Mayer, J. D. (2014). Personal intelligence: The power of personality and how it shapes our lives. New York: Farrar, Straus and Giroux.
Mayer, J. D., Lortie, B., Panter, A. T., & Caruso, D. R. (2018). Employees high in personal intelligence differ from their colleagues in workplace perceptions and behavior. Journal of Personality Assessment, 100,
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, doi: /a
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi: /j.jesp
Permut, S., Fisher, M., & Oppenheimer, D. M. (2019). TaskMaster: A tool for determining when subjects are on task. Advances in Methods and Practices in Psychological Science, 2, 188–196. doi: /
Sylaska, K. (2019). [Monmouth College students and choosing a major.] Unpublished raw data.
Sylaska, K., & Mayer, J. D. (2019a). Major Decisions: Personal intelligence and reasoning about college major contribute to success. Manuscript submitted for publication.
Sylaska, K., & Mayer, J. D. (2019b). [Personal intelligence and choosing a college major.] Unpublished raw data.

26. Thank you