Bringing the crowdsourcing revolution to research in communication disorders Tara McAllister Byun, PhD, CCC-SLP Suzanne M. Adlof, PhD Michelle W. Moore, PhD, CCC-SLP 2014 ASHA Convention Orlando, Florida

Disclosure The individuals presenting this information are involved in recruiting individuals to complete tasks through AMT or other online platforms. This session may focus on one specific approach, with limited coverage of other alternative approaches. Portions of the research were supported by funding from IES. No other conflicts to disclose.

What is crowdsourcing? Traditional method: Assign a task (rating, analysis) to a small number of specially trained individuals. Crowdsourcing: Assign the same task to a large number of non-experts, typically recruited online. Taken individually, experts outperform non-specialists. In the aggregate, crowdsourcing has been successful in solving remarkably complex problems. Foldit: Non-experts playing an online game solved a problem in protein structure modeling that had eluded scientists (Khatib et al., 2011).

What is Amazon’s Mechanical Turk? Amazon’s crowdsourcing platform Requesters electronically post human intelligence tasks (HITs). Members of AMT worker community sign up to complete HITs for payment. What are HITs? Simple, repetitive microtasks Things that humans do better than computers (for now)

Why do they call it Mechanical Turk?

“The man inside the machine” Requester sees only the computer interface, as if task were automated “Artificial artificial intelligence”

Using AMT in research In the past, used primarily for commercial purposes. Recent surge of interest in AMT as vast, inexpensive participant pool for behavioral research. Psychology (e.g. Goodman, Cryder, & Cheema, 2012; Paolacci, Chandler, & Ipeirotis, 2010) Linguistics (e.g. Sprouse, 2011; Gibson, Piantadosi, & Federenko, 2011) Communication sciences and disorders (McAllister Byun, Halpin, & Szeredi, under review) Published studies suggest crowdsourced data are broadly comparable to results collected from typical laboratory samples.

Benefits for use in research Ease of access to participant pool Get away from overused college student population. Inexpensive AMT workers choose whether or not to complete a given task. Crump, McDonnell, & Gureckis (2013) found participants willing to complete a minute study for only $0.75. But important not to be an exploitative requester. Speed of data collection “revolutionary” (Crump et al., 2013). Sprouse (2011): Task that required 88 experimenter hours in the laboratory setting was replicated on AMT in two hours

Points to consider Workers may be less attentive than in lab-based studies (faster clicking  more $). Requesters can screen workers and decline to pay for poor performance. Less control over experimental environment (sound volume, processor speed, background noise level, etc.). Researchers recognize that there is more noise in crowdsourced data than lab-collected data. Idea is to offset this noise by collecting data from larger n of listeners (Ipeirotis et al., 2013).

Getting started: Basics of navigating AMT To post a job… Create an account on Get IRB compliance (if necessary)

Getting started: Basics of navigating AMT Create a task Title and description

Getting started: AMT basics Create a task Compensation offered Number of assignments Time allotted per HIT Click “Advanced” to set preferences for workers: Percent of worker’s previous HITs that were accepted by requesters Worker location (IP address)

Getting Started: Basics of navigating AMT To collect data… Internal HIT: Build a task or survey with AMT’s standard interface External HIT: Link to a task hosted on another website

Getting Started: Basics of Navigating MTurk To verify validity/reliability of data… Reviewing HIT completions

Questions? Interested in trying AMT?