Crowdsourcing Ling 240

What is crowdfunding?

Crowdsourcing—definition “the practice of obtaining information or services by soliciting input from a large number of [non-expert] people, typically via the internet” (OED) Examples: Wikipedia Google Translate FamilySearch Indexing

COCA's registers based on publication type

Crowdsourcing What are the benefits of collecting data through crowdsourcing? What are the limitations/weaknesses? What can be done to ensure that crowdsourcing workers are giving quality data?

Crowdsourcing in linguistics Wilhelm Kaeding (1897) Thousands of non-experts helped compile and analyze an 11 million word corpus of German Oxford English Dictionary (1858 – 1928) Hundreds of non-expert readers submitted 6 million quotation slips Perceptual dialectology Dialect perceptions elicited from non-experts

Mechanical Turk (Amazon) Strengths Inexpensive Fast Quality control Access to thousands of people Growing body of research strongly supports the quality of MTurk data E.g., Buhmester et al., 2011; Kittur et al., 2008; Suri & Watts, 2011; Urbano et al., 2010

Case study--

Register classification Traditional ‘user’-based approach ‘Expert’ classifies texts into registers by simply sampling from the publication type of interest Limitations ‘Publication type’ is not a meaningful criterion for web documents Experts can’t agree on register category for internet texts

Corpus Extracted from the Corpus of Global Web-based English (GloWbE), constructed by Mark Davies (Near) random sampling methods used to build the corpus Google searches of highly frequent English 3-grams (e.g., is not the, and from the) used to identify URLs links for each n-gram (i.e., Google results pages) Davies randomly extracted c. 49,300 URLs from GloWbE Only web pages from USA, UK, Canada, Aus., and NZ Documents < 75 words were excluded Non-textual material was removed from all web pages (HTML scrubbing and boilerplate removal) using JusText 1,445 URLs were excluded from subsequent analysis because they consisted mostly of photos or graphics. Final corpus for the study: 48,555 web documents.

People asked to determine mode of passage, then participants, purpose, etc. This led to 7 sub-registers

Crowdsourcing end-user data: Classification Developed a computer-adaptive survey for register classification Tested the tool through 10 rounds of piloting, resulting in numerous revisions Recruited 908 raters through Mechanical Turk 6 responses x 4 raters x 49,300 texts = 1.2 million individual ratings

Agreement results for the general register classification of 48,147 web documents (Fleiss’ Kappa =.47, moderate agreement) 69% of documents achieved majority agreement Additional 11.8% are potential 2-way hybrids

Frequencies of general register categories (i.e., documents where 3 or 4 raters were in agreement)

Systematic patterns of disagreement 28 different 2-2 combinations are possible in theory But, only 7 of those combinations occurred > 100 times in our corpus of 48,000 documents Because these are widely attested user-based patterns, we are able to interpret disagreement as a special pattern of agreement

Frequencies of 2-way hybrids that occur 100+ times

Multi-Dimensional analysis Factor analysis to identify dimensions based on co-occurrence among a large set of linguistic features Interpret dimensions functionally Calculate scores for each text on each dimension 17

Features used by Biber adopted: Positive features: Verbs: present tense verbs, mental verbs, do as pro ‑ verb, be as main verb, possibility modals Pronouns: 1st person pronouns, 2nd person pronouns, it, demonstrative pronouns, indefinite pronouns Adverbs: general emphatics, hedges, amplifiers Dependent clauses: that complement clauses (with that deletion), causative adverbial clauses, WH clauses Other: contractions, analytic negation, discourse particles, sentence relatives, WH questions, clause coordination ================================== Negative features: Nouns, long words, prepositional phrases, attributive adjectives, lexical diversity

The results Linguistic (use-based) variation across user-based register categories

Web registers along Dimension 1

What have we learned? Non-expert users can reliably classify web documents At least 1 in 10 internet texts belongs to a hybrid register category Publication type ≠ register (at least for the web) E.g., blogs showed up in several register categories Triangulating end-user classifications with linguistic analysis gives us a more complete understanding of register variation on the web

Web register research: Next steps Comprehensive linguistic description of the patterns of register variation on the web A new multi-dimensional analysis of web registers Detailed linguistic descriptions of ‘unique’ web registers Automatic prediction of register (‘AGI’) Automatically coded large corpus of web documents Extend descriptions to include ‘private’ web registers

Areas for future user-based research Register classification of printed texts Reader/listener perceptions Corpus annotation Word sense disambiguation

5. The future of crowdsourcing in user-based linguistics User-based analyses have always happened; now we can do them in a more valid way using crowdsourcing Triangulating use-based linguistic data offers a more complete understanding of discourse Linguists are often unable to fully analyze and interpret patterns in use-based datasets, particularly those that are very large Harnessing the power of user-based data via crowdsourcing could help us tackle big, difficult problems in linguistics

Mechanical Turk The name comes from an 18 th century machine that played chess. A person actually hid inside and played

Mechanical Turk Amazon's Mechanical Turk is a crowdsourcing tool. Researchers who need human evaluation can get data People who want to make some money help with the project (less than minimum wage) – Image recognition – Speech processing – Subjective evaluation – Giving opinions – Tagging corpora – Match picture with product

Mechanical Turk Example: word sense disambiguation in corpora – What should head be tagged as? Noun or verb? – What does head mean in a sentence? They charged the head of finances with the crime. (person with office) The beer was flat with no head. (froth) They were going head first (manner of movement) Computers can't do it well but people can

How does it work?

Couldn't people cheat? After reviewing results the requester can reject a worker When rejected, they don't get paid Workers have approval rates Requesters can choose only workers with good rates

Advantages Thousands of potential workers available You can get results fast Demographic variety (not just undergrads) Cheap (average $1.40 per hour)

Disadvantages Cheating Some studies show it's at same rates as in lab Ways to test “While exercising how often have you had a fatal heart attack?” It requires money Can't do many types of experiments (RT)

Go look at it Mechanical Turk website