Human Computation and Crowdsourcing Uichin Lee May 8, 2011

Content Networking Human intelligence: – Distributed human computation, crowdsourcing Mobile device intelligence: – Sensing (camera, GPS) Network intelligence: – Internet, mobile networks (w/ advanced services) Application intelligence: – Agent, processing, mining Internet Smart home/office On the move Applications Content provider Fixed access Content Networking Radio access Device Network Application Human Crowd

Contents Overview Genres of distributed human computation – Games with a purpose, mechanized labor, wisdom of crowds, crowdsourcing, dual-purpose work, grand search, human-based genetic algorithms, knowledge collection from volunteer contributors Dimensions – Motivation, quality, human skill, participation time, cognitive load Analyzing Amazon Mechanical Turk Marketplace

Overview Distributed human computation (DHC) aims at solving rich computation problems through collaboration between humans and computers – Particularly, in some problem domains where humans could be much better than machines – Examples: artificial intelligence, natural language processing, and computer visions Well artificial intelligence has been trying hard to solve these problems using machines – But its quality may be not satisfactory.. DHC offers the possibility of combining humans and computers: faster than individual human efforts, and quality is as good as human efforts (or even better) Taxonomy of Distributed Human Computation Alexander J. Quinn, Benjamin B. Bederson, 2009

Overview How? The system has global knowledge of the problem and forms small sub-problems that take advantage of humans’ special abilities – Delegating sub-problems to a large number of people connected via Internet (could be geographically dispersed) Examples: – Searching for a person in a large number of satellite photos covering thousands of square miles of ocean (e.g., Jim Gray) – Image labeling (e.g., ESP game) – Human-computer interaction, cryptograph, business, genetic algorithms, etc. (and many others!)

DHC Genres Games with a purpose Mechanized labor Wisdom of crowds Crowdsourcing Dual-purpose work Grand search Human-based genetic algorithms Knowledge collection from volunteer contributors People sensing

Games with a purpose Game that requires the player to perform some computation to gain points or to succeed Defining factor: people are motivated by the fun of a game

Mechanized labor Crowdsourcing with monetary rewards Amazon’s Mechanical Turk, ChaCha (paid per micro task) – Cf) Mturk was lunched in 2005 by the needs of Amazon; they wanted to eliminate all the duplicate pages as much as possible which couldn’t be done using automated algorithms

Wisdom of crowds Crowd intelligence: very difficult when done individually, but very easy when aggregated (asking opinions of crowds) Example services: online polling, prediction markets

Crowdsourcing Coined by Jeff Howe in a Wired magazine article – Displacement of usual internal labor by soliciting unpaid help from the general public – Motivated by curiosity or serendipity while browsing the web (e.g., online product reviews) Examples: – Question answering services: Naver KiN, Yahoo Answer, Askville, Aardvark – (finding elusive particles from space images)

Dual-purpose work Translating a computation into an activity that many people were already doing frequently

Grand search Finding a solution (instead of aggregation) Examples: finding an image that contains something (e.g., search for a missing person, or for elusive particles as in

Human-based genetic algorithms Humans contribute solutions to problems and subsequent participants by performing functions such as initialization, mutation, and recombinant crossover Defining factor is that solutions consists of a sequence of small parts and that they evolve in a way that is controlled by human evaluation

Knowledge collection from volunteer contributors Aims to advance artificial intelligence research by using humans to build large databases of common sense facts – E.g., “people cannot brush their hair with a table” Common methods have been using data mining, e.g., Cyc Human-based methods could help, e.g., FACTory, Verbosity, 1001 Paraphrases, etc.

People sensing Community awareness (participatory sensing) Emergency/rescue operations Safecast.org seeks to aggregate worldwide sensor information Geiger counter; 방사능측정기 Pictures from

Dimensions Motivation – Pay (e.g., Mturk), altruism (e.g., Naver KiN, Wikipedia), fun (e.g., games), implicit (e.g., embedded in regular activities) Quality – Mechanisms: forced agreement (e.g., games), economic models (when money is involved), defensive task design, redundancy – Checking: statistical, redundant work, multilevel review, expert review, forced agreement, automatic check, reputation systems Aggregation – Knowledge base, statistical, grand search, unit tasks (ChaCha, Mturk) Human skill – Language understanding, vision, communications, reasoning, common knowledge/sense Participation time: 10min Cognitive load (affecting contributor’s willingness to help)

Analyzing the Amazon Mechanical Turk Marketplace Panagiotis G. Ipeirotis (NYU)

AMT Screenshot

Screenshot

AMT questions Who are the workers that complete these tasks? What type of tasks can be completed in the marketplace? How much does it cost? How fast can I get results back? How big is the AMT market place?

Demographics Countries: 46.80% US, India: 34%, Misc: 19.2% (from 66 different countries) 1

Demographics Why do you complete tasks in Mechanical Turk? Please check any of the following that applies: – [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV) – [2] I find the tasks to be fun – [3] To kill time – [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards) – [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out) – [6] I am currently unemployed, or have only a part time job 123

Demographics Why do you complete tasks in Mechanical Turk? Please check any of the following that applies: – [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV) – [2] I find the tasks to be fun – [3] To kill time – [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards) – [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out) – [6] I am currently unemployed, or have only a part time job 456

Type of tasks

Requester distribution

Price distribution

Keywords vs. Ranks

Posting vs. completion rate

Completion time