Playing GWAP with strategies - using ESP as an example Wen-Yuan Zhu CSIE, NTNU

Motivation “Games with a purpose” (GWAP) is an innovative concept in computer science There are a lot of GWAP systems has been created Research on enhancing GWAP systems is scarce

State of the art “Human Computation” represents a new paradigm of applications – Some problems solved by human, not computer “Games with a purpose” (GWAP) – created by Dr. Luis von Ahn (CMU) – the most popular application

Our Contribution We propose an evaluation metric for GWAP systems We study the inner properties of the ESP game using analysis We propose an “Optimal Puzzle Selection Algorithm” (OPSA)

Our Contribution (2) We implement a quasi ESP game, ESP Lite, to demonstrate our proposed algorithms We confirm GWAP systems are more efficient if they are designed and played with strategies

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

Human Computation There is a lot of things that human can easy do that computers can not yet do – Speech recognition – Natural language understanding – Computer graphics

Games with a Purpose It combine computation with game People spend a lot of time playing games It makes Human Computation more efficient There are a lot of GWAP systems has been created (e.g. ESP game and Google Image Labeler)

What is the ESP game? AliceBob shoeflower rocks agreement on “flower”

What is the ESP game? (2) it is efficient – 200,000+ players have contributed 50+ million labels – each player plays for a total of 91 minutes – 233 labels/human/hour (i.e. one label every 15 seconds) Google bought a license to create its own version of the game in 2006

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

Objectives 1.Design a metric to measure the performing of the ESP game 2.Design proper strategies to improve the ESP game 3.Validate the proposed strategies in real world systems

Observations The ESP game has two goals 1.Quantity : the system prefers to maximize the number of images which have been played 2.Quality : the system prefers to take as many labels as possible for each image There is a trade-off between the two goals

System metric average scores per labeled image # of labeled images # of labels per labeled image # of labels

Modeling(2)

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

Puzzle Selection Algorithms Optimal Puzzle Selection Algorithm (OPSA) – select an image based on our analysis Random Puzzle Selection Algorithm (RPSA) – select a image by random Fresh-first Puzzle Selection Algorithm (FPSA) – select a image that has been played least frequently

OPSA The idea is there are optimal r labels per labeled image in system We group images into 3 sets – contains all the images that have not been played – contains all the images that have been played at least once, but less than r rounds –

OPSA (2)

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

Simulation Setup – There are 100,000 images – Running 20 times simulation Observation – T vs. r → OPSA – T vs. System gain → 3 strategies Discussion

T vs. r

T vs. System Gain

Discussion OPSA is superior to RPSA & FPSA in the simulation A systematically & thorough study to verify the purposed strategies in real systems is highly desirable To this end, we decide to implement the ESP Lite system

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

ESP Lite We implement ESP using Flash/Java It is a mimic ESP game It implemented three playing strategies – RPSA, FPSA, OPSA

Flow chart the strategy selection process gives priority to the strategy that has been used least in terms of the # of rounds played previously

Client interface

Client interface (2)

Score System The score system is used to measure the quality of the agreed words The quality of each the agreed word should depends on its popularity – high frequency → low quality → low score – low frequency → high quality → high score

Score System (2) i-th level of the w_i n words in the score table frequency of the word score of w_i reserved for agreed words which are not in score table

Score System (3) Apply the Porter Stemming Algorithm to remove common morphological and inflectional endings of English words – Prevent words with the same root, but receiving different scores (e.g. determinant and determine) – Reduce the plural form to the singular form (e.g. experiments and experiment)

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

100,000 images from ESP dataset ESP dataset – 100,000 images – Average 15 labels per image collected from the ESP game Score system – – Range of score (10 levels) Experiment Setting

Experiment Setting (2) Score system (cont.) – The 5,000 most frequency words from Brown Corpus (a general corpus in the field of corpus linguistics) – Processed by Porter Stemming Algorithm – 3,476 words in score table

Experiment From 2009/3/9 to 2009/4/9 3,103 games 9,376 labeled pictures 12,312 agreements

Score Statistics

Behavior of OPSA

What happened in each round? Behavior of OPSA (2)

The operation of OPSA depend on r

Observe players’ behavior Behavior of OPSA (3)

Performance

Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

Conclusion In this thesis – We propose a metric to evaluate the performance of GWAP – We play GWAP systems with strategies and make GWAP systems more efficient

Conclusion (2) This is the first GWAP study that implements and evaluates an analytical model on real- world GWAP systems our experiment results confirm that GWAP systems are more efficient if they are designed and played with strategies

Future work Thinking about the time factor in analysis Thinking about adaptive r for each image Constructing the framework of GWAP systems Developing the GWAP portal for fast accessing more GWAP systems Developing the GWAP systems toward mobile environment

Thank You!