1. Playing GWAP with strategies - using ESP as an example Wen-Yuan Zhu CSIE, NTNU

2. Motivation “Games with a purpose” (GWAP) is an innovative concept in computer science There are a lot of GWAP systems has been created Research on enhancing GWAP systems is scarce

3. State of the art “Human Computation” represents a new paradigm of applications – Some problems solved by human, not computer “Games with a purpose” (GWAP) – created by Dr. Luis von Ahn (CMU) – the most popular application

4. Our Contribution We propose an evaluation metric for GWAP systems We study the inner properties of the ESP game using analysis We propose an “Optimal Puzzle Selection Algorithm” (OPSA)

5. Our Contribution (2) We implement a quasi ESP game, ESP Lite, to demonstrate our proposed algorithms We confirm GWAP systems are more efficient if they are designed and played with strategies

6. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

7. Human Computation There is a lot of things that human can easy do that computers can not yet do – Speech recognition – Natural language understanding – Computer graphics

8. Games with a Purpose It combine computation with game People spend a lot of time playing games It makes Human Computation more efficient There are a lot of GWAP systems has been created (e.g. ESP game and Google Image Labeler)

9. What is the ESP game? AliceBob shoeflower rocks agreement on “flower”

10. What is the ESP game? (2) it is efficient – 200,000+ players have contributed 50+ million labels – each player plays for a total of 91 minutes – 233 labels/human/hour (i.e. one label every 15 seconds) Google bought a license to create its own version of the game in 2006

11. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

12. Objectives 1.Design a metric to measure the performing of the ESP game 2.Design proper strategies to improve the ESP game 3.Validate the proposed strategies in real world systems

13. Observations The ESP game has two goals 1.Quantity : the system prefers to maximize the number of images which have been played 2.Quality : the system prefers to take as many labels as possible for each image There is a trade-off between the two goals

14. System metric average scores per labeled image # of labeled images # of labels per labeled image # of labels

15. Modeling(2)

16. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

17. Puzzle Selection Algorithms Optimal Puzzle Selection Algorithm (OPSA) – select an image based on our analysis Random Puzzle Selection Algorithm (RPSA) – select a image by random Fresh-first Puzzle Selection Algorithm (FPSA) – select a image that has been played least frequently

18. OPSA The idea is there are optimal r labels per labeled image in system We group images into 3 sets – contains all the images that have not been played – contains all the images that have been played at least once, but less than r rounds –

19. OPSA (2)

20. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

21. Simulation Setup – There are 100,000 images – Running 20 times simulation Observation – T vs. r → OPSA – T vs. System gain → 3 strategies Discussion

22. T vs. r

23. T vs. System Gain

24. Discussion OPSA is superior to RPSA & FPSA in the simulation A systematically & thorough study to verify the purposed strategies in real systems is highly desirable To this end, we decide to implement the ESP Lite system

25. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

26. ESP Lite We implement ESP using Flash/Java It is a mimic ESP game It implemented three playing strategies – RPSA, FPSA, OPSA

27. Flow chart the strategy selection process gives priority to the strategy that has been used least in terms of the # of rounds played previously

28. Client interface

29. Client interface (2)

30. Score System The score system is used to measure the quality of the agreed words The quality of each the agreed word should depends on its popularity – high frequency → low quality → low score – low frequency → high quality → high score

31. Score System (2) i-th level of the w_i n words in the score table frequency of the word score of w_i reserved for agreed words which are not in score table

32. Score System (3) Apply the Porter Stemming Algorithm to remove common morphological and inflectional endings of English words – Prevent words with the same root, but receiving different scores (e.g. determinant and determine) – Reduce the plural form to the singular form (e.g. experiments and experiment)

33. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

34. 100,000 images from ESP dataset ESP dataset – 100,000 images – Average 15 labels per image collected from the ESP game Score system – – Range of score (10 levels) Experiment Setting

35. Experiment Setting (2) Score system (cont.) – The 5,000 most frequency words from Brown Corpus (a general corpus in the field of corpus linguistics) – Processed by Porter Stemming Algorithm – 3,476 words in score table

36. Experiment http://nrl.iis.sinica.edu.tw/GWAP/ESPLite/ From 2009/3/9 to 2009/4/9 3,103 games 9,376 labeled pictures 12,312 agreements

37. Score Statistics

38. Behavior of OPSA

39. What happened in each round? Behavior of OPSA (2)

40. The operation of OPSA depend on r

41. Observe players’ behavior Behavior of OPSA (3)

42. Performance

43. Outline Introduction Analysis Propose our algorithm Evaluation & Simulation Implementation Result Conclusion & Future work

44. Conclusion In this thesis – We propose a metric to evaluate the performance of GWAP – We play GWAP systems with strategies and make GWAP systems more efficient

45. Conclusion (2) This is the first GWAP study that implements and evaluates an analytical model on real- world GWAP systems our experiment results confirm that GWAP systems are more efficient if they are designed and played with strategies

46. Future work Thinking about the time factor in analysis Thinking about adaptive r for each image Constructing the framework of GWAP systems Developing the GWAP portal for fast accessing more GWAP systems Developing the GWAP systems toward mobile environment

47. Thank You!