Human Computation & ESP Game 2008/12/19 Presenter: Lin, Sin-Yan 1

References Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems. Ahn, L. V. (2007) Human computation, paper presented at the Proceedings of the 4th international conference on Knowledge capture. Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), Chen, L.-J., Wang, B.-C., Chen, C.-Y., King, I. & Lee, J. (2008) An analytical study of puzzle selection strategies for the esp game. Report for Institute of Information Science, Academia Sinica (Taipei, Taiwan, ROC). Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the esp game. Report for Microsoft Research, Microsoft Corporation. 2

Outline Human Computation Human Computation System and Application Human Computation in E-Learning The ESP Game Mathematical Model in ESP Game System Gain Image selection Automatic playing the ESP Game Problems of ESP Game Conclusion and Comments 3

Human Computation Concepts are started at 2001: The Open Mind Initiative Some task are trivial for humans, but continue to challenge computer algorithms Treat human brains as processors in a distributed system, each can perform a small part of a massive computation But humans require some incentive to become part of a collective computation By Games (Ahn and Dabbish, 2004) (Ahn and Dabbish, 2008) Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8),

Human Computation System - Peekaboom 5 Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.

Human Computation System - PhotoShoot (1/2) 6 Cheng, K.-Y. (2006). PhotoShoot: A Web-Game for User Assisted ROI Labeling. Information Management. Taipei, Taiwan, National Taiwan University. Master.

Human Computation System - PhotoShoot (2/2) 7

Human Computation System - ImageHunter 8 Chen, L.-H. (2008). Accuracy Analysis in Productivity Games: A Case Study on Landmark Annotation. Department of Computer Science and information Engineering. Taipei, Taiwan, National Taiwan University. Master: 62.

Human Computation System - PhotoSlap 9 Chang, T.-H. (2007). Productivity Game Design and Gameplay Analysis: Play and Annotate with PhotoSlap. Department of Computer Science and Information Engineering. Taipei, Taiwan, National Taiwan University. Master: 52.

Human Computation System - reCAPTCHA 10 Proposed by Ahn (2008) Gathering the characters that optical character recognition (OCR) software has been unable to read

Human Computation in E-Learning 人力計算具有利用人類直覺的特性 動機 傳統英文閱讀學習及評量無法達成的地方 目的 用 TAGs 表達英文閱讀學習過程中最直覺的想法 方法 Human Computation and Game Design 11

Human Computation in E-Learning - why tag Free, uncontrolled vocabulary Social relations of TAGs 12 photo art design image color tagging research

Agenda Human Computation Human Computation System and Application Human Computation in E-Learning The ESP Game Mathematical Model in ESP Game System Gain Image selection Automatic playing the ESP Game Problems of ESP Game Conclusion and Comments 13

The ESP Game The first human computation system to take advantage of people’s desire to be entertained and provide useful metadata Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems Randomly selected image Input Keywords Keyword Match 14

Mathematic Model in ESP Game There are many parameters in ESP Game We can use those parameters to modeling many things that we want to know Ex. Number of players Number of labels per game Number of guesses to reach a consensus Good and bad word assumption (Human Computation) System Gain Image selection User scoring model 15

Mathematic Model in ESP Game - System Gain Modeling (1/2) The performance of human computation system The purpose of ESP Game tries to collect images’ tags Two aspect of thinks Take as many tags as possible for each image Playing the maximum number of image A metric to evaluate the system gain 16

Mathematic Model in ESP Game - System Gain Modeling (2/2) 17

Mathematic Model in ESP Game - Image Selection (1/3) Different image selection method Random Fresh-first Maximize ln(N) Optimal (OPSA) Largest possible system gain P 0 : all images that have not been played P 1 : all images that have been played at least once, but less than r times P 2 : other images 18

Mathematic Model in ESP Game - Image Selection (2/3) 19 T / N

Mathematic Model in ESP Game - Image Selection (3/3) 20 M: Total number of images in the system Wang, B.-C., C.-W. Lin, et al. (2008). An Analytical Study and Modeling of the ESP game. Taipei, Taiwan, Institute of Information Science, Academia Sinica.

Mathematic Model in ESP Game - Automatic Playing the ESP Game (1/4) Given enough instances of labeled images as training data Similar with Language model Assign probability to the next label to be added Plays the ESP Game without looking at the image Limitation : at least one “off-limits” in the image 21 Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

Mathematic Model in ESP Game - Language Modeling (1/2) Model the sequence of word usage of a certain language Given a Language Model: M P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this) 22

Mathematic Model in ESP Game - Language Modeling (2/2) P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this) Markov assumption: To give reasonable predictions, only the prior local context (the last few words) affects the next word n-gram word model Bi-gram: P(I love this game) = P(I) P(love|I) P(this|love) P(game|this) 23

Mathematic Model in ESP Game - Automatic Playing the ESP Game (2/4) Given enough instances of labeled images as training data At least one “off-limits” in the image t : next label T : off-limits set P(“ t is next label” | “set T already present”) 24 Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

Mathematic Model in ESP Game - Automatic Playing the ESP Game (3/4) P(“ t is next label” | “set T already present”) Conditional independence assumption The problem of zero probability 25 Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

Mathematic Model in ESP Game - Automatic Playing the ESP Game (4/4) The problem of zero probability 26 Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

Mathematic Model in ESP Game - Others User scoring model More informative tags 27

Problems of ESP Game There is a lot of redundancy in the tag sets Even when tags are not exactly synonyms, they are often “to be expected ” given the other tags There is a tendency to match on colors People tend to add more generic labels Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 28

Conclusion and Comments The first thing of Human Computing research is finding a clear PURPOSE How your game or method to achieve the purpose 29