1. Center for Computational Analysis of Social and Organizational Systems http://www.casos.cs.cmu.edu/ Labeling images with a computer game By von Ahn & Dabbish

2. Motivation ‘Our goal is ambitious: to label the majority of images on the World Wide Web.’ Machine learning to identify and label image components is generally ineffective. Peter Landwehr2

3. Past Work Peter Landwehr3 Lempel & Soffer, 2002 Duygulu et al., 2002

4. Peter Landwehr4

5. Implementation notes Image reintroduction possibility Age restriction possibility Play alone possibility Taboo threshold (X=1) Image completion Randomized partners 350K images from random.bounceme.net 73K word dictionary ‘[T]he game doesn’t ask the players to describe the image: all they are told is that they have to “think like each other” and type the same string.’ Peter Landwehr5

6. August 9 – December 10, 2003 13,360 Players 1,271,451 labels for 293,760 images 80% of players returned at least once, 33 spent 50 hours playing Mean = 3.89 labels/image/minute of play, std. dev = 0.69. Peter Landwehr6

7. 7 Validation (1) cardog manwoman stampWitherspoon smilingAlias cartoongreen

8. 1.‘Please type the six individual words that you feel best describe the contents of this image. Type one word per line below; words should be less than 13 character’ –For all of the images… at least 5 (83%) of the 6 labels produced by the game were entered by at least one participant. the three most common words entered by participants were contained among the labels produced by the game. Peter Landwehr8 15 participants aged 20-25, 20 images with >5 labels Validation (2)

9. 1.‘How many of the words above would you use in describing this image to someone who couldn’t see it?’ –Mean = 5.105, std. dev = 1.3087 (85% useful) 2.‘How many of the words have nothing to do with the image (i.e., you don't understand why they are listed with this image)?’ –Mean = 0.105, std. dev = 0.2529 (1.7% useless) Peter Landwehr9 15 participants aged 20-25, 20 images with >5 labels Validation (3)

10. Broad implications ‘At this rate, 5,000 people playing the ESP game 24 hours a day would label all images on Google (425,000,000 images) [with one tag] in 31 days.’ ‘…[O]ur main contribution stems from the way in which we attack the labeling problem. …[W]e have shown that it’s conceivable that a large-scale problem can be solved with a method that uses people playing on the Web. We’ve turned tedious work into something people want to do.’ Peter Landwehr10

11. Peter Landwehr11 ‘The ESP game can be used, with minor modifications, to label sound or video clips (i.e., there is nothing inherent about images).’

12. Peter Landwehr12 ‘Other problems that could be solved by having people play games include categorizing web pages into topics and monitoring security cameras.’

13. More recent work Peter Landwehr13 Walsh & Golbeck, 2010

14. Recent Work McGonigal, 2003 Law, von Ahn, Dannenberg, and Crawford, 2007 Hacker and von Ahn, 2009 Dong and Fu, 2010 Walsh and Golbeck, 2010 Peter Landwehr14

15. Center for Computational Analysis of Social and Organizational Systems http://www.casos.cs.cmu.edu/ Questions?