1 Human Computation Play a Game to Develop an Ontology Peyman Nasirifard p+e+y+m+a+b-b+n dot deri.org

2 Agenda Introduction CAPTCHA CAPTCHA Games with a purpose ESP game ESP game Peekaboom Peekaboom Verbosity Verbosity Possible game for developing simple ontologies Play a game Conclusion

3 Introduction Human-based computation is a technique when a computational process performs its function via outsourcing certain steps to humans.

4 Back to History Yahoo! and Gmail are not interested to enable a bot to create thousands accounts per day for sending spam They use CAPTCHA to prevent it They use CAPTCHA to prevent it plus

5 CAPTCHA Stands for “Completely Automated Public Turing test to tell Computers and Humans Apart” Luis von Ahn et al. coined the term in 2000 A Program that can tell whether a user is a human or a computer Many different techniques

6 Some Examples

7 Dog or Cat? Human : mmm… dog Computer : mmmmmmmmmmmmmmm…

8 Human Computation If we use people to break CAPTCHA, we are doing human computation In some countries, some companies hire people to break CAPTCHA and send spam In some countries, some companies hire people to break CAPTCHA and send spam Some companies cleverly use humans to break CAPTCHA and send spam Some companies cleverly use humans to break CAPTCHA and send spam How? How?

9 Clever spammers

10 Clever Spammers Type the word in the box if you want to see the next picture Free Nude Photos

11 Really?! Jan 2004: world without spam by 2006! Huge amount of investment Huge amount of investment Bill Gates receives 4 million spams per day Bill Gates receives 4 million spams per day

12 Nice Quote Luis von Ahn: Instead of hiring people and pay them to solve our problems, we can design games and people will pay us to play our games and solve our large-scale problems!

13 The ESP game Object of the game: type the same word Only thing in common is: an image Players Do not know each other (randomly paired) Do not know each other (randomly paired) Can not communicate Can not communicateAdvantages: Two different sources labels the image Two different sources labels the image enjoyable enjoyable labels all images on Google image in a short time labels all images on Google image in a short time Help to improve English! Help to improve English! There are many people that play over 20 hours a week There are many people that play over 20 hours a week

14 The ESP game Player 1 Player 2 CAR GIRL TREE WOMAN CAR Agree: CAR Get points

15 Taboo words More difficult, but more fun More difficult, but more fun CAR WOMAN

16 Single version of ESP game CAR GIRL TREE WOMAN CAR The engine records everything from previous players A single player will actually play with another player, but not at the same time

17 Cheating and Repetition Problem: Agreement on cheating Let’s label all images with “dog” Let’s label all images with “dog” Solution: At random, system gets players test images to check whether they play honestly or not If they do not play honestly, the system will let them play, but nothing will be recorded If they do not play honestly, the system will let them play, but nothing will be recorded For certainty, only labels which at least N pairs agreed upon will be stored

18 The Limitations of ESP The ESP Game can label images (and consequently tell you what’s in them), but it cannot: Find the objects being labelled Find the objects being labelled Determine the way in which the object appears – does the label “car” refer to the text “car” or an actual car in the image? Determine the way in which the object appears – does the label “car” refer to the text “car” or an actual car in the image?

19 The place of objects in an image Such information would be extremely useful for computer vision research Such information would be extremely useful for computer vision research dog man

20 The Revealer clicks on parts of the image and shows them to the Guesser. The Guesser guesses: Flower Petal Butterfly Server: Correct, Butterfly

21 Hints The label “car” is ambiguous -- this is “car” this is also “car” The hints help distinguish the manner in which the label “car” appears: this is the object “car” this is the text “car”

22 Verbosity Collect common-sense facts Water quenches thirst Water quenches thirst Sky is blue Sky is blue Lions eat meat Lions eat meat We as human know hundreds of millions common sense facts Computers do not know Computers do not know If know, potentially make them more intelligent (e.g. search better)

23 Common sense fact samples Milk It is liquid It is white it has lactose cereal is eaten with it

24 Verbosity Narrator MILK is typically near cereal is a liquid Guesser

25 Verbosity Narrator MILK Guesser is typically near cereal is a liquid MILK

26 Verbosity Narrator Object Common sense facts about the object Guesser

27 Verbosity Narrator Object Guesser Common sense facts about the object Object

28 Templates ___ is a kind of ___. Allows for hierarchical categorization. ___ is used for ___. Provides information about the purpose of a word. ___ is typically near/in/on ___ (three templates). Provide spatial data. ___ is the opposite of ___ / ___ is related to ___ (two templates). Provide data about basic relations between words. ___. In the game, this is a “wildcard” that collects related words.

29 Symmetric vs. Asymmetric Verbosity is a asymmetric game, whereas ESP game is a symmetric game. Symmetric games: constraint is number of outputs per input Asymmetric games: constraint is number of inputs that produces the same output

30 Possible game to build an ontology Several game should work together Images come from ESP game Images come from ESP game Not always: only those images are selected which have one object in it Not always: only those images are selected which have one object in it i.e. car, bike, monitor, mouse, house i.e. car, bike, monitor, mouse, house These images are input to next game which tries to catch the properties of objects These images are input to next game which tries to catch the properties of objects car has colour, car has wheels, car has manufacture, car has owner, car has building year, etc. car has colour, car has wheels, car has manufacture, car has owner, car has building year, etc.

31 Possible game to build an ontology Cardinality will be caught by templates, as soon as we have properties. Car has four wheels Car has four wheels Car has one plaque Car has one plaque These sentences will be transferred to OWL representation using a mediator. The more pairs play the game, the more complex the ontology will be

32 Contact me if you are interested to work on it

33 This page intentionally left blank This page intentionally left blank

34 Guess what! It has usually four wheels It has usually one seat It is kind of vehicle It operates with human power It operates with batteries It has a break system It is a kind of chair

35 Answer

36 Conclusion Games are enjoyable! More than 9 billion Human-hours of solitaire are played each year We may cleverly using humans to solve large-scale problems by designing interesting games Many people play word-guessing games to improve their English Go and play to promote science!

37 References [1] Verbosity: A Game for Collecting Common-Sense Facts, [2] Peekaboom: A Game for Locating Objects in Images, [3] Labeling Images with a Computer Game, [4] Games with a Purpose, [5] Wikipedia, [6] We'll End Spam Within 2 Years, [7] CAPTCHA, [8] ESP game, [9] Peekaboom game, [10] Verbosity game, [11] Presentation, [12] Presentation,

38 Game Over Game Over p+e+y+m+a+b-b+n dot deri.org p+e+y+m+a+b-b+n dot deri.org