2 IndexingRankingClustering …… Recommendation Annotation Multimedia Information Retrieval

3 Image Similarity/ Distance Concept Similarity/ Distance Annotation Indexing Ranking Clustering …… Recommendation

4 Image Similarity/ Distance Concept Similarity/ Distance Image Similarity/Distance

5 Numerous efforts have been made. Concept Similarity/ Distance Concept Similarity/Distance

Image Similarity/Distance 6 Concept Similarity/Distance Olympic Numerous efforts have been made. SportsCat TigerPaw More and more used, but not well studied.

7 WordNet Distance Google Distance Tag Concurrence Distance

8 Built by human experts, so close to human perception Coverage is limited and difficult to extend

9 Easy to get and huge coverage Only reflects concurrency in textual documents. Not really concept distance (semantic relationship)

10

11 Images are taken into account a)Tags are sparse so visual concurrency is not well reflected b)Training data is difficult to get similarity matrix: 500 tags similarity matrix: 50 tags

12

13 Synonymy different words but the same meaning table tennis ping- pong — Visually Similar similar things or things of same type horsedonkey — Meronymy part and the whole carwheel — Concurrency exist at the same scene/place airplaneairport —

14 Image tag concurrence distance Image tag concurrence distance implicitly uses image information, but tags are too sparse Google distance Google distance’s coverage is very high, but it is for text domain Concept Distance WordNet distance WordNet distance is good, but coverage is too low Mine from ontology Mine from text documents Mine from image tags

15 Can we mine concept distance from image content?

16 To mine concept distance from a large tagged image collection based on image content

17 Concept A: Airplane Concept B: Airport Concept Model A Concept Model B Flickr Distance (A, B)

18 Flickr Distance is able to cover the four different semantic relationships Synonymy, Visually Similar, Meronymy, and Concurrency

19

20 SVM, Boosting, … Discriminative Generative Global Feature Local Feature w/o Spatial Relation w/ Spatial Relation Bag-of-Words (pLSA, LDA), … 2D HMM, MRF, … Concept Models

21 SVM, Boosting, … Discriminative Generative Global Feature Local Feature w/o Spatial Relation w/ Spatial Relation Bag-of-Words, … 2D HMM, MRF, … Concept Models VLM – Visual Language Model Spatial-relation sensitive Efficient Efficient Can handle object variations Can handle object variations

22 Iamtalkingaboutstatisticallanguagemodel. Unigram ModelBigram ModelTrigram Model

23 Unigram ModelBigram ModelTrigram Model Image  Patch Patch  Gradient Texture Histogram Hashing  Visual Word Visual Word Generation

24

25

26

27

28

29 Concept A: Airplane Concept B: Airport Concept Model A Concept Model B Flickr Distance (A, B) Tag search in Flickr Jensen-ShannonDivergence LT-VLM

30

31

32

33

34 Normalized Google Distance Tag Concurrence Distance Flickr Distance Group1Group2Group3 Group 1 Group2Group3Group1Group2Group3 bears horses moon space bowling dolphin donkey Saturn sharks snake softball spiders turtle Venus whale wolf baseball basketball football golf soccer tennis volleyball moon space Venus whale baseball donkey softball wolf basketball bears bowling dolphin football golf horses Saturn sharks soccer spiders tennis turtle volleyball moon Saturn space Venus bears dolphin donkey golf horses sharks spiders tennis whale wolf baseball basketball football snake soccer bowling softball volleyball

35 The number of correctly annotated keywords at the first N words

36

37

38 If we find similar patterns in the images associated with different concepts, the corresponding concept relationships can be discovered.

39

40

41

42

43

44 International Network for Social Network Analysis

45

46

47 Flickr Distance is able to cover the four different semantic relationships Synonymy, Visually Similar, Meronymy, and Concurrency

48

Image  Patch Patch  GradientTexture Histogram Hashing  Visual Word

50

51 compared with ground-truth distance pair NGD Ground- Truth

52

53