1. HANOLISTIC: A HIERARCHICAL AUTOMATIC IMAGE ANNOTATION SYSTEM USING HOLISTIC APPROACH Özge Öztimur Karadağ & Fatoş T. Yarman Vural Department of Computer Engineering Middle East Technical University, Ankara, Turkey

2. A UTOMATIC I MAGE A NNOTATION Image Annotation : Assigning keywords to digital images. Labor intensive Time consuming Need a system that automatically annotates images.

3. I MAGE A NNOTATION L ITERATURE Annotation problem has become popular since 1990s. Related to CBIR. CBIR processes visual information Annotation processes visual and semantic information Relate visual content information to semantic context information.

4. P ROBLEMS A BOUT A UTOMATIC I MAGE A NNOTATION Human subjectivity Semantic Gap Availability of datasets

5. I MAGE A NNOTATION A PPROACHES IN THE L ITERATURE … Segmental Approaches 1. Segment or partition the image into regions 2. Extract features from the regions 3. Quantize features into blobs 4. Model the relation between the image regions and annotation words Holistic Approaches Features are extracted from the whole image.

6. S EGMENTAL A NNOTATION : SKY, BOAT, SEA, TREE

7. H OLISTIC ANNOTATION : T IGER, TREE, SNOW

8. T HE P ROPOSED S YSTEM : HANOLISTIC Introducing semantic information as supervision. each word is considered as a class label, an image belongs to one or more classes Holistic Approach: multiple visual features are extracted from the several whole image. Multiple feature spaces

9. D ESCRIPTION OF AN I MAGE Content Description by Visual Features of Mpeg-7 Color Layout Color Structure Scalable Color Homogenous Texture Edge Histogram Context Description by Semantic Words Annotation words

10. S YSTEM A RCHITECTURE OF HANOLISTIC Level-0 : consists of level-0 annotators, one annotator for each visual description space. Meta-level : consists of a meta-annotator

11. L EVEL -0 A NNOTATOR refers to the features of the i th image in the j th description space refers to the membership value of the l th word for the i th image in the j th description space.

12. M ETA -L EVEL The results of level-0 annotators are aggregated. is a vector, referring to the final word membership values for the i th image.

13. E XPERIMENTAL STUDIES Realization of HANOLISTIC Instance based realization of Level-0 Eager realization of Level-0 Realization of Meta-level Performance criteria Results

14. E XPERIMENTAL S ETUP Data set: A subset of Corel Stock Photo Collection, consisting of 5000 images. Training set: 4500 images (500 images for validation) Testing set: 500 images Each image is annotated with 1-5 many words.

15. I NSTANCE BASED R EALIZATION OF L EVEL - 0 A NNOTATOR BY F UZZY - KNN Level-0 annotators are realized by fuzzy-knn. For each description space; k nearest neighbors of the image is determined. Word membership values are estimated considering the neighbors’ words and their distance from the image. High membership values are assigned to words that appear in close neighborhood.

16. E AGER R EALIZATION OF L EVEL -0 BY A NN For a given image Ii, ANN receives visual description of the image as input and semantic annotation words of the image as target. Each ANN is trained with backpropagation and a randomly selected set of images is used for validation to determine when to stop training. K- fold cross validation is applied.

17. R EALIZATION OF M ETA -L EVEL BY M AJORITY V OTING Adds the membership values returned by level-0 annotators using the formula where, P i,j is a vector containing the word membership values returned from the j th level-0 annotator. For each word select the maximum of the five word membership values estimated by the level-0 annotators.

18. P ERFORMANCE C RITERIA Precision Recall F-score

19. P ERFORMANCE OF L EVEL -0 A NNOTATORS Performance of Level-0 annotators with fuzzy- knn

20. P ERFORMANCE OF HANOLISTIC Comparison of HANOLISTIC with other systems in the literature:

21. A NNOTATION E XAMPLES

22. A NNOTATION E XAMPLES …

23. C ONCLUSION We proposed a hierarchical automatic image annotation system using holistic approach. We tested the system both with an instance based and an eager method. We realized that the instance based methods are more promising in the considered problem domain.

24. C ONCLUSION … The power of the proposed system comes from the following main principles: Simplicity Fuzziness Simultaneous processing of content and context information Holistic view of image through different perspectives

25. F UTURE W ORK Conduct experiments on other descriptors Test other algorithms at level-0 conforming to the principle of least commitment Apply holistic approach followed by a segmentation process, for annotation or intelligent segmentation.

26. Thank you Questions and Comments

27. R EFERENCES Duygulu, Barnard, Freitas and Foryth ‘Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary’ in ECCV’02: Proceedings of the 7th European Conference on Computer Vision,2002 Jeon, Lavrenko, Manmatha ‘Automatic image annotation and retrieval using cross-media relevance models’, in SIGIR’03 Monay and Perez ‘Plsa-based image auto-annotation: constraining the latent space’ in MULTIMEDIA’04 Akbaş and Vural ‘Automatic image annotation by ensemble of visual descriptors’ in CVPR’07 Feng, Manmatha and Lavrenko ‘Multiple bernoulli relevance models for image and video annotation’. CVPR’02. Tang and Lewis, ‘Image auto-annotation using ‘easy’ and ‘more challenging’ training sets’, 7th International Workshop on Image Analysis for Multimedia Interactive Services, 2006