Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University of Massachusetts – Amherst Presenter: Carlos Diuk

Introduction The Problem: Automatically annotate and retrieve images from large collections. Retrieval example: answer query “Tigers in grass” with

Introduction Manual annotation being done in libraries. Different approaches to automatic image annotation: Co-occurence Model Translation Model Cross-media relevance model

Introduction – related work Co-occurence Model Looks at co-occurence of words with image regions created using a regular grid. Translation Model Image annotation viewed as task of translating from vocabulary of blobs to vocabulary of words.

Introduction – CMRM Cross-media relevance models (CMRM) Assume that images may be described from small vocabulary of blobs. From a training set of annotated images, learn the joint distribution of blobs and words.

Introduction – CMRM Cross-media relevance models (CMRM) Allow query expansion: Standard technique for reducing ambiguity in information retrieval. Perform initial query and expand by using terms from the top relevant documents. Example in image context: tigers more often associated with grass, water, trees than with cars or computers.

Introduction – CMRM Variations: Document based expansion PACMRM (probabilistic annotation CMRM) Blobs corresponding to each test image are used to generate words and associated probabilities. Each test generates a vector of probabilities for every word in vocabulary. FACMRM (fixed annotation-based CMRM) Use top N words from PACMRM to annotate images. Query based expansion DRCMRM (direct-retrieval CMRM) Query words used to generate a set of blob probabilities. Vector of blob probabilities compared with vector from test image using Kullback- Lieber divergence and resulting KL distance.

Discrete features in images Segmentation of images into regions yields fragile and erroneous results. Normalized-cuts are used instead (Duygulu et al): 33 features extracted from images. K (=500) clustering algorithm used to cluster regions based on features. Vocabulary of 500 blobs.

CMRM Algorithms Image I = {b 1.. b m } set of blobs Training collection of images J = {b 1.. b m ; w 1.. w n } Two problems: Given un-annotated image I, assign meaningful keywords. Given text query, retrieve images that contain objects mentioned.

CMRM Algorithms Calculating probabilities.

CMRM Algorithms Image retrieval INPUT: query Q = w 1.. w n and collection C of images OUTPUT: images described by query words. Annotation-based retrieval model (PACMRM- FACMRM) Annotate images as shown. Perform text retrieval as usual. Fixed-length annotation vs probabilistic annotation:

CMRM Algorithms Image retrieval INPUT: query Q = w 1.. w n and collection C of images OUTPUT: images described by query words. Direct retrieval model (DRCMRM) Convert query into language of blobs, instead of images into words. Estimation: Ranking:

Results Dataset Corel Stock Photo CDs (5000 images – 4000 training, 500 evaluation, 500 testing). 371 words and 500 blobs. Manual annotations. Metrics: Recall: number of correctly retrieved images divided by number of relevant images. Precision: number of correctly retrieved images divided by number of retrieved images. Comparisons Co-occurence vs Translation vs FACMRM

Results Dataset Corel Stock Photo CDs (5000 images – 4000 training, 500 evaluation, 500 testing). 371 words and 500 blobs. Manual annotations. Metrics: Recall: number of correctly retrieved images divided by number of relevant images. Precision: number of correctly retrieved images divided by number of retrieved images. Comparisons Co-occurence vs Translation vs FACMRM

Results Precision and recall for 70 one-word queries.

Results PACMRM vs DRCMRM

Some nice examples Automatically annotated as sunset, but not manually

Some nice examples Response to query “pillar” Response to query “tiger”

Some bad examples

Questions - Discussion No semantic representation (just color, texture, shape). How could we annotate a newspaper’s collection? (“Kennedy”, not just “people”) Google: cooperative annotation? Google search for “tiger”: Google search for “Kennedy”: