Semi-Automatic Image Annotation Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski and Brent Field Microsoft Research

Outline Introduction: What, Why, and How Our Approach: Semi-Automatic Processes and Algorithms Automated Performance Evaluation Usability Studies Concluding Remarks

What it is and Why Image Annotation is a process of labeling images with keywords to describe semantic content For image indexing and retrieval in image databases Annotated images can be found more easily using keyword-based search

Image Annotation Approaches Totally Manual Labeling (Gong et al., 1994) Enter keywords when image is loaded/registered/browsed Accurate but labor-intensive, tedious, and subjective Direct Manipulation Annotation (Shneiderman and Kang 2000) Drag and drop keywords (from a predefined list ) onto image Still manual, also limited to predefined keywords (can’t be many) Automatic Approaches: Efficient but less reliable and not always applicable compared to human annotation---how to grab this when no text context? By Image Understanding/Recognition (Ono et al. 1996) By Associating with environmental text (Shen et al. 2000; Srihari et al. 2000; Lieberman 2000)

Our Proposed Approach Semi-Automatic Approach User provides initial query and relevance feed back. Feedback used to “semi-automatically” annotate images Trade-off between manual and automatic Achieve both accuracy and efficiency Increase productivity Employ Content-Based Image Retrieval (CBIR), text matching, and Relevance Feedback (RF)

CBIR and RF Process and Framework

Algorithms for Matching Visual Similarity Measurement Features: color histogram/moments/coherence, Tamura coarseness, pyramid wavelet texture, etc Distance model: Euclidean distance Semantic (Keywords) Similarity Measurement Features: keyword vectors, TF*IDF Metrics: dot product and cosine normalization Overall similarity: weighted average of the above two

Algorithms to Refine Search Image Relevance Feedback Algorithms There are many algorithms can be used Cox et al. (1996) Rui and Huang (2000) Vasconcelos and Lippman (1999) Lu et al is employed in MiAlbum for text and images Modified Rocchio’s Formula Uses both semantics (keywords) and image-based features during relevance feedback

Semi-Automatic Annotation During Relevance Feedback In each keyword-query search cycle When positive and negative examples provided, Increase the weight of the keyword for all positive examples Decrease the weight of the keyword for all negative examples Relevance feedback algorithm refines and puts more relevant images in top ranks for further selection as positive examples Repeat the feedback process

Possible Future Automatic Annotation When a new image is added… Find top N similar images using image metrics Most frequent keywords among annotations of these top N similar images are potential annotations, and could be automatically added with low weight or presented to user as potential annotations TBD--Need to be confirmed in further RF process

Automated Performance Evaluation Test Ground Truth Database 12,200 images in 122 categories from Corel DB Category name is ground truth annotation Automatic Experimental Process Use category name as query feature for image retrieval Among first 100 retrieved images, those belonging to this category are used as positive feedback examples others as negative Performance Metrics Retrieval accuracy and annotation coverage

Image retrieval accuracy and annotation coverage

Usability Studies Objectives 2 studies examined overall usability of MiAlbum The usability of the semi-automatic annotation strategy Tasks Import pictures, annotate pictures, find pictures, and use relevance feedback Questionnaires including but not limited to Overall ease of entering annotations for images Impact of annotation on ease of searching for images Satisfaction of search refinement & relevance feedback

Questionnaire Results Overall ease of entering annotations: 5.6/7.0 Ease to search annotated photos: 6.3/7.0 Intuitiveness of refining search: 4.1/7.0 Other Comments Positive on “semi-automatic”: (1) When using the up and down hands the software automatically annotated the photos chosen. (2) The ability to rate pictures on like/dislike and have the software go from there. Negative: difficulties in understanding the feedback process and how the matching algorithm operated.

Concluding Remarks A Semi-automatic Annotation Strategy Employing Available image retrieval algorithms and Relevance feedback Automatic Performance Evaluation Efficient compared to manual annotation? More accurate than automatic annotation Usability Studies Preliminary usability results are promising Need to improve the discoverability of the feedback process and the underlying matching algorithm