Image Annotation and Feature Extraction Digital Forensics: Image Annotation and Feature Extraction Latifur Khan, November 2007

Outline How do we retrieve Images? Motivation Annotation Enhancement Correspondence: Models Enhancement Future Work Results Reference

How do we retrieve images? Use Google image search ! Google uses filenames, surrounding text and ignores contents of the images.

Motivation How to retrieve images/videos? CBIR is based on similarity search of visual features Doesn’t support textual queries Doesn’t capture “semantics” Automatically annotate images then retrieve based on the textual annotations. Example Annotations: Tiger, grass.

Motivation Query by CBIR Retrieved image There is a gap between perceptual issue and conceptual issue. Semantic gap: Hard to represent semantic meaning using low-level image features like color, texture and shape. It’s possible to answer query ‘Red ball’ with ‘Red Rose’. Query by CBIR Retrieved image

Motivation Most of current automatic image annotation and retrieval approaches consider Keywords Low-level image features for visual token/region/object Correspondence between keywords and visual tokens Our goal is to develop automated image annotation tecniques with better accuracy

Annotation

Annotation Major steps: Segmentation into regions Clustering to construct blob-tokens Analyze correspondence between key words and blob-tokens Auto Annotation

Annotation: Segmentation & Clustering Images Segments Blob-tokens

Annotation: Correspondence/Linking Our purpose is to find correspondence between words and blob-tokens. P(Tiger|V1), P(V2|grass)…

Auto Annotation Lion Grass Tiger … …. ??

Segmentation: Image Vocabulary Can we represent all the images with a finite set of symbols? Text documents consist of words Images consist of visual terms V123 V89 V988 V4552 V12336 V2 V765 V9887 copyright © R. Manmatha

Construction of Visual Terms Segmented images ( e.g., Blobworld, Normalized-cuts algorithm.) Cluster segments. Each cluster is a visual term/blob-token Images Segments Visterms/blobtoken V1 V2 V3 V4 V1 V5 V6 … …

Discrete Visual terms Rectangular partition works better! Partition keyframe, clusters across images. Segmentation problem can be avoided at some extent. copyright © R. Manmatha

Visual terms Or partition using a rectangular grid and cluster. Actually works better.

Grid vs Segmentation Segmentation vs Rectangular Partition. Results - Rectangular Partition better than segmentation! Model learned over many images. Segmentation over one image.

Feature Extraction & Clustering Color Texture Shape K-means clustering: To generate finite visual terms. Each cluster’s centroid represents a visual term.

Co-Occurrence Models Mori et al. 1999 w1 w2 w3 w4 V1 12 2 1 V2 32 40 13 V3 V4 65 43 Mori et al. 1999 Create the co-occurrence table using a training set of annotated images Tend to annotate with high frequency words Context is ignored Needs joint probability models P( w1 | v1 ) = 12/(12+2+0+1)=0.8 P( v3 | w2 ) = 12/(2+40+12+43)=0.12

Correspondence: Translation Model (TM) Pr(f|e) = ∑ Pr(f,a|e) a Pr(w|v) = ∑ Pr(w,a|v)

Translation Models Duygulu et al. 2002 Use classical IBM machine translation models to translate visterms into words IBM machine translation models Need a bi-lingual corpus to train the models Mary did not slap the green witch Mary no daba una botefada a la bruja verde V2 V4 V6 Maui People Dance V1 V34 V321 V21 Tiger grass sky … … … … … …

Correspondence (TM ) B B N X = W N W

Correspondence (TM ) B W Bj Wi N N

Results Dataset Corel Stock Photo CDs. 600 CDs, each of them consists of 100 images under same topic. We select 5000 images (4500 training, 500 testing). Each image has manual annotation. 374 words and 500 blobs. sun city sky mountain grizzly bear meadow water

Results Experimental Context 3,000 training objects 300 images for testing Each object is represented by a vector of 30 dimensions: color, texture, and shape

Results Each Image Object/Blob-token has 30 features: Size -- portion of the image covered by the region. Position -- coordinates of the region center of mass normalized by the image dimensions. Color -- average and standard deviation of (R,G, B), (L, a, b) over the region. Texture -- average and variance of 16 filter responses, four differences of Gaussian filters with different sigmas, and 12 oriented filters, aligned in 30-degree increments. For shape, we use six features (i.e., area, x, y, boundary, convexity, and moment of inertia).

Results Examples for automatic annotation

Results The number of segments annotated correctly among 299 testing segments for different models

Results Correspondence based on K-means--- PTK. Correspondence based on Weighted Feature Selection --- PTS. With GDR dimensionality of image object will be reduced (say from 30 to 20) and then apply K-means and so on.

Results Precision p Recall r Result of Common E measure NumCorrect NumCorrect means the number of retrieved images which contain query keyword in its original annotation NumRetrieved is the number of retrieved images NumExist is the total number of images in test set containing query keyword in annotation Result of Common E measure E=1-2/(1/p+1/r) NumCorrect NumExist NumRetrieved

Results: Precision, Recall and E Precision of retrieval for different models

Results: Precision, Recall and E-measure Recall of retrieval for different models

Results: Precision, Recall and E-measure E Measure of retrieval for different models