Color-Attributes-Related Image Retrieval Student: Kylie Gorman Mentor: Yang Zhang

Problem and Solution Content based image retrieval is a common problem in computer vision Object-related image retrieval is a popular area related to this issue Attributed-related image retrieval is a possible solution Enable a person to retrieve an image based on attributes of an object Some people have tried to use color as a starting point, but this is still a very novel concept

Data Sets Learn colors from real-world images Train data: Google data set ▫11 colors with 100 images per color Test data: EBay data set ▫11 colors with 12 images per color ▫Corresponding binary image for each image

Train Data Steps Calculate feature matrix based on Color Moments ▫Calculate every box rather than every pixel Concatenate feature matrices Calculate PCA (Principal Component Analysis) Calculate GMM (Gaussian Mixture Model) based on PCA results Multiply individual feature matrices by coefficient matrix Use GMM results to calculate Fisher Vectors Train 11 SVM’s

Test Data Steps Calculate feature matrix of each image, isolating the object first using binary images Use PCA and GMM results from training data to calculate fisher vectors Apply Fisher Vector to each individual result to obtain vectors that are the same size Classify eBay images using 11 SVM’s from training data Calculate Precision

Steps

CIELAB Results Average Precision: ~42%

HSV Images Average Precision: ~45%

RGB Images Average Precision: ~50%

New Data Sets Birds 200 ▫200 species/categories with 11,788 images total Flowers 102 ▫102 categories with images per category ▫8189 images total Cartoon ▫590 images total

Flowers 102 and Birds 200 Part One ▫Get Feature Matrices with Color Moments ▫Calculate PCA and GMM of training data: 1,020 images Part Two ▫Get Feature Matrices with Dense SIFT ▫Calculate PCA and GMM of training data: 100 images Part Three ▫Use new Color Descriptor

Future Goals Compare our color moment plus Dense SIFT against new color descriptor and Dense SIFT ▫If no improvement, determine why Incorporate object detection and image retrieval