C OLOR -A TTRIBUTES -R ELATED I MAGE R ETRIEVAL W EEK 4 Student: Kylie Gorman Mentor: Yang Zhang

GMM AND F ISHER V ECTOR C ODE

S TEP O NE : C ONCATENATE THE F EATURE M ATRICES OF E ACH I MAGE

S TEP T WO : A PPLY GMM F UNCTION Generate mean, covariance, and prior mode probabilities Mean

Covariance Prior Mode Probabilities

S TEP T HREE : C REATE F ISHER V ECTORS encoding = vl_fisher(new', means, covariances, priors);

R EPEAT P ROCESS

F INAL S TEPS Calculate feature matrix of each image, isolating the object first Concatenate matrices Apply PCA function to preprocess data Multiply each individual feature matrix by result Concatenate output into 1 matrix Apply GMM function and obtain mean, covariance, and prior mode probabilities Apply Fisher Vector to each individual result to obtain vectors that are the same size Use those fisher vectors for 11 SVM’s (one for each color)

C OMPLETE S TEPS Using Ebay Data (omitting binary images) Use all Google Data (from 30 to 100 images per color) Increase cluster size in GMM from 10 to 128

SVM

L INEAR SVM First tried it with libsvm code MATLAB Function: svmtrain (Training, Group) Training: Data to be processed (transpose matrix) Group: Specifies +1 or -1 data Use SVM for each color (black, blue, brown, green, grey, orange, pink, purple, red, white, yellow) Changed to fitcsvm(X,Y)

SVM T RAIN O UTPUT

F ITCSVM O UTPUT

C LASSIFY D ATA MATLAB Function: svmclassify(SVMStruct,Sample) Use SVMStruct from svmtrain (from each color) Sample: Concatenated Ebay Fisher Vectors Changed to predict(SVMModel, X) SVMModel from fitcsvm

SVM C LASSIFY O UTPUT Column vector with the same number of rows as Sample. Each entry (row) in Group represents the class of the corresponding row of Sample.

P REDICT O UTPUT Returns Label and Score

C URRENT P ROGRESS

C ALCULATE P RECISION Calculate 12 highest scores for each color, using first column only Determine if each score is a correct match by checking indices Calculate each color’s precision

F UTURE G OALS Fix Binary function Try process with new data set Data set available: Fahad Shahbaz Khan, Rao Muhammad Anwer, Joost van de Weijer, Andrew D. Bagdanov, Maria Vanrell, Antonio M. Lopez Image retrieval test Object Detection