WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota Technological Institute Chicago (TTIC)

Approach Dataset At each iteration, the top K < N ranked images

Binary Relevance Feedback

Current reference set Score function

Relative Attribute Feedback

Learning Relative Attributes Mechanical Turk (MTurk)

Updating the scoring function from feedback Feedback form ▫“ What I want is more/less/similarly m than image Itr ”

Three cases

Score function of Relative Attribute Feedback Take the intersection of all F feedback

Hybrid Feedback Approach

x denotes the Cartesian product

Datasets Shoes ▫14,658 shoe images from like.com ▫attributes—‘pointy-at-the-front’, ‘open’, ‘bright-in- color’, ‘covered-with-ornaments’, ‘shiny’, ‘highat-the- heel’, ‘long-on-the-leg’, ‘formal’, ‘sporty’, and ‘feminine’ PubFig ▫the Public Figures dataset of human faces ▫772 images from 8 people and 11 attributes OSR ▫the Outdoor Scene Recognition dataset of natural scenes ▫2,688 images from 8 categories and 6 attributes

Datasets Training set ▫750 triplets of images (i, j, k) from each dataset Score correlation ▫Normalized Discounted Cumulative Gain at top K ▫K = 50

Iteration experiments on the three datasets

Amount of feedback

Ranking accuracy with first feedback Attribute meaning ▫“amount of perspective” on a scene is less intuitive than “shininess” on shoes