Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz

Acknowledgements Talks by Moshe Bar; writings of Shimon Edelman Murphy Big Book of Concepts Weinberger Everything is Miscellaneous Many great discussions with many colleagues, especially Tomasz Malisiewicz, James Hays, and Derek Hoiem

Unreasonable Effectiveness of Data Parts of our world can be explained by elegant mathematics: –physics, chemistry, astronomy, etc. But much cannot: –psychology, genetics, economics, etc. Enter: The Magic of Big Data –Great advances in several fields: e.g. speech recognition, machine translation, search engines [Halevy, Norvig, Pereira 2009]

Categorization vs. The Data 4 Philosophy and Psychology Language Arts and recreation Literature Technology Religion Weinberger, Everything is Miscellaneous

categorization is losing… vs.

Whats in a name?

…That which we call a rose By any other name would smell as sweet. chair category (PASCAL VOC) train category (PASCAL VOC)

Why Categorize? 1.Knowledge Transfer 2.Communication Tiger cat dog Leopard

Classical View of Categories Dates back to Plato & Aristotle 1. Categories are defined by a list of properties shared by all elements in a category 2. Category membership is binary 3. Every member in the category is equal

Problems with Classical View Humans dont do this! – People dont rely on abstract definitions / lists of shared properties (Wittgenstein 1953, Rosch 1973) e.g. define the properties shared by all games e.g. are curtains furniture? Are olives fruit? – Typicality e.g. Chicken -> bird, but bird -> eagle, pigeon, etc. – Language-dependent e.g. Women, Fire, and Dangerous Things category is Australian aboriginal language (Lakoff 1987) – Doesnt work even in human-defined domains e.g. Is Pluto a planet?

Visual Problems with Categories A lot of categories are functional Categories are 3D, but images are 2D World is too varied Chair car train

Typical HOG car detector Felzenszwalb et al, PASCAL 2007

Why not? + submitted to CVPR 2011

Semantic -> Visual Categories Aspect ratio splits partsPoselets All use a priori domain information

Fundamental Problem with Categorization Making decisions too early! Like Amazinn.com, can we just categorize at run-time, once we know the task!

On-the-fly Categorization? 1.Knowledge Transfer 2.Communication

Association instead of categorization Ask not what is this?, ask what is this like – Moshe Bar Exemplar Theory (Medin & Schaffer 1978, Nosofsky 1986, Krushke 1992) –categories represented in terms of remembered objects (exemplars) –Similarity is measured between input and all exemplars –think non-parametric density estimation

What is this? Car Road Building Input Image He 2004, Tu 2004, Shotton 2006, Galleguillos 2008, Fei-Fei 2009, Gould 2009, etc.

What is this like? Malisiewicz & Efros, CVPR08

20 What is the ultimate goal? Understanding / Parsing Images A what is it like? machine

21 Our Contributions Posing Recognition as Association –Use large number of object exemplars 21 Learning Object Similarity –Different distance function per exemplar Recognition-Based Object Segmentation –Use multiple segmentation approach

Visual Associations How are objects similar? Shape Color

Distance Similarity Functions Positive Linear Combinations of Elementary Distances Computed Over 14 Features Building e Distance Function Building e

Learning Object Similarity Learn a different distance function for each exemplar in training set Formulation is similar to Frome et al [1,2] [1] Andrea Frome, Yoram Singer, Jitendra Malik. "Image Retrieval and Recognition Using Local Distance Functions." In NIPS, [2] Andrea Frome, Yoram Singer, Fei Sha, Jitendra Malik. "Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification." In ICCV, 2007.

25 Non-parametric density estimation Color Dimension Shape Dimension Class 1 Class 2 Class 3

26 Non-parametric density estimation Color Dimension Shape Dimension Class 1 Class 2 Class 3

27 Non-parametric density estimation Color Dimension Shape Dimension Class 1 Class 2 Class 3

Learning Distance Functions 28 Dshape Dcolor Focal Exemplar similar side DecisionBoundary dissimilar side Dont Care

Visualizing Distance Functions (Training Set) Query Top Neighbors with Tex-Hist Dist Top Neighbors with Learned Dist

Visualizing Distance Functions (Training Set)

Labels Crossing Boundary

Object Segmentation via Recognition Generate Multiple Segmentations (Hoiem 2005, Russell 2006, Malisiewicz 2007) – Mean-Shift and Normalized Cuts – Use pairs and triplets of adjacent segments – Generate about 10,000 segments per image Enhance training with bad segments Apply learned distance functions to bottom- up segments

33 Example Associations Bottom-Up Segments

34 Quantitative Evaluation 34 Object hypothesis is correct if labels match and OS >.5 *We do not penalize for multiple correct overlapping associations OS(A,B) = Overlap Score = intersection(A,B) / union(A,B)

35 Towards Image Parsing Need for Context 35

Image Parsing with Context

Bushs Memex (1945) Store publications, correspondence, personal work, on microfilm Items retrieved rapidly using index codes –Builds on rapid selector Can annotate text with margin notes, comments Can construct a trail through the material and save it –Roots of hypertext Acts as an external memory

Visual Memex, a proposal [Malisiewicz & Efros] Nodes = instances Edges = associations types of edges: visual similarity spatial, temporal co- occurrence geometric structure language geography.. New object

Torralbas Context Challenge

2 1 Slide by Antonio Torralba

Torralbas Context Challenge Chance ~ 1/30000 Slide by Antonio Torralba

Our Challenge Setup Malisiewicz & Efros, NIPS09

3 models Visual Memex: exemplars, non-parametric object-object relationships –Recurse through the graph Baseline: CoLA: categories, parametric object-object relationships Reduced Memex: categories, non- parametric relationships

Qual. results

Quant. results

Next Step: top-down segmentation Visual Memex A B C B C

Take Home Message Categorization is not a goal in itself –Rather, it is a means for transferring knowledge onto a new instance Skipping explicit categorization might make things easier, not harder –The harder intermediate problem syndrome Keeping around all your data isnt so bad… –you never know when you will need it

Questions? +