Object recognition

Object Classes

Individual Recognition

Is this a dog?

Variability of Airplanes Detected

Variability of Horses Detected

ClassNon-class

Recognition with 3-D primitives Geons

Visual Class: Common Building Blocks

Optimal Class Components? Large features are too rare Small features are found everywhere Find features that carry the highest amount of information

Entropy Entropy: x =01H p =0.50.5?

Mutual Information I(x,y) X alone: p(x) = 0.5, 0.5H = 1.0 X given Y: Y = 0 Y = 1 p(x) = 0.8, 0.2 H = 0.72 p(x) = 0.1, 0.9 H = 0.47 H(X|Y) = 0.5* *0.47 = H(X) – H(X|Y) = 1 – = I(X,Y) = 0.405

Mutual information H(C) when F=1H(C) when F=0 I(C;F) = H(C) – H(C/F) F=1 F=0 H(C)

Mutual Information II

Computing MI from Examples Mutual information can be measured from examples: 100 Faces100 Non-faces Feature: 44 times 6 times Mutual information: H(C) = 1, H(C|F) =

Full KL Classification Error FC p(F|C) q(C|F) p(C)

Optimal classification features Theoretically: maximizing delivered information minimizes classification error In practice: informative object components can be identified in training images

Selecting Fragments

Adding a New Fragment (max-min selection) ? MIΔ MI = MI [ Δ ; class ] - MI [ ; class ] Select: Max i Min k ΔMI (Fi, Fk) (Min. over existing fragments, Max. over the entire pool)

Highly Informative Face Fragments

Intermediate Complexity

Decision Combine all detected fragments F k : ∑w k F k > θ

Optimal Separation SVMPerceptron ∑wk Fk = θ is a hyperplane

Combining fragments linearly Conditional independence: P(F1,F2 | C) = p(F1|C) p(F2|C) > θ> θ > θ> θ W(Fi) = log Σw(Fi) > θ

If Fi=1 take log If Fi=0 take log Instead: Σ w i > θ On all the detected fragments only With: w i = w(Fi=1) – w(Fi=0)

Class II

ClassNon-class

Fragments with positions ∑w k F k > θ On all detected fragments within their regions

Horse-class features

Examples of Horses Detected

Interest points (Harris) SIFT Descriptors I x 2 I x I y I x I y I y 2 ∑

Harris Corner Operator < I x I y < H = Averages within a neighborhood. Corner: The two eigenvalues λ1, λ2 are large Indirectly: ‘Corner’ = det(H) – k trace 2 (H)

Harris Corner Examples

SIFT descriptor David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp Example: 4*4 sub-regions Histogram of 8 orientations in each V = 128 values: g 1,1,…g 1,8,……g 16,1,…g 16,8

SIFT

Constellation of Patches Using interest points Fegurs, Perona, Zissermann 2003 Six-part motorcycle model, joint Gaussian,

Bag of words and Unsupervised Classification

Bag of visual words A large collection of image patches –

Each class has its words historgram – – –

pLSA Classify document automatically, find related documents, etc. based on word frequency. Documents contain different ‘topics’ such as Economics, Sports, Politics, France… Each topic has its typical word frequency. Economics will have high occurrence of ‘interest’, ‘bonds’ ‘inflation’ etc. We observe the probabilities p(w i | d n ) of words and documents Each document contains several topics, z k A word has different probabilities in each topic, p(w i | z k ). A given document has a mixture of topics: p(z k | d n ) The word-frequency model is: p(w i | d n ) = Σ k p(w i |z k ) p(z k | d n ) pLSA was used to discover topics, and arrange documents according to their topics.

pLSA The word-frequency model is: p(w i | d n ) = Σ k p(w i |z k ) p(z k | d n ) We observe p(w i | d n ) and find the best p(w i |z k ) and p(z k | d n ) to explain the data pLSA was used to discover topics, and then arrange documents according to their topics.

Discovering objects and their location in images Sivic, Russel, Efros, Freedman & Zisserman CVPR 2005 Uses simple ‘visual words’ for classification Not the best classifier, but obtains unsupervised classification, using pLSA

Visual words – unsueprvised classification Four classes: faces, cars, airplanes, motorbikes, and non-class. Training images are mixed. Allowed 7 topics, one per class, the background includes 3 topics. Visual words: local patches using SIFT descriptors. –(say local 10*10 patches) codewords dictionary

Learning Data: the matrix D ij = p(w i | I j ) During learning – discover ‘topics’ (classes + background) p(w i | I j ) = Σ p(w i | T k ) p(T k | I j ) Optimize over p(w i | T k ), p(T k | I j ) The topics are expected to discover classes Got mainly one topic per class image.

Results of learning

Classifying a new image New image I: Measure p(w i | I) Find topics for the new image: p(w i | I) = Σ p(w i | T k ) p(T k | I) Optimize over the topics T k Find the largest (non-background) topic

Classifying a new image

On general model learning The goal is to classify C using a set of features F. F have been selected (must have high MI(C;F)) The next goal is to use F to decide on the class C. Probabilistic approach: Use observations to learn the joint distribution p(C,F) In a new image, F is observed, find the most likely C, Max (C) p(C,F)

General model learning To learn the joint distribution p(C,F): The model is of the form p θ (C,F) –Or: p θ (C,X,F) For example we had – words in documents: –p(w,D) = Πp(w i,D) –p(w i | D) = Σ p(w i | T k ) p(T k | D) Training examples used to determine optimal θ by maximizing p θ (data) – max (C,X, θ) p θ (C,X,F) When θ known, classify new example: –max (C,X) p θ (C,X,F)