Algebraic Functions of Views for 3D Object Recognition CS773C Advanced Machine Intelligence Applications Spring 2008: Object Recognition

Object Appearance The appearance of an object can have a large range of variation due to: –Photometric effects –Scene clutter –Changes in shape (e.g., non-rigid objects) –Viewpoint changes

Algebraic Functions of Views (AFoVs) A powerful mathematical foundation for investigating variations in the geometrical appearance of an object due to viewpoint changes. “the variety of of 2D views depicting the geometrical appearance of a 3D object can be expressed as a combination of a small number of 2D views of the object” “the variety of of 2D views depicting the geometrical appearance of a 3D object can be expressed as a combination of a small number of 2D views of the object” S. Ullman and R. Basri, "Recognition by Linear Combinations of Models", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 10, pp , 1991.

Orthographic Projection Case of 3D rigid transformations (3 ref. views)

Orthographic Projection Case of 3D linear transformations (2 ref views)

More Results … Perspective projection (2 ref. views, obtained under orthographic projection) Objects with smooth surfaces and non-rigid objects –More reference views are required. A. Shashua, “Algebraic functions for recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 8, pp , 1995.

A Word of Caution! Only common features in the reference views can be predicted in a novel view. novel view reference view

Recognition Framework Using AFoVs “novel 2D views of a 3D object can be recognized by matching them to combinations of a small number of known 2D views of the object” “novel 2D views of a 3D object can be recognized by matching them to combinations of a small number of known 2D views of the object”

Representation and Matching using AFoVs Representation –Objects are represented by a small number of views. –Each view is represented by some geometric features (e.g., points) Matching – Predict the geometric appearance of an object in a novel view by combining a small number of reference views of the object.

Advantages of the Method No 3D models or camera calibration are required. Only a small number of 2D views are required. Novel views can be different from the stored ones. Simpler verification scheme. More general framework (“family” of methods). Evidence that the human visual system works similarly.

Main Challenges Which model views to combine to predict a novel view? How to establish the correspondences between novel and reference views? How to find the coefficients of the combination?. How to handle occlusions? How to choose the reference views? Integrate AFoVs with Indexing!

Method Overview (G. Bebis, M. Georgiopoulos, M. Shah, and N. da Vitoria Lobo, "Indexing Based on Algebraic Functions of Views", Computer Vision and Image Understanding (CVIU), Vol. 72, No. 3, pp , 1998) Preprocessing step (1) Extract groups of points from each model. (2) Sample the space of appearances of each group. (3) Store information about the groups in an index table Recognition step (1) Extract groups of points from the scene. (2) Predict their appearance. (3) Verify the predictions.

Overview of the Method (cont’d)

Which Model Groups to Choose? Cluster geometric features into higher level descriptions. Consider properties that are unlikely to occur at random. convexityProperty used in our work: convexity

Which Model Groups to Choose? (cont’d)

How to generate the appearances of a group? Estimate each parameter’s range of values Sample the space of parameter values Generate a new appearance for each sample of values

Estimate the Range of Values of the Parameters or Using SVD: and

Estimate the Range of Values of the Parameters (cont’d) Assume normalized coordinates: Use Interval Arithmetic (Moore, 1966) ( note that the solutions will be identical )

Example

Preconditioning the Reference Views : Transform the original views to new views such that has the best possible condition. effect of the condition number of P on the intervals

Preconditioning the Reference Views (cont’d) Choosing: This implies: Thus:

Example (preconditioned views)

Decouple Image Coordinates Same transformation generates the x- and y-coordinates: Represent only the x-coordinates in the index table. For each group, store the following entry:

Hypothesis Generation and Verification 1. take intersection of hypotheses model 2. apply constraints to reject invalid hypotheses

How to Choose the Scene Groups? convex groupingUsing convex grouping to extract salient scene groups.

Implementation Issues Space requirements –select salient groups –reject groups giving rise to bad conditioned matrices –coarse sampling of parameters Index computation and table size

Important Implementation Issues (cont’d) Sampling step (i.e., parameters of AFoVs) Noise tolerance actual: predicted: make additional entries in a neighborhood around the indexed location

Experiments and Results model objects and reference views used in our experiments

Experiments and Results (cont’d) reference views novel view

Experiments and Results (cont’d) reference views novel view

Experiments and Results (cont’d) reference views novel view

Criticism of the Method Relies heavily on feature extraction It has high memory requirements. The index table might represent unrealistic model appearances. Indexing based on hashing is not very efficient. No explicit ranking of hypotheses.

Improving AFoVs Recognition Framework Reject unrealistic appearances Reduce storage requirements and improve speed Develop a probabilistic hypothesis generation scheme –Learn shape appearance –Rank hypotheses Represent object appearance more efficiently using improved indexing schemes and probabilistic models. W. Li, G. Bebis, and N. Bourbakis, "Integrating Algebraic Functions of Views with Indexing and Learning for 3D Object Recognition", IEEE Workshop on Learning in Computer Vision and Patter Recognition (in conjunction with CVPR04), Washington DC, June 28, 2004.

Combine Indexing with Learning Sample the space of appearances sparsely and represent the samples in a K-d tree Sample the space of views densely and represent the samples using probabilistic models. Given a novel view: (1) Use K-d tree to retrieve a small number of candidate models (2) For each candidate model, compute the probability that it might have produced the novel view (3) Verify most likely hypotheses first

Combine Indexing with Learning (cont’d) The first stage provides hypothetical matches fast. The second stage evaluates the feasibility of hypothetical matches fast, without having to apply verification explicitly. Only “highly likely” hypotheses are verified explicitly.

Improved Framework Reference views Extract model groups Estimate the range of AFoVs parameters Sampling AFoVs parameter space New image Extract image groups K-d Tree Rank hypotheses Estimate AFoVs parameters Verify hypotheses TRAINING PHASE RECOGNITION PHASE Using SVD & IA Access Retrieve Validate views Hypothetical matches Low-dimensional representation Manifold learning using EM Recognition results Random Projection densecoarse dense coarse

Eliminate Unrealistic Model Appearances Under the assumption of linear transformations, many unrealistic views could be generated. Impose rigidity constraints to eliminate them. –Storage requirements can be reduced significantly. –Recognition becomes faster and more efficient.

Eliminate Unrealistic Model Appearances Unrealistic Views (without constraints) Realistic Views (with constraints)

Indexing Appearances Sample the space of views “coarsely” and represent the samples in an index table. Hashing might not very well in this case... Need an improved indexing scheme.

Range Search vs Nearest Neighbor Search Nearest Neighbor Search Range Search Range search is not appropriate when storing a sparse number of views. K-d trees perform a nearest-neighbor search.

K-d Trees for Indexing P1 P2 P3 P4 P1P2P3 P4 K-d trees perform a nearest-neighbor search.

Learning Geometric Appearance We can pre-compute the views that an object can produce off-line. These views form a manifold in lower dimensional space. Model object appearance using a pdf. –Sample the space of appearances. –Fit a parametric model (e.g., mixtures of Gaussians using EM). –Use mutual information theory to choose the number of components. EM has problems when the dimensionality of the data is high. Apply “Random Projection” first, then run EM algorithm.

Manifolds of Real Objects: An Example Need to store a small number of parameters only for each model

Hypothesis Ranking where Each hypothesis generated by the K-d tree is ranked by computing its probability using mixture models. For each test group, we compute two probabilities, one from x coordinates, and the other from y coordinates. The overall probability for a particular hypothesis is computed according to the following equation:

Reference Views 1 st Reference view 2 nd Reference view

Reference Views (cont’d) 1 st Reference view 2 nd Reference view

Test Views (a)(b)(c) (d) (e) (f)

Test Views (cont’d) Hypothesis rejected

Integrate Geometric Appearance with Intensity Appearance Using geometrical information only does not provide enough discrimination for objects having similar “geometric” appearance but probably different “intensity” appearance. Integrating geometric and intensity apperance during hypothesis verification to improve discrimination power and robustness. W. Li, G. Bebis, and N. Bourbakis, "3D Object Recognition Using 2D Views", IEEE Transactions on Image Processing (under revision).

Dense Correspondences For each group of corresponding points, apply triangulation recursively to get denser correspondences. Divide triangles into four sub-triangles by considering the middle point of each side of each triangle.

Refine AFoVs parameters (before refinement)(after refinement)

Predict Intensity Appearance - Example Reference view 1Reference view 2 Test view Prediction

Predict Intensity Appearance - Example Reference view 1 Reference view 2 Test viewPrediction

Predict Intensity Appearance - Example (hypothesis rejected) (hypothesis accepted)