1. Efficient Sampling Methods for Robust Estimation Problems
Wei Zhang and Jana Kosecka
Department of Computer Science
Computer Vision and Robotics Laboratory
George Mason University

2. Motivational Problems
Localization
How to enhance navigational capabilities of robots/people in urban areas
Where am I ? – Localization problem
GPS - global coordinates - longitude/latitude
Relative pose with respect to known
landmarks
Given database of location views
Recognize the most likely location
Compute relative pose of the camera with respect to the reference view

3. Now imagine a visitor on the campus of George Mason University
Now imagine a visitor on the campus of George Mason University. In order to know his location, he can look around and see which building is in his view. Provided that the buildings can be recognized, he can take a snapshot of the building (click) “I am on the left side (30 degress) of Fenwick”. Or (click) “I am about 45 degree left to Fenwick”. Or “I am in front of Krug Hall”. If the location of buildings is known more detailed navigational suggestions (such as path planning) can be accomplished.

4. First part of the talk
Wide base line correspondences in natural environment are difficult to obtain

5. Robust Estimation Problems in Computer Vision
Motion estimation problems
- estimating rigid body motion (essential, fundamental matrix, homography)
Range estimation problems
Data are corrupted with outliers
- mismatches due to errorneous correspondence and/or tracking errors
- range data outliers
Traditional means for tackling the presence of outliers
- sampling based methods (RANSAC)
- use of robust objective function

6. Sampling RANSAC-like methods
Original algorithm developed by Fischler and Bolles
RANSAC algorithm
Generate number of model hypothesis by sampling minimal number of points needed to estimate the model
For each hypothesis compute the residual of all datapoints with respect to the model
Points with the (residual)^2 < th are classified as
inliers and the rest as outliers
Find the hypothesis with maximal support
Re-estimate the model using all inliers

7. Theoretical Analysis
Given m samples and p points per sample probability of an outlier free sample is
Given the knowledge of the percentage of the outliers in the data and the desired probability of finding at least one outlier free sample the number of samples which needs to be generated can be computed as

8. Practice For fundamental matrix computation
using 8-point alg. 766 samples is needed
for 95% confidence of outlier free sample
1177 samples - 99% confidence
766 samples – 95% confidence
Theoretical estimates are too optimistic
In practice around 5000 samples needed
Once the hypothesis are generated by sampling
- evaluation process is very expensive – each data
point needs to be evaluated

9. Drawbacks Drawbacks of currently existing methods
- require large number of samples
- need to know the percentage of outliers
- require threshold for determining which points
are inliers and which outliers
- additional efficiency concerns are related to the
time consuming process of evaluation of individual
hypotheses
Improvements to standard approaches [Torr’99, Murray’02, Nister’04, Matas’05, Sutter’05 and many others]
Mostly improvement in stopping sampling criteria, hypothesis evaluation criteria, efficient search though hypothesis evaluation space
Hypothesis evaluation idea remained unchanged

10. Robust Objective function
Techniques using robust statistics - use
of robust objective function, nonlinear minimization
Robust estimators have typical cut-off point
below 50% of outliers [MeerStewart’99, Meer-Comaniciu’00]
LMeds estimators [Rouseaw’95, Torr, Black]

11. Our Approach
Observation: the final step of the sampling based methods is for classification of points as inliers and outliers
We instead of classifying points based on finding a good hypothesis and evaluating points with respect to that hypothesis, we propose to classify the points directly
Basic observation – analyze the distribution of the residuals for each datapoint with respect to all hypotheses
Example:

12. Example inlier outlier
Distribution of residuals of one datapoint with respect
to all generated hypothesis (line fitting example, 20% of
outliers
Distributions of outliers and inliers are qualitatively different
Compute n-th order statistics and use it for classification
inlier
outlier

13. Example inlier outlier Inlier/outlier distributions with 50% outliers
Computing n-th order statitics of these distributions will enable us to disambiguate inlier and outlier distributions
Formulate outlier and inlier identification as classification
problem
inlier
outlier

14. Skewness and Kurtosis skewness kurtosis inliers outliers
Compute skewness and kurtosis of each data point
skewness
kurtosis
inliers outliers
inliers outliers

15. Skewnews/Kurtosis Features
inliers
outliers
Classification by k-means clustering in kurtosis space

16. Classification Results
True positive rate of 68% and false positive rate 1%

17. Final Algorithm 1. Randomly generate N hypotheses by sampling
2. For each datapoint compute its residual error
with respect to each hypothesis
3. Compute distribution of the errors for each datapoint
4. Compute skewness and kurtosis of each distribution
5. Perform K-means clustering in skewness (kurtosis) space
6. Classify the points as inliers/outliers depending
which cluster they belong
Alternatively we can assign a probability of each points
being and inlier or outlier
Fixed number of samples used to compute the histogram
(500 for Fundamental matrix estimation)
in these experiments

18. Sensitivity
Means and 95% confidence intervals for inliers/outliers

19. Experiments Feature detection – SIFT features
Matching – find the nearest neighbour in the
next view with the distance below 0.96 threshold
on the cos of the angle between two descriptors
Due to the repetitive structures often present in natural, street/building scenes, wide baseline matching contains many outliers (sometimes not problem for recognition)

20. 36 correctly identified, 1 false positive
Experimental Results
776 matches, 50% outliers
No false positives
285 matches, 70% outliers,
36 correctly identified, 1 false positive

21. Conclusions New inlier identification scheme for RANSAC
like robust regression problems
Inliers can be identified directly without evaluating each hypothesis
Up to 70% of inliers can be tolerated, with
only 500 samples (depends on the estimation
problem and objective function surface sensitivity) – experiments for Fundamental matrix and homography estimation
References (submitted to ECCV 2006)
Novel inlier identification scheme for robust estimation problems. W. Zhang and J. Kosecka
Nonparametric estimation of multiple structures with outliers. W. Zhang and J. Kosecka

22. Robust Estimation of Multiple Structure
What does the distribution of residuals reveal
in the presence of multiple structures ?
Line fitting examples
Distribution of residuals
for one datapoint
In case of inliers # of peaks
corresponds to
# of models

23. Distribution of residuals
Observation – hypothesis generated by sampling form clusters in the hypothesis spaces, yielding the similar residual errors
These correspond to the peaks in histogram of residuals for each datapoint
peaks identification
more difficult
outliers 83% for each line

24. Mode detection possibility – mean shift algorithm – difficulties
with choosing the bandwith parameter (window size)
Peak detection algorithm
1. Smooth the histogram and located local maxima and minima
2. Remove spurious weak modes so only one valley is present between the two modes
3. Choose the weakest mode, if distinct enough add it to the list of modes, else remove

25. Calculating the number of models
Due to the difficulties of peak detection – different number of peaks is detected in different distributions
Median – reliable estimate of # of models
Example: # of models = 3, estimates obtained from
distributions for all datapoints

26. Identifying the model parameters
correct line hypotheses
spurious line hypotheses
1. Select datapoints with distinctive modes in error distribution
2. Modes generated by various hypotheses (good and bad)
yielding same residuals
3. Choose another point, although the residuals are different
cluster can now be identified
4. Enables efficient seach for clusters of correct hypotheses

27. Examples – Line Fitting

28. Examples Motion Estimation
a) Piecewise planar models - homography estimation
b) Multiple motions – purely translational models

29. Dataset Zurich building database
201 buildings, each with 5 reference images. 115 test images in query database.
Additional campus dataset 68 buildings, 3 grayscale images each.

30. Approach overview Hierarchical Building Recognition
Localized color histogram based screening.
Local appearance based decision.
Relative Pose recovery
Based on rectangular structure:
Rectangular structure extraction.
Given single (query) view, recover the relative camera pose (translation up to scale and orientation).
With respect to the reference view:
Feature matching between query and model view
Compute the relative pose between the current and reference view
There are two different modes how the pose can be recovered. The rectangular structure extraction may not work for all
Buildings and hence it may be more restrictive

31. Vanishing point estimation
a group of parallel lines in the world intersect in the image.
Based on detected line segments, estimate both grouping information and vanishing points locations using EM algorithm. [Kosecka&Zhang, ECCV02]
Man made environments, The group information is unknown.

32. Localized color histograms
The background pixels are mostly remove, so they have little influence in the final histograms.

33. Localized color histogram indexing
Given query view select a variable number of candidate models. The number of models depends on confidence of each individual recognition .
Correct recognition results
Recognition is not correct, but true model is listed.
About half of buildings can be recognized with high confidence, only 1 model is selected. No further action is needed. While other half selected number of candidates is on average 3. Further steps is needed to make final decision. We will local appearance based technique to get finer result.
JK-comment - add the table with the recognition results of the first stage - as in the paper - the numbers should appear in the slide

34. Local feature based refinement
In this stage we exploit the SIFT keypoints and their associated descriptors introduced by D. Lowe [Lowe04].
Keypoints of test image are matched to those of the models selected by the first stage.
Voting determines the most likely model - model with highest
number of matches
Wei, you cannot see anything in the figures. You have to make them bigger - don’t just cut and paste from the
Paper -

35. Summary of recognition results
Recognition based on localized color histogram alone is fairly good, 90% recognitions are correct, comparable to other appearance based works [Hshao03][Goedeme04].
When using keypoint matching directly, it takes over 10 minutes to compare a query image with all the reference images.
The localized color histogram uses about 3 seconds to process same number of reference images.
The hierarchical scheme achieves 96% recognition rate, better than previously reported results.
Point 2 here should be taken out - since using fast nearest neighbour search methods this can be easily done in real time