Approximate Models for Fast and Accurate Epipolar Geometry Estimation James Pritts Ondrej Chum and Jiri Matas Center for Machine Perception (CMP) Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics

Introduction RANSAC commonly used for robust epipolar geometry estimation very effective Runtime is given by proportion of measurements that are good in the data The sample size needed to hypothesize a model Approximate models requires smaller sample size But give worse estimates of epipolar geometry Epipolar geometry estimation by hypothesizing approximate models fit from 2 region-to-region correspondences Contributions We propose 2 new 2 region-to-region methods for estimating epipolar geometry Experimentally compare to previous approaches

Fitting a Line Least squares fit

RANSAC Sample s points

RANSAC Sample s points Fit model to sample

RANSAC Select sample of m points at random Fit model to sample Calculate error for each measurement

RANSAC Select sample of m points at random Fit model to sample Calculate error for each measurement Find consensus set

RANSAC Select sample of m points at random Fit model to sample Calculate error for each measurement Find consensus set Repeat sampling Hypothesize Verify

RANSAC Select sample of m points at random Fit model to sample Calculate error for each measurement Find consensus set Repeat sampling

RANSAC Select sample of m points at random Fit model to sample Calculate error for each measurement Find consensus set Repeat sampling

RANSAC Large sample size means more samples required! Terminates when finding a better solution is improbable. Large sample size means more samples required!

How bad is it? I / N [%] Size of the sample m Reduce sample size for fundamental matrix estimation by hypothesizing approximate models I / N [%] Size of the sample m Difficult wide baseline stereo matching problems at 95% confidence

Classic RANSAC for F estimation PROBLEM Measurements are noisy Hypotheses from small samples are only approximate Classic RANSAC does not really work for complex models Low accuracy, low precision Need Locally Optimized RANSAC

Generalized Model Optimization Approximate hypothesis: We propose 2 reduced parameter models to estimate the Fundamental Matrix from 2 regions Model Optimization: Local optimization step estimates the Fundamental Matrix from the consensus set of the approximate hypothesis O. Chum, J. Matas, and J. Kittler. Locally optimized RANSAC. In DAGM Symposium, 2003.

Feature-to-feature correspondences Affine covariant features detected in images independently Correspondences established based on SIFT descriptors x’3 x3 DG x2 DG x1 x’1 x’2 image 1 image 2 Centers of the region regions provide point-to-point correspondences (x1, x’1) (7 correspondences needed to estimate F) Dominant orientation (gradient) provides additional point-to-point correspondence (x2, x’2) Full affine coordinate frame is given by an ellipse and dominant orientation (x3, x’3) Region-to-region correspondence of affine covariant features provides three point-to-point correspondences: independent but ill-conditioned

Prior methods I: BEEM Constructs artificial points by locally approximating the image-to-image mapping to generate 8 linear constraints to estimate epipolar geometry 2 oriented ellipses, e.g. gotten by SIFT dominant gradient (DG) DG Artificial point L. Goshen and I. Shimshoni. Balanced exploration and exploitation model search for efficient epipolar geometry estimation. Pattern Analysis and Machine Intelligence, 30(7):1230–1242, 2008.

Prior Methods II: Perdoch et al. RANSAC hypothesis is Essential matrix and focal length by 6-point algorithm Estimation requires solving a system of polynomial equations 6 points gotten from oriented ellipse Introduce constraints on intrinsic camera parameters: fixed focal length, square pixels and centered principal point DG Local Affine Frame (LAF) M. Perdoch, J. Matas, and O. Chum. Epipolar geometry from two correspondences. In Proc. of ICPR, pages 215–220, 2006.

Affine epipolar constraint Measured points are used to estimate an approximation of the global model The first order approximation of can be written , where Geometric meaning: affine cameras Weak perspective geometry

MLE RANSAC hypothesis is the affine fundamental matrix Point-point constraints used for estimation Use extents and origin of 2 Local Affine Frames (LAFs) to get 4-6 points Optional point DG DG DG Local Affine Frame (LAF)

Arendelovic RANSAC hypothesis is the affine fundamental matrix Ellipses directly used for estimation Introduces polynomial constraints on , results in quartic (0-4) solutions Arbitrary unknown rotation Arandjelovic, Zisserman: Efficient Image Retrieval for 3D Structures , BMVC 2010 Affine epipolar geometry estimated from two elliptical correspondences 0-4 hypotheses

Summary of Hypothesis Generators Proposed approximate models Prior approximate models

KUSVOD2 dataset 16 challenging wide-baseline stereo pairs with varied geometry Hand annotated ground truth point correspondences Reprojection error is used to evaluate accuracy of Fundamental matrix Epipolar geometry estimated 500x for each pair K. Lebeda, J. Matas, and O. Chum. Fixing the locally optimized RANSAC. In Proc. of BMVC .

CDF reprojection error Empirical CDF of the RMS Sampson error for 8000 F estimations. Error calculated on hand-annotated point correspondences for each stereo pair in the test set.

Speedup 20% more than 25x speed up 50% more than 10x speed up Empirical CDF of the sample ratio, calculated as the ratio of the number of RANSAC samples required by the 7-point method to the denoted approximate model. Again for 8000 F estimations

Conclusions Two two-region methods are proposed that reliably estimate F on a large class of stereo pairs Most significant reduction in samples drawn and models evaluated can be achieved with the use of MLE. Using the first order approximation to the global model improves accuracy and precision of epipolar geometry estimation compared to using local image approximations or constraining intrinsic camera calibration Hypothesizing the affine fundamental matrix from 2 Local Affine Frames (LAFs) should be preferred to the previously proposed methods BEEM and Perdoch et al..

Additional material Comes after this slide…

Affine epipolar constraint Seek an approximation to the global model that uses only reliable constraints from local measurements. First order approximation of about : Where is the 1x4 Jacobian of The approximation can be written , where Weak perspective geometry