Video Compass Jana Kosecka and Wei Zhang George Mason University www.cs.gmu.edu/~kosecka

Motivation Information encoded in vanishing points and lines Visually Guided Agents - Navigation 3D Reconstruction Camera pose recovery Man made environments Many structural regularities Parallel and orthogonal lines and planes Reveal a lot about the scene structure And camera pose Information encoded in vanishing points and lines

Previous Approaches Vanishing point recovery approaches vary in Line detection and representation Grouping of line segments to vanishing directions Choice of objective function and optimization technique Assumptions about the camera Antone’00, Collins’90,Caprile’90,Kanatani’93, McLean’97,Tuytelaars’98 Our approach Line represented in projective space Grouping and VP estimation done simultaneously using EM Efficient initialization using orientation histograms Partial camera calibration

Line detection Stage Edge detection, non-maximum suppression (traditionally Hough Transform – issues of resolution, threshold selection and search for peaks in Hough space) Connected components on edge pixels with similar orientation - group pixels with common orientation Line fitting Lines determined from eigenvalues and eigenvectors of A Candidate line segments - associated line quality

Pinhole Camera Imaging Model Perspective Projection: Image points Image lines

vp Vanishing point – intersecting point of two parallel lines Given a set of line segments belonging to the same vanishing direction vanishing point can be estimated LLS on Gaussian Sphere [Teller 00, Collins 97] For calibrated camera Given a set of line segments (parallel in 3D), belonging to the same vanishing direction, vanishing point can be estimated as LLS problem on a Gaussian sphere

Grouping and Estimation Basic Premise - In man made environments majority of parallel lines is aligned with the principal axis of the world reference frame. Given a set of line segments automatically group the line segments into three dominant vanishing directions and estimate the associated vanishing points

Probabilistic inference with an unknown model Estimate simultaneously the coordinates of vanishing points as well as probabilities of individual line segments belonging to particular vanishing direction [Antone’00] Bayes rule - Vanishing point posterior – line likelihood Conditional mixture model Number of models – one for each dominant direction and one for outlier process

EM algorithm Objective function – estimate VP by maximizing posterior Probability of the vanishing points given a set of line segments where yields:

Uncalibrated Camera Image points Image lines Perspective Projection: Image points Image lines Augmented minimization problem is not well conditioned Line segments are not well separated on the Gaussian Sphere

Initialization All the measurements need to be normalized Use approximate camera intrinsic parameter matrix Normalize all the coordinates by Compute initial groups of lines Initialization Compute initial vanishing directions

Initial groups are determined from the peaks of the line orientation histogram Find zero crossings of the histogram curvature Line orientation –Pi, Pi Line orientation –Pi, Pi

Compute initial estimates of vanishing directions E-step for each line estimate likelihood M-step estimate so as to maximize complete log-likelihood k=1,…,m Iterate until you reach the equilibrium For finite vanishing points refine final estimates using nonlinear ML objective function [Hartley& Zisserman,00]

EM Iterations number of models is adjusted during the iteration

Applications: Single View Analysis uncalibrated single view geometry exploit information about vanishing points for partial calibration and reconstruction

Vanishing points constraints three vanishing points in the image plane orthogonal zero skew two vanishing points in the image plane – recover focal length linearly

Applications: Matching and Mosaic Construction

Efficient Technique for Estimation of Vanishing points Conclusions Efficient Technique for Estimation of Vanishing points two passes over image and 3-7 iterations of EM on average Constraints of man-made environment were used in different stages of image processing pipeline Ongoing work Pose recovery – recovery of relative orientation Partial camera calibration Reconstruction from a single view Image matching