May 9, 2005 Andrew C. Gallagher1 CRV2005 Using Vanishing Points to Correct Camera Rotation Andrew C. Gallagher Eastman Kodak Company

May 9, 2005 Andrew C. Gallagher2 CRV2005 Problem An unintentionally tilted camera can negatively affect image appearance. Caused by lightweight cameras that are difficult to hold level. People prefer images where the horizon is level. Human can see as little at 1 o tilt.

May 9, 2005 Andrew C. Gallagher3 CRV2005 Solution Vanishing point location can be used to detect and correct image tilt resulting from camera rotation. A vanishing point is the image of a world line at infinity. Vanishing point location is useful for: –computing focal length [Kanatani] –finding principal point [Caprile et al.] –determining camera parameters and rotation matrix[Cipolla et al.]

May 9, 2005 Andrew C. Gallagher4 CRV2005 Vanishing Points Parallel scene lines meet at a vanishing point in the image. Horizontal Line Vanishing Point Vertical Line Vanishing Point

May 9, 2005 Andrew C. Gallagher5 CRV2005 The Camera Model The camera model describes the projection of 3D world to 2D camera plane. K is a 3x3 matrix of the internal camera parameters. R is a 3x3 matrix describing the rotation from the world to the camera frame. T is a 3x1 matrix describing translation between the world and camera coordinate frame. Assume no skew, square pixels. The vanishing points of world directions are: camera coordinate frame world coordinate frame

May 9, 2005 Andrew C. Gallagher6 CRV2005 The Rotation Matrix R is any matrix in the special orthogonal group SO(3). In practice the camera positions used by typical consumers follow a fairly predictable nonlinear distribution. This distribution is then used to find where vanishing points will occur.

May 9, 2005 Andrew C. Gallagher7 CRV2005 Camera Position Analysis Default Position World rotation by  about the Y-axis World rotation about the Z-axis TILTED IMAGE World rotation by  about the X-axis

May 9, 2005 Andrew C. Gallagher8 CRV2005 Camera Position Analysis Rotation about both X- and Y- axes This position model encompasses all “preferred” camera positions. The vanishing point associated with vertical world direction (Y-axis) is constrained to fall on the image y-axis. The horizon is parallel to image x-axis. Location of V y Location of V x or V z

May 9, 2005 Andrew C. Gallagher9 CRV2005 Camera Position Analysis Additional rotation about the Z-axis The original rotation matrix is multiplied by a rotation about the Z-axis. The new vanishing points are simply rotated by the same amount! In essence, the rotation of the camera from the ideal position is equivalent to the rotation of the vanishing points. Location of V x or V z Location of V y

May 9, 2005 Andrew C. Gallagher10 CRV2005 Ground Truth Analysis Location of V x or V z Location of V y 357 vanishing points were manually labeled to compare with expected distribution. 160 vertical (V y ) vanishing points 197 horizontal (V x or Vz ) vanishing points. The match is visually good. EXPECTED DISTRIBUTION MEASURED DISTRIBUTION

May 9, 2005 Andrew C. Gallagher11 CRV2005 Vanishing Point Classification The vertical and horizontal vanishing point distributions are well-separated. A classifier can be used to identify vertical vanishing points. The camera rotation is found from the vertical vanishing point. On ground truth, only 2 vanishing points (0.6%) are misclassified. Vertical vanishing point classifier

May 9, 2005 Andrew C. Gallagher12 CRV2005 The Tilt Correction Algorithm Find vanishing points Identify vertical vanishing points Compute camera rotation angle  from the vertical vanishing point Compute correction angle  c according to table: Rotate image The rotated image can be shown to be equivalent to capturing with a camera having no component of rotation about the Z-axis. ^

May 9, 2005 Andrew C. Gallagher13 CRV2005 Vanishing Point Detection Initial work by Barnard Line Segment Detection –Lines are found by calculating local gradients, then clustering, or by using Hough transform. Line Intersection Computation –Intersections of the lines are found. Line intersections are possible locations of a vanishing point. Maximum Detection –A detected vanishing point is hypothesized to be at a location of many line intersections.

May 9, 2005 Andrew C. Gallagher14 CRV2005 Vanishing Point Detection Original Image Detected Line Segments Plot of all line segment Intersections (Higher probabilities are red). Lines associated with 1 st VP. Lines associated with 2 nd VP. Detected Vanishing Points

May 9, 2005 Andrew C. Gallagher15 CRV2005 Algorithm Results Original Lines associated with vertical VP Corrected

May 9, 2005 Andrew C. Gallagher16 CRV2005 Algorithm Results Original Lines associated with vertical VP Corrected

May 9, 2005 Andrew C. Gallagher17 CRV2005 Conclusions Rotation of the camera about the principal axis moves the vertical vanishing point from the image y-axis. This novel algorithm corrects a tilted image by detecting the vertical vanishing point, and determining the magnitude of camera rotation.