The Brightness Constraint

Slides:



Advertisements
Similar presentations
Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.
Advertisements

MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
Two-View Geometry CS Sastry and Yang
Lecture 8: Stereo.
Image alignment Image from
776 Computer Vision Jared Heinly Spring 2014 (slides borrowed from Jan-Michael Frahm, Svetlana Lazebnik, and others)
Computer Vision Optical Flow
Camera calibration and epipolar geometry
Image alignment Image from
Structure from motion.
3D Computer Vision and Video Computing 3D Vision Topic 4 of Part II Visual Motion CSc I6716 Fall 2011 Cover Image/video credits: Rick Szeliski, MSR Zhigang.
Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.
Image alignment.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
Uncalibrated Geometry & Stratification Sastry and Yang
Feature matching and tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on.
Multiple-view Reconstruction from Points and Lines
CSc83029 – 3-D Computer Vision/ Ioannis Stamos 3-D Computational Vision CSc Optical Flow & Motion The Factorization Method.
Feature tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on good features.
Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
COMP 290 Computer Vision - Spring Motion II - Estimation of Motion field / 3-D construction from motion Yongjik Kim.
Matching Compare region of image to region of image. –We talked about this for stereo. –Important for motion. Epipolar constraint unknown. But motion small.
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and.
KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track
3D Motion Estimation. 3D model construction Video Manipulation.
Optical Flow Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)
3D Computer Vision and Video Computing 3D Vision Topic 8 of Part 2 Visual Motion (II) CSC I6716 Spring 2004 Zhigang Zhu, NAC 8/203A
Structure Computation. How to compute the position of a point in 3- space given its image in two views and the camera matrices of those two views Use.
Automatic Camera Calibration
Computer vision: models, learning and inference
Image Stitching Ali Farhadi CSE 455
Image alignment.
Lecture 11 Stereo Reconstruction I Lecture 11 Stereo Reconstruction I Mata kuliah: T Computer Vision Tahun: 2010.
Final Exam Review CS485/685 Computer Vision Prof. Bebis.
Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T Computer Vision Tahun: 2010.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
The Measurement of Visual Motion P. Anandan Microsoft Research.
Image stitching Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac.
CSCE 643 Computer Vision: Structure from Motion
Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:
Uses of Motion 3D shape reconstruction Segment objects based on motion cues Recognize events and activities Improve video quality Track objects Correct.
3D Imaging Motion.
Feature Matching. Feature Space Outlier Rejection.
Motion Estimation I What affects the induced image motion?
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
Uncalibrated reconstruction Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration.
776 Computer Vision Jan-Michael Frahm Spring 2012.
Optical flow and keypoint tracking Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
776 Computer Vision Jan-Michael Frahm Spring 2012.
Lecture 16: Image alignment
SIFT Scale-Invariant Feature Transform David Lowe
Motion and Optical Flow
René Vidal and Xiaodong Fan Center for Imaging Science
The Brightness Constraint
3D Motion Estimation.
Epipolar geometry.
Image Stitching Slides from Rick Szeliski, Steve Seitz, Derek Hoiem, Ira Kemelmacher, Ali Farhadi.
3D Photography: Epipolar geometry
Structure from motion Input: Output: (Tomasi and Kanade)
The Brightness Constraint
The Brightness Constraint
Uncalibrated Geometry & Stratification
Filtering Things to take away from this lecture An image as a function
Computational Photography
Chapter 11: Stereopsis Stereopsis: Fusing the pictures taken by two cameras and exploiting the difference (or disparity) between them to obtain the depth.
Structure from motion Input: Output: (Tomasi and Kanade)
Presentation transcript:

The Brightness Constraint Brightness Constancy Equation: Linearizing (assuming small (u,v)): Where: ) , ( y x J I t - = Each pixel provides 1 equation in 2 unknowns (u,v). Insufficient info. Another constraint: Global Motion Model Constraint

Requires prior model selection Camera induced motion = The 2D/3D Dichotomy Requires prior model selection Camera motion + Scene structure Independent motions Camera induced motion = + Independent motions = Image motion = 2D techniques 3D techniques Do not model “3D scenes” Singularities in “2D scenes”

Global Motion Models 2D Models: Affine Quadratic Homography (Planar projective transform) 3D Models: Rotation, Translation, 1/Depth Instantaneous camera motion models Essential/Fundamental Matrix Plane+Parallax

Example: Affine Motion Substituting into the B.C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns Least Square Minimization (over all pixels): (minimum 6 pixels necessary) Every pixel contributes  Confidence-weighted regression

Example: Affine Motion Differentiating w.r.t. a1 , …, a6 and equating to zero  6 linear equations in 6 unknowns:

Coarse-to-Fine Estimation Parameter propagation: Pyramid of image J Pyramid of image I image I image J Jw warp refine + u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels ==> small u and v ... image J image I

Other 2D Motion Models Quadratic – instantaneous approximation to planar motion Projective – exact planar motion (Homography H)

Panoramic Mosaic Image Alignment accuracy (between a pair of frames): error < 0.1 pixel Original video clip Generated Mosaic image

Video Removal Original Original Outliers Synthesized

Video Enhancement ORIGINAL ENHANCED

Direct Methods: Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image. Minimization step: Direct methods: Error measure based on dense measurable image quantities (Confidence-weighted regression; Exploits all available information) Feature-based methods: Error measure based on distances of a sparse set of distinct feature matches.

Example: The SIFT Descriptor Compute gradient orientation histograms of several small windows (128 values for each point) Normalize the descriptor to make it invariant to intensity change To add Scale & Rotation invariance: Determine local scale (by maximizing DoG in scale and in space), local orientation as the dominant gradient direction. Image gradients The descriptor (4x4 array of 8-bin histograms) Compute descriptors in each image Find descriptors matches across images  Estimate transformation between the pair of images. In case of multiple motions: Use RANSAC (Random Sampling and Consensus) to compute Affine-transformation / Homography / Essential-Matrix / etc. D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

Benefits of Direct Methods High subpixel accuracy. Simultaneously estimate matches + transformation  Do not need distinct features. Strong locking property.

Limitations Limited search range (up to ~10% of the image size). Brightness constancy assumption.

Video Indexing and Editing

Ex#4: Image Alignment (2D Translation) Differentiating w.r.t. a1 and a2 and equating to zero  2 linear equations in 2 unknowns:

A camera-centric coordinate system (R,T,Z) Camera induced motion = The 2D/3D Dichotomy A camera-centric coordinate system (R,T,Z) Camera motion + Scene structure Independent motions Camera induced motion = + Independent motions = Image motion = 2D techniques 3D techniques Do not model “3D scenes” Singularities in “2D scenes”

The Plane+Parallax Decomposition Original Sequence Plane-Stabilized Sequence The residual parallax lies on a radial (epipolar) field: epipole

Benefits of the P+P Decomposition Eliminates effects of rotation Eliminates changes in camera parameters / zoom 1. Reduces the search space: Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole) Image displacements: Constrained to lie on radial lines (1-D search problem) A result of aligning an existing structure in the image.

Benefits of the P+P Decomposition 2. Scene-Centered Representation: Translation or pure rotation ??? Focus on relevant portion of info Remove global component which dilutes information !

Benefits of the P+P Decomposition 2. Scene-Centered Representation: Shape = Fluctuations relative to a planar surface in the scene STAB_RUG SEQ

Benefits of the P+P Decomposition 2. Scene-Centered Representation: Shape = Fluctuations relative to a planar surface in the scene Height vs. Depth (e.g., obstacle avoidance) Appropriate units for shape A compact representation - fewer bits, progressive encoding total distance [97..103] camera center scene global (100) component local [-3..+3] component

Benefits of the P+P Decomposition 3. Stratified 2D-3D Representation: Start with 2D estimation (homography). 3D info builds on top of 2D info. Avoids a-priori model selection.

Dense 3D Reconstruction (Plane+Parallax) Epipolar geometry in this case reduces to estimating the epipoles. Everything else is captured by the homography. Original sequence Plane-aligned sequence Recovered shape

Dense 3D Reconstruction (Plane+Parallax) Original sequence Plane-aligned sequence Recovered shape

Dense 3D Reconstruction (Plane+Parallax) Original sequence Plane-aligned sequence Epipolar geometry in this case reduces to estimating the epipoles. Everything else is captured by the homography. Recovered shape

P+P Correspondence Estimation 1. Eliminating Aperture Problem Brightness Constancy constraint Epipolar line p epipole The intersection of the two line constraints uniquely defines the displacement.

Multi-Frame vs. 2-Frame Estimation 1. Eliminating Aperture Problem Brightness Constancy constraint another epipole other epipolar line Epipolar line p epipole The other epipole resolves the ambiguity ! The two line constraints are parallel ==> do NOT intersect