Imaging and Depth Estimation in an Optimization Framework

Imaging and Depth Estimation in an Optimization Framework
Avinash Kumar International Institute of Information Technology, Hyderabad 2007 Advisor : Dr C. V. Jawahar

Thesis: Motivation Computer Vision Algorithms can be applied to real life problems only if they are FAST and ACCURATE FAST: Optimization framework required. ACCURATE: Better algorithms required.

1. Omnifocus Imaging: Completely sharp images
Thesis: Objective Propose New algorithms in Optimization framework for the following Three computer vision problems 1. Omnifocus Imaging: Completely sharp images

2. Depth Estimation: Obtain 3D depth from 2D images
Thesis: Objective Propose New algorithms in Optimization framework for the following Three computer vision problems 2. Depth Estimation: Obtain 3D depth from 2D images

3. Background Subtracted Imaging: Remove static/unwanted image regions
Thesis: Objective Propose New algorithms in Optimization framework for the following Three computer vision problems 3. Background Subtracted Imaging: Remove static/unwanted image regions

Thesis: Outline and Contributions
Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization

Thesis: Outline Optimization Framework Imaging Depth Estimation
Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization

Optimization Framework
Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori (MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions

Input: Given a set of Sites S and Labels L
Labeling Problem Input: Given a set of Sites S and Labels L 1 2 3 4 5

Labeling Problem Problem: Optimally assign Labels to Sites 2 5 1 3 4
Total number of possible assignments C 1 3 4 Optimal assignment is encoded in an Energy Function

Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori(MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions

Markov Random Fields (MRF)
Given a labeling , it belongs to a MARKOV RANDOM FIELD iff the following two hold.

Ensure Optimal Labeling ?
A Labeling l can be realized if some Observation d is given Observation: d Unknown label: l Labeling l belongs to MRF Given d what is the optimal l ?

Bayesian Justification
For optimality: labeling in Bayesian Framework Observation: d Unknown label: l Maximum A Posteriori (MAP) estimate

Bayesian Justification
Prior is defined since l belongs to a Markov Random Field (MRF)

Energy Function E Taking negative on MAP makes Maximization a Energy Minimization problem Separation Cost Assignment Cost

Assignment Cost = Cost of assigning label li to di
Energy Function E Assignment Cost = Cost of assigning label li to di Separation Cost = Cost of assigning labels li and lj to neighboring sites

Graph Cuts for Minimization
Source Graph G A cut Sink Minimum Cut on G = Local Minima of E(l) Alpha Expansion Algorithm based on Minimum Cut finds approximate Global Minima of E(l)

Lens Imaging Lens Object Optical Axis Image

Narrow Field of View (FoV)
Two Limitations Depth of Field (DoF) Narrow Field of View (FoV)

Depth of Field Sensor Plane Lens 1 CCD = 1 pixel where all rays get focused Depth of Field A Range of depths get focused at same pixel location in an image.

But, we want images where all objects are in FOCUS !!
Sample Image Depth of Field But, we want images where all objects are in FOCUS !!

Something like this where ALL objects are in FOCUS !!
Desired Image Something like this where ALL objects are in FOCUS !!

Narrow Field of View (FoV)
Two Limitations Depth of Field (DoF) Narrow Field of View (FoV)

FOV is limited by the dimensions of Sensor Plane
Field of View Gets Imaged Does not get Imaged Sensor Plane Limited Field of View 1 CCD = 1 pixel Lens FOV is limited by the dimensions of Sensor Plane

Sample Image SMALL Field of View

Desired Image LARGE Field Of View

Solving DoF and FoV Simultaneously
Move the sensor plane and take multiple images These images focus different depths

Solving DoF and FoV Simultaneously
For an object select the image in which it is best focused Rotate the camera about optic center to obtain larger FOV

A Modified Imaging System
Non-Frontal Imaging CAMera (NICAM) [Ahuja’93] Conventional Camera Frontal Sensor Single depth is captured

Non-Frontal Imaging CAMera (NICAM)
Modified Camera Non Frontal Sensor Multiple depths are captured simultaneously

Non-Frontal Imaging CAMera (NICAM)
Rotate NICAM to get large FOV

Calibration Required for pixel level registration of NICAM images.
Tilted Sensor Plane Pan Center Required for pixel level registration of NICAM images. Pan Centering of NICAM Calculating the tilt of the sensor plane

Various Coordinate Systems
Checkerboard is placed in front of the camera Camera is placed on a rotating stage Coordinate systems assigned to World, Stage and Camera

Outline Pan Centering Tilt of Lens

Pan Centering Align the optic center with the rotation axis. Camera is movable on the Stage Rotation Axis

Pan Centering: Algorithm
STEP 1: Take images of a checkerboard pattern by rotating NICAM

STEP 2: Use MATLAB Calibration toolbox to obtain extrinsic parameters of each camera position Yw Zc Yc Xc (R t) Zw Xw R = Rotation t= Translation Extrinsic Parameters (R and t): Camera coordinate system (c) in a fixed world coordinate system (w) attached to the checker board.

STEP 3: Since the camera is not pan centered, camera positions are obtained in a elliptical arc as shown below

STEP 4: Project the points on the XY plane and do circle fitting to obtain the center of the circle ≈ 8 mm Centering Error ≈ 8 mm. The camera is off center by this amount

STEP 5: Move the camera in X and Y directions on the stage by the centering error calculated above and repeat STEP 1 to 4 till very small centering errors After Convergence ≈ 0.2 mm Centering Error ≈ 0.2 mm

Outline Pan Centering Tilt of Lens

Tilt of the Lens Define the following Transformations between Coordinate Systems (CS) Transformation from Board CS to Camera CS Transformation from Board CS to World CS Transformation from World CS to Stage CS Transformation from Stage CS to Camera CS

Tilt of the Lens {R,T} = Unknown Rotation and Translation matrices [3 rotation variables] Rotation about {Xs/Xw} axis by θ angle increments Since pan centering is done, Translation matrix is 0 [3 rotation variables] Obtained from MATLAB Calibration Toolbox during Pan Centering

Tilt of the Lens Thus, we formulate the following equality :
After simplification, we get and

The Error Minimization Function is
Tilt of the Lens The Error Minimization Function is where, are the rotation angles associated with the rotation matrices. 6 variables, use more than 6 images to formulate an over determined set of equations. Minimize using FMINSEARCH (MatLab)

Final Rotation angles between the coordinate systems
Results Final Rotation angles between the coordinate systems

Registered Images of a checkerboard
Results Registered Images of a checkerboard

Conclusions A new algorithm for Pan Centering is proposed
An optimization technique obtain lens tilt is given Accurate registration of NICAM images is done

Omnifocus Imaging Extend the Depth of Field of the image
Input: Given a set of multi focus images. Output: An image where EVERYTHING is in FOCUS.

Images with different depths in focus
Multi focused Images Images with different depths in focus Focused Depth

Images with all depths in focus
Omni focused Image Images with all depths in focus All Depths in Focus

Algorithm Outline Capture multi focus images using NICAM.
Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.

Sensor Plane at multiple locations
Imaging Optics Lens Sensor Plane at multiple locations Blur Circle Si Object D Optical Axis u f = Blurred Image g = Focused Image v

Imaging: Some Details u = Object Distance v = Image Distance
F = Focal Length Si = Distance of Ith Sensor Plane R = Radius of Blur f = Focused Image g = Blurred Image D = Aperture Diameter Thin Lens Equation Blur Radius Gaussian Blur Kernel Blur Image Formation

Focus Measure Focus Measure is a metric to find the BEST FOCUSED pixel
Conventional metric: Energy of Gradient Higher the Gradient  More Focused is the image Focused Blurred Calculate intensity gradient and assign the best focus image in which it maximizes

But there are Drawbacks!
Near an edge, the gradient maximizes for defocused pixel Focused No Gradient Defocused High Gradient

A Generative Focus Measure
INPUT: Set of multi focus images captured from NICAM. ‘3’ is blurred with varying amounts in each image

STEP 1: Select a pixel location p = (x, y) in image frame k. k p = (x, y) Assume I(x, y) is a focused pixel

STEP 2: Calculate the radius R of blur produced at location (x, y) in image frame k-d. Similarly in frame k+d. k-d k k+d k p = (x, y) From Imaging Optics, Std. Dev. for Gaussian Blur

STEP 3: Obtain the Gaussian Blur Kernel and blur p(x, y) to obtain new intensities p’ and p’’ for k-d and k+d frames respectively. Gaussian Blur Kernel New intensity for k-d and k+d frames

STEP 4: Compare p’ and p’’ with the actual intensities at (x, y) location in k-d and k+d frames. Let actual intensities be q(x, y) in k-d frame and r(x, y) in k+d frame. The criteria for comparison is

STEP 5: If the criteria is satisfied, k becomes a candidate frame in which p could be focused. Calculate the focus measure Due to ambiguity, there could be multiple such candidates for each pixel. (Proof in thesis) The focus measure for frame k being in focus at p = (x, y) is

Discrete Optimization
To obtain smooth and fast solution, omnifocus imaging is formulated in a discrete optimization framework The Energy Minimization Function is formulated as Labels = {1,…N}, where N is the number of multi focus images Apply Graph Cuts to minimize

Two Checkerboards at different depths
Results: Synthetic Two Checkerboards at different depths Single Checkerboard

Results: Real Data Set1

Results: Real Data Set2

Conclusions Omnifocus Imaging is proposed in an Optimization framework
New Generative Focus Measure is proposed Fast convergence since Graph cuts takes few seconds to minimize E

Outline Depth Estimation Labeling Problem
Discrete Optimization Approach Results

Depth Estimation Estimate the 3D Depth of objects in a scene
Input: Given a set of multi focus images. Output: Depth Map of the scene

Set of multi focus images captured from NICAM
Input Set of multi focus images captured from NICAM

It’s a Labeling Problem!
Object at different depths produce a different set of multi focus images Label set is possible depth values in the 3D world

Discrete Optimization Approach
Cost of assigning depth label to pixel p = Focus measure at p across the set of multi focus images. Pott’s Model Apply Graph Cuts to E to obtain depth map

Results 1

Results 2

Results 3

Conclusions Depth Estimation is proposed in an Optimization framework
Obtain difficult to obtain smooth and sharp depth boundaries. Fast convergence since Graph cuts takes few seconds to minimize E

Outline Background Subtraction Intermodal Train Monitoring
Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison

Background Subtraction
Background : Static/Slow moving objects Foreground : Moving objects

Gaussian Mixture Model (GMM)
Intensities in an image are modeled as Mixture of K Gaussians The weights and parameters of each Gaussian are learned with time Able to model multimodal background distributions

Intermodal Train Monitoring
Intermodal Train Video Image Frames Extraction Background Subtraction Train Velocity Estimation Train Mosaic Creation Load Types Detection Gap Length detection

Edge & Continuous Optimization
STEP 1: Apply Edge Detection on each frame

STEP 2: Detect Top Edge

STEP 3: Detect Side Edges

STEP 4: GMM learning near the edges of gaps Thus, background is subtracted

Discrete Optimization
Background Subtraction is modeled as Velocity Estimation Energy Function is given as: Label set f is unknown velocity of the train in pixel shifts per frame v = Maximum possible velocity

Pictorial Explanation
At any pixel location, take a window of size w_p and cross correlate across frames with different velocity shifts v.

Data Term 0 Average out NCC over (2n+1) frames p Input
Background Removed Data Cost Vs Velocity plot at p

Data Term 1 Weighted NCC over distance of a frame from the reference frame p Input Background Removed Data Cost Vs Velocity plot at p

Data Term 2 From DT1, choose the max from both sides of reference frame p Input Background Removed Data Cost Vs Velocity plot at p

Results Input Frames Velocity Map Velocity Map after Graph Cuts
Background Subtracted Frames

Proposed Technique performs better than conventional techniques
Comparison Input Frames Template based GMM based Proposed method Proposed Technique performs better than conventional techniques

Edge based and GMM based background subtraction techniques is proposed
Conclusions Edge based and GMM based background subtraction techniques is proposed Background Subtraction for Train Monitoring is proposed in a Discrete Optimization Framework Proposed Discrete Optimization technique gives better results than existing methods.

Thank You !!

Imaging and Depth Estimation in an Optimization Framework

Similar presentations

Presentation on theme: "Imaging and Depth Estimation in an Optimization Framework"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Imaging and Depth Estimation in an Optimization Framework

Similar presentations

Presentation on theme: "Imaging and Depth Estimation in an Optimization Framework"— Presentation transcript:

Similar presentations

About project

Feedback