Imaging and Depth Estimation in an Optimization Framework Avinash Kumar 200507010 International Institute of Information Technology, Hyderabad 2007 Advisor : Dr C. V. Jawahar
Thesis: Motivation Computer Vision Algorithms can be applied to real life problems only if they are FAST and ACCURATE FAST: Optimization framework required. ACCURATE: Better algorithms required.
1. Omnifocus Imaging: Completely sharp images Thesis: Objective Propose New algorithms in Optimization framework for the following Three computer vision problems 1. Omnifocus Imaging: Completely sharp images
2. Depth Estimation: Obtain 3D depth from 2D images Thesis: Objective Propose New algorithms in Optimization framework for the following Three computer vision problems 2. Depth Estimation: Obtain 3D depth from 2D images
3. Background Subtracted Imaging: Remove static/unwanted image regions Thesis: Objective Propose New algorithms in Optimization framework for the following Three computer vision problems 3. Background Subtracted Imaging: Remove static/unwanted image regions
Thesis: Outline and Contributions Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Optimization Framework Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori (MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions
Input: Given a set of Sites S and Labels L Labeling Problem Input: Given a set of Sites S and Labels L 1 2 3 4 5
Labeling Problem Problem: Optimally assign Labels to Sites 2 5 1 3 4 Total number of possible assignments C 1 3 4 Optimal assignment is encoded in an Energy Function
Optimization Framework Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori(MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions
Markov Random Fields (MRF) Given a labeling , it belongs to a MARKOV RANDOM FIELD iff the following two hold.
Optimization Framework Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori (MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions
Ensure Optimal Labeling ? A Labeling l can be realized if some Observation d is given Observation: d Unknown label: l Labeling l belongs to MRF Given d what is the optimal l ?
Bayesian Justification For optimality: labeling in Bayesian Framework Observation: d Unknown label: l Maximum A Posteriori (MAP) estimate
Bayesian Justification Prior is defined since l belongs to a Markov Random Field (MRF)
Optimization Framework Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori (MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions
Energy Function E Taking negative on MAP makes Maximization a Energy Minimization problem Separation Cost Assignment Cost
Assignment Cost = Cost of assigning label li to di Energy Function E Assignment Cost = Cost of assigning label li to di Separation Cost = Cost of assigning labels li and lj to neighboring sites
Optimization Framework Labeling Problem Markov Random Fields (MRF) Maximum a Posteriori (MAP) - MRF Labeling Energy Functions Graph Cuts for Minimizing Energy Functions
Graph Cuts for Minimization Source Graph G A cut Sink Minimum Cut on G = Local Minima of E(l) Alpha Expansion Algorithm based on Minimum Cut finds approximate Global Minima of E(l)
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Lens Imaging Lens Object Optical Axis Image
Narrow Field of View (FoV) Two Limitations Depth of Field (DoF) Narrow Field of View (FoV)
Narrow Field of View (FoV) Two Limitations Depth of Field (DoF) Narrow Field of View (FoV)
Depth of Field Sensor Plane Lens 1 CCD = 1 pixel where all rays get focused Depth of Field A Range of depths get focused at same pixel location in an image.
But, we want images where all objects are in FOCUS !! Sample Image Depth of Field But, we want images where all objects are in FOCUS !!
Something like this where ALL objects are in FOCUS !! Desired Image Something like this where ALL objects are in FOCUS !!
Narrow Field of View (FoV) Two Limitations Depth of Field (DoF) Narrow Field of View (FoV)
FOV is limited by the dimensions of Sensor Plane Field of View Gets Imaged Does not get Imaged Sensor Plane Limited Field of View 1 CCD = 1 pixel Lens FOV is limited by the dimensions of Sensor Plane
Sample Image SMALL Field of View
Desired Image LARGE Field Of View
Solving DoF and FoV Simultaneously Move the sensor plane and take multiple images These images focus different depths
Solving DoF and FoV Simultaneously Move the sensor plane and take multiple images These images focus different depths
Solving DoF and FoV Simultaneously Move the sensor plane and take multiple images These images focus different depths
Solving DoF and FoV Simultaneously For an object select the image in which it is best focused Rotate the camera about optic center to obtain larger FOV
Solving DoF and FoV Simultaneously For an object select the image in which it is best focused Rotate the camera about optic center to obtain larger FOV
A Modified Imaging System Non-Frontal Imaging CAMera (NICAM) [Ahuja’93] Conventional Camera Frontal Sensor Single depth is captured
Non-Frontal Imaging CAMera (NICAM) Modified Camera Non Frontal Sensor Multiple depths are captured simultaneously
Non-Frontal Imaging CAMera (NICAM) Rotate NICAM to get large FOV
Non-Frontal Imaging CAMera (NICAM) Rotate NICAM to get large FOV
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Calibration Required for pixel level registration of NICAM images. Tilted Sensor Plane Pan Center Required for pixel level registration of NICAM images. Pan Centering of NICAM Calculating the tilt of the sensor plane
Various Coordinate Systems Checkerboard is placed in front of the camera Camera is placed on a rotating stage Coordinate systems assigned to World, Stage and Camera
Outline Pan Centering Tilt of Lens
Outline Pan Centering Tilt of Lens
Pan Centering Align the optic center with the rotation axis. Camera is movable on the Stage Rotation Axis
Pan Centering Align the optic center with the rotation axis. Camera is movable on the Stage Rotation Axis
Pan Centering: Algorithm STEP 1: Take images of a checkerboard pattern by rotating NICAM
Pan Centering: Algorithm STEP 1: Take images of a checkerboard pattern by rotating NICAM
Pan Centering: Algorithm STEP 1: Take images of a checkerboard pattern by rotating NICAM
Pan Centering: Algorithm STEP 1: Take images of a checkerboard pattern by rotating NICAM
Pan Centering: Algorithm STEP 1: Take images of a checkerboard pattern by rotating NICAM
Pan Centering: Algorithm STEP 2: Use MATLAB Calibration toolbox to obtain extrinsic parameters of each camera position Yw Zc Yc Xc (R t) Zw Xw R = Rotation t= Translation Extrinsic Parameters (R and t): Camera coordinate system (c) in a fixed world coordinate system (w) attached to the checker board.
Pan Centering: Algorithm STEP 3: Since the camera is not pan centered, camera positions are obtained in a elliptical arc as shown below
Pan Centering: Algorithm STEP 4: Project the points on the XY plane and do circle fitting to obtain the center of the circle ≈ 8 mm Centering Error ≈ 8 mm. The camera is off center by this amount
Pan Centering: Algorithm STEP 5: Move the camera in X and Y directions on the stage by the centering error calculated above and repeat STEP 1 to 4 till very small centering errors After Convergence ≈ 0.2 mm Centering Error ≈ 0.2 mm
Outline Pan Centering Tilt of Lens
Tilt of the Lens Define the following Transformations between Coordinate Systems (CS) Transformation from Board CS to Camera CS Transformation from Board CS to World CS Transformation from World CS to Stage CS Transformation from Stage CS to Camera CS
Tilt of the Lens {R,T} = Unknown Rotation and Translation matrices [3 rotation variables] Rotation about {Xs/Xw} axis by θ angle increments Since pan centering is done, Translation matrix is 0 [3 rotation variables] Obtained from MATLAB Calibration Toolbox during Pan Centering
Tilt of the Lens Thus, we formulate the following equality : After simplification, we get and
The Error Minimization Function is Tilt of the Lens The Error Minimization Function is where, are the rotation angles associated with the rotation matrices. 6 variables, use more than 6 images to formulate an over determined set of equations. Minimize using FMINSEARCH (MatLab)
Final Rotation angles between the coordinate systems Results Final Rotation angles between the coordinate systems
Registered Images of a checkerboard Results Registered Images of a checkerboard
Conclusions A new algorithm for Pan Centering is proposed An optimization technique obtain lens tilt is given Accurate registration of NICAM images is done
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Omnifocus Imaging Extend the Depth of Field of the image Input: Given a set of multi focus images. Output: An image where EVERYTHING is in FOCUS.
Images with different depths in focus Multi focused Images Images with different depths in focus Focused Depth
Images with different depths in focus Multi focused Images Images with different depths in focus Focused Depth
Images with different depths in focus Multi focused Images Images with different depths in focus Focused Depth
Images with different depths in focus Multi focused Images Images with different depths in focus Focused Depth
Images with all depths in focus Omni focused Image Images with all depths in focus All Depths in Focus
Algorithm Outline Capture multi focus images using NICAM. Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.
Algorithm Outline Capture multi focus images using NICAM. Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.
Sensor Plane at multiple locations Imaging Optics Lens Sensor Plane at multiple locations Blur Circle Si Object D Optical Axis u f = Blurred Image g = Focused Image v
Imaging: Some Details u = Object Distance v = Image Distance F = Focal Length Si = Distance of Ith Sensor Plane R = Radius of Blur f = Focused Image g = Blurred Image D = Aperture Diameter Thin Lens Equation Blur Radius Gaussian Blur Kernel Blur Image Formation
Algorithm Outline Capture multi focus images using NICAM. Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.
Algorithm Outline Capture multi focus images using NICAM. Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.
Focus Measure Focus Measure is a metric to find the BEST FOCUSED pixel Conventional metric: Energy of Gradient Higher the Gradient --------- More Focused is the image Focused Blurred Calculate intensity gradient and assign the best focus image in which it maximizes
But there are Drawbacks! Near an edge, the gradient maximizes for defocused pixel Focused No Gradient Defocused High Gradient
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
A Generative Focus Measure INPUT: Set of multi focus images captured from NICAM. ‘3’ is blurred with varying amounts in each image
A Generative Focus Measure STEP 1: Select a pixel location p = (x, y) in image frame k. k p = (x, y) Assume I(x, y) is a focused pixel
A Generative Focus Measure STEP 2: Calculate the radius R of blur produced at location (x, y) in image frame k-d. Similarly in frame k+d. k-d k k+d k p = (x, y) From Imaging Optics, Std. Dev. for Gaussian Blur
A Generative Focus Measure STEP 3: Obtain the Gaussian Blur Kernel and blur p(x, y) to obtain new intensities p’ and p’’ for k-d and k+d frames respectively. Gaussian Blur Kernel New intensity for k-d and k+d frames
A Generative Focus Measure STEP 4: Compare p’ and p’’ with the actual intensities at (x, y) location in k-d and k+d frames. Let actual intensities be q(x, y) in k-d frame and r(x, y) in k+d frame. The criteria for comparison is
A Generative Focus Measure STEP 5: If the criteria is satisfied, k becomes a candidate frame in which p could be focused. Calculate the focus measure Due to ambiguity, there could be multiple such candidates for each pixel. (Proof in thesis) The focus measure for frame k being in focus at p = (x, y) is
Algorithm Outline Capture multi focus images using NICAM. Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Discrete Optimization To obtain smooth and fast solution, omnifocus imaging is formulated in a discrete optimization framework The Energy Minimization Function is formulated as Labels = {1,…N}, where N is the number of multi focus images Apply Graph Cuts to minimize
Algorithm Outline Capture multi focus images using NICAM. Register the multi focus images. Find the image in which a pixel is most focused. Extract the pixel and paste on a new image. Repeat the procedure for all pixels in the image. Obtain an Omni focus image.
Two Checkerboards at different depths Results: Synthetic Two Checkerboards at different depths Single Checkerboard
Results: Real Data Set1
Results: Real Data Set2
Conclusions Omnifocus Imaging is proposed in an Optimization framework New Generative Focus Measure is proposed Fast convergence since Graph cuts takes few seconds to minimize E
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Outline Depth Estimation Labeling Problem Discrete Optimization Approach Results
Outline Depth Estimation Labeling Problem Discrete Optimization Approach Results
Depth Estimation Estimate the 3D Depth of objects in a scene Input: Given a set of multi focus images. Output: Depth Map of the scene
Set of multi focus images captured from NICAM Input Set of multi focus images captured from NICAM
Outline Depth Estimation Labeling Problem Discrete Optimization Approach Results
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
It’s a Labeling Problem! Object at different depths produce a different set of multi focus images Label set is possible depth values in the 3D world
Outline Depth Estimation Labeling Problem Discrete Optimization Approach Results
Discrete Optimization Approach Cost of assigning depth label to pixel p = Focus measure at p across the set of multi focus images. Pott’s Model Apply Graph Cuts to E to obtain depth map
Outline Depth Estimation Labeling Problem Discrete Optimization Approach Results
Results 1
Results 2
Results 3
Conclusions Depth Estimation is proposed in an Optimization framework Obtain difficult to obtain smooth and sharp depth boundaries. Fast convergence since Graph cuts takes few seconds to minimize E
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Outline Background Subtraction Intermodal Train Monitoring Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison
Outline Background Subtraction Intermodal Train Monitoring Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison
Background Subtraction Background : Static/Slow moving objects Foreground : Moving objects
Gaussian Mixture Model (GMM) Intensities in an image are modeled as Mixture of K Gaussians The weights and parameters of each Gaussian are learned with time Able to model multimodal background distributions
Outline Background Subtraction Intermodal Train Monitoring Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison
Intermodal Train Monitoring Intermodal Train Video Image Frames Extraction Background Subtraction Train Velocity Estimation Train Mosaic Creation Load Types Detection Gap Length detection
Outline Background Subtraction Intermodal Train Monitoring Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Edge & Continuous Optimization STEP 1: Apply Edge Detection on each frame
Edge & Continuous Optimization STEP 2: Detect Top Edge
Edge & Continuous Optimization STEP 3: Detect Side Edges
Edge & Continuous Optimization STEP 4: GMM learning near the edges of gaps Thus, background is subtracted
Outline Background Subtraction Intermodal Train Monitoring Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison
Thesis: Outline Optimization Framework Imaging Depth Estimation Background Subtraction Omni focus Imaging Discrete Optimization Edge Based Discrete Optimization Camera Calibration Generative Focus Measure Discrete Optimization
Discrete Optimization Background Subtraction is modeled as Velocity Estimation Energy Function is given as: Label set f is unknown velocity of the train in pixel shifts per frame v = Maximum possible velocity
Pictorial Explanation At any pixel location, take a window of size w_p and cross correlate across frames with different velocity shifts v.
Data Term 0 Average out NCC over (2n+1) frames p Input Background Removed Data Cost Vs Velocity plot at p
Data Term 1 Weighted NCC over distance of a frame from the reference frame p Input Background Removed Data Cost Vs Velocity plot at p
Data Term 2 From DT1, choose the max from both sides of reference frame p Input Background Removed Data Cost Vs Velocity plot at p
Outline Background Subtraction Intermodal Train Monitoring Edge and Continuous Optimization Approach Discrete Optimization Approach Results and Comparison
Results Input Frames Velocity Map Velocity Map after Graph Cuts Background Subtracted Frames
Proposed Technique performs better than conventional techniques Comparison Input Frames Template based GMM based Proposed method Proposed Technique performs better than conventional techniques
Edge based and GMM based background subtraction techniques is proposed Conclusions Edge based and GMM based background subtraction techniques is proposed Background Subtraction for Train Monitoring is proposed in a Discrete Optimization Framework Proposed Discrete Optimization technique gives better results than existing methods.
Thank You !!