Download presentation
Presentation is loading. Please wait.
1
What is the Range of Surface Reconstructions from a Gradient Field?
Amit Agrawal, Ramesh Raskar and Rama Chellappa Presented by: Ramesh Raskar* University of Maryland, MD, USA * Mitsubishi Electric Research Labs (MERL), MA, USA Source Code:
2
? Overview Goal: Reconstruct height field from a gradient field
Challenges Handle noise and outliers Tradeoff: Preserve features or smooth Provide knobs for control ? X Gradient Y Gradient Height Field
3
Choices of Reconstructions
Noisy Gradients Height Field Least Sq Solution Robust Solution MSE = 10.81 MSE = 2.26
4
A Range of Reconstructions
1. Generalized Solution Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Binary weights Continuous weights Affine Transformation Scaling 2. Linear transformation of gradient vectors
5
Source of Gradients: Image Manipulation
Grad X New Grad X 2D Integration Gradient Processing Grad Y New Grad Y Forward Difference Modified Gradients (Non-integrable Gradients) Image
6
Source of Gradients: Photometric Stereo
Mozart Images under varying illumination X Gradient Y Gradient (Non-integrable Gradients) 2D Integration Robust Solution MSE=2339.2 MSE=373.72
7
Gradient Field Integration
Photometric Stereo Phase unwrapping, Mesh smoothing Retinex (Horn’74) HDR compression (Fattal’02) Poisson image editing (Perez’04) Poisson matting (Sun’04) Image fusion (Raskar’04), Multi-flash Camera (Raskar’04) Image Stitching (Levin’04) Flash/No-flash (Agrawal’05) ..
8
Space of Solutions Non-Integrable Gradient Field Robust solutions
Residual Field Least Squares solution Subspace of Integrable Gradient Fields Space of All Gradient Fields
9
Example Application: Photometric Stereo
How to obtain heights from gradient field? Preserve features during integration Handle noise and outliers Z p q ? X Gradient Y Gradient Height Field
10
Integrability Gradient field of solution Z should be integrable
Curl(Zx,Zy) (loop integrals) should be zero Noisy estimated gradient field is non-integrable Curl(p,q) ≠ 0
11
Common approaches to handle Integrability
Least Square: Minimize error between Estimated gradients (p,q) and Gradients of solution Z [Horn et al. IJCV’90], [Simchony et al. PAMI’90] Solution: Poisson equation Laplacian Divergence
12
Integrability: Reconstruction using Basis Functions
Project non-integrable gradients onto basis functions Fourier: Frankot-Chellappa (PAMI’88) Cosine: Georghiades (PAMI’01) Redundant basis: Kovesi (ICCV’05)
13
Limitations of Least Squares Approaches
Lack of robustness Smooth solution rather than feature preserving Cannot handle outliers Height Field Least Sq Solution Robust Solution MSE = 10.81 MSE = 2.26
14
Key Ideas All gradients are not required for integration
Replace gradients by functions of gradients
15
Approach Transform input and output gradients Poisson equation
Unknown (Output) Gradients of Height Field Z Estimated (Input) Gradients
16
Approach Transform input and output gradients Poisson equation
New Generalized Equation Functions of Output Gradients Functions of Input Gradients
17
Approach fi Transform input and output gradients Poisson equation
New Generalized Equation Knobs for control: functions f1, f2, f3, f4 fi
18
Least Squares Poisson Solution
Isotropic Anisotropic Alpha-Surface M-estimator, Regularization Diffusion Poisson Solver Binary weights Continuous weights Affine Transformation Scaling No Change f1(Zx,Zy) f2(Zx,Zy) f3(p,q) f4(p,q) Poisson solver Zx Zy p q
19
A Range of Solutions by Transforming Gradients
Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Binary weights Continuous weights Affine Transformation Scaling fi Linear Transformation of Gradient Vectors using fi
20
A Range of Solutions by Transforming Gradients
Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Binary weights Continuous weights Affine Transformation Scaling fi f1(Zx,Zy) f2(Zx,Zy) f3(p,q) f4(p,q) Poisson solver Zx Zy p q 1. -surface bxZx byZy bxp byq 2. M-estimators wxZx wyZy wxp wyq 3. Regularization wxZx wyZy p q 4. Diffusion d11Zx+d12Zy d12Zx+d22Zy d11p+d12q d12p+d22q
21
A Range of Solutions Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Binary weights Continuous weights Affine Transformation Diffusion Scaling 1. Robust Estimation by ignoring outliers in gradients
22
1. -Surface: Binary Weights
Classifying gradients as inliers/outliers Based on tolerance Graph Analogy 2D grid as a planar graph Nodes correspond to height values Edges correspond to gradient values Given edges compute nodes [Agrawal et al ICCV 2005]
23
All Gradients are not required
Minimal set is the spanning tree of the graph All nodes can be reached via spanning tree N2 -1 edges in spanning tree for N2 nodes Dimensionality of gradient field ~= 2N2 Dimensionality of solution space = N2 - 1
24
- Surface Start with minimum spanning tree Iterate
Minimal set of (N2-1) edges Iterate Integrate height field using (inlier) edges Compute gradient edges of the solution Find inlier edges using tolerance
25
Tolerance d d < d
26
= 0 Minimum Spanning Tree Unique Solution Robust to outliers
Choice of = 0 Minimum Spanning Tree Unique Solution Robust to outliers >> Overconstrained: All Gradients Least Squares Solution Smooth
27
A Range of Solutions 2. Robust Estimation by weighting gradients
Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Binary weights Continuous weights Affine Transformation Diffusion Scaling 2. Robust Estimation by weighting gradients
28
2. Continuous Weights Solution
M-estimators Formulated as iterative re-weighted least squares wx, wy are weights applied to gradients Least squares Huber Influence function
29
3. Regularization Add edge-preserving term using function Ф
Solved iteratively Estimate weights wx, wy using Z Update Z using weights We use
30
A Range of Solutions Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Binary weights Continuous weights Affine Transformation Diffusion Scaling 4. Robust Estimation by affine transformation of gradients
31
4. Affine transformation of Gradient Field
General Transform of Gradients Vectors Matrix D2x2 is a field of tensors Estimated using the input gradient field (p,q) Similar to edge-preserving diffusion tensor Image Restoration [Weickert’96]
32
4. Affine transformation of Gradient Field
Transformed gradient vectors Non-iterative, sparse linear system f1(Zx,Zy) f2(Zx,Zy) f3(p,q) f4(p,q) d11Zx+d12Zy d12Zx+d22Zy d11p+d12q d12p+d22q
33
Mozart: Sharp Features
MSE=2339.2 MSE=1316.6 MSE=219.72 MSE=359.12 MSE=806.85 MSE=373.72
34
Vase: Smooth Object MSE=294.5 MSE=239.6 MSE=22.2 MSE=15.14 MSE=164.98
35
Flowerpot
36
Generalized Solution using Gradient Vector Transformation
Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Binary weights Continuous weights Affine Transformation Diffusion Scaling Ground Truth Least Squares Feature Preserving
37
Reconstruction from Gradients
Gradient Field integration as robust estimation Generalized equation Isotropic case: Least-square solution Anisotropic transforms: Range of feature preserving solutions Knobs for meaningful control Future Directions Automatic choice of anisotropic function Nth order error functions Using control/seed points information Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Binary weights Continuous weights Affine Transformation Diffusion Scaling Source Code:
38
Extra Slides
39
Toy Example
40
Space of All solutions Equations H Unknowns W
For a H*W grid, all solutions lie in a subspace of dimension HW-1 H W Equations W-1 loops for each row, H-1 loops for each column Total number of independent equations N = (H-1)(W-1) = HW – H –W + 1 Unknowns W-1 x gradients in H rows, H-1 y gradients in W columns Total unknowns M = (W-1)*H + (H-1)*W = 2HW – H - W
41
Linear System Stack all equation to form Ax = b
b contains negative of the curl values x contains all the residual gradient values (pε,qε) A is a sparse matrix with each row having 4 non-zero elements Under-constrained system. A has null space of dimension M-N = H*W-1 xp = particular solution Axp = b Xh = homogeneous solution Axh= 0
42
Example Application: Photometric Stereo
Multiple images, varying illumination Obtain surface gradient field from images Lambertian reflectance model Images X Gradient Y Gradient p q
43
Mozart: Sharp Features
MSE=2339.2 MSE=1316.6 MSE=219.72 MSE=359.12 MSE=806.85 MSE=373.72
44
Vase: Smooth Object MSE=294.5 MSE=239.6 MSE=22.2 MSE=15.14 MSE=164.98
45
Flowerpot
46
= 0 Minimum Spanning Tree Unique Solution Robust to outliers
Choice of = 0 Minimum Spanning Tree Unique Solution Robust to outliers >> Overconstrained: All Gradients Least Squares Solution Smooth Toy Example
47
Choosing spanning tree
Assign weights to edges based on gradients Find minimum spanning tree (MST) Compute based on variance
48
Estimated Heights with -tolerance
Face Input Images Estimated Heights with -tolerance
49
3. Regularization Add edge-preserving term using function Ф
Solved iteratively Estimate weights wx, wy using Z Update Z using weights We use
Similar presentations
