Download presentation
Presentation is loading. Please wait.
Published byAlannah Warner Modified over 9 years ago
1
Extracting Depth and Matte using a Color-Filtered Aperture Yosuke Bando TOSHIBA + The University of Tokyo Bing-Yu Chen National Taiwan University Tomoyuki Nishita The University of Tokyo
2
2 Outline Background Related Work Our Method Results Conclusion
3
3 Computational Cameras Capture various scene properties –High dynamic range, high resolution, –Large field of view, reflectance, depth, –... and more With elaborate imaging devices –Camera arrays –Additional optical elements [Wilburn 2005] [Nayar 1997]
4
4 Compact Computational Cameras Small devices Simple optical elements [Ng 2005][Levin 2007]
5
5 Our Goal Capture scene properties With minimal modification to the camera
6
6 Our Goal Capture scene properties –Depth maps –Alpha mattes With minimal modification to the camera Captured imageDepthMatte
7
7 Our Goal Capture scene properties –Depth maps –Alpha mattes With minimal modification to the camera –Put color filters in a camera lens aperture Captured imageDepthMatteLens with color filters
8
8 Our Goal Capture scene properties –Depth maps –Alpha mattes With minimal modification to the camera –Put color filters in a camera lens aperture –This idea itself is not new Captured imageDepthMatteLens with color filters
9
9 Contents Background Related Work Our Method Results Conclusion
10
10 Previous Color-Filter Methods Extract (only) depth maps –With low precision –Or, a specialized flashbulb is used Spoils the visual quality of captured images [Amari 1992][Chang 2002]
11
11 Coded Aperture Patterned mask in the aperture –Changes frequency characteristics of defocus –Facilitates blur identification/removal [Levin 2007] [Veeraraghavan 2007] Lens with a mask Captured imageAmount of defocus blur ( depth)
12
12 Single-Lens Multi-View Capture Records light rays separately depending on their incident angle –Enables light field rendering [Ng 2005] [Adelson 1992] [Liang 2008][Georgeiv 2006] [Veeraraghavan 2007]
13
13 Matting Automatic matting by multiple cameras [Joshi 2006] [McGuire 2005] 3 cameras with half mirrors Array of 8 cameras Video matting
14
14 Our Method Features –Automatic depth and matte extraction –Single hand-held camera –Single shot Contributions 1. Improved depth estimation 2. Novel matting algorithm for images captured thru a color-filtered aperture
15
15 Outline Background Related Work Our Method Results Conclusion
16
16 Our Method Color-filtered aperture Depth estimation Matting
17
17 Our Method Color-filtered aperture Depth estimation Matting
18
18 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens
19
19 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Aperture part of the disassembled lens
20
20 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45
21
21 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters
22
22 Our Prototype Camera Lens Took me just a few hours to fabricate –Using a micro-screwdriver and a box cutter Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters
23
23 Captured Image
24
24 Red Plane
25
25 Green Plane
26
26 Blue Plane
27
27 Captured Image Has depth-dependent color-misalignment –NOT due to chromatic aberration
28
28 Why Do Colors Get Misaligned? Image sensor Lens Foreground object Color filters Background
29
29 Why Do Colors Get Misaligned? Image sensor Lens Foreground object Color filters Background
30
30 Why Do Colors Get Misaligned? Image sensor Lens Foreground object Color filters Background Image sensor Lens Color filters Background
31
31 Why Do Colors Get Misaligned? Image sensor Lens Foreground object Color filters Background Image sensor Lens Color filters Background
32
32 Our Method Color-filtered aperture Depth estimation Matting
33
33 Depth Estimation Our camera captures 3 views in the RGB planes – Stereo reconstruction problem Red planeGreen planeBlue plane
34
34 Depth Estimation Our camera captures 3 views in the RGB planes – Stereo reconstruction problem However, their intensities don’t match –Contribution 1: improved correspondence measure between the RGB planes Red planeGreen planeBlue plane
35
35 Original Image
36
36 Disparity = 1
37
37 Disparity = 2
38
38 Disparity = 3
39
39 Disparity = 4
40
40 Disparity = 5
41
41 Disparity = 6
42
42 When Is The Color Aligned? Disparity
43
43 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]
44
44 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]
45
45 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]
46
46 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]
47
47 Color-Alignment Measure Local color distribution of natural images tends to form a line Misalign by 1 pixel Disparity = 0Disparity = 1 [Omer 2004, Levin 2006]
48
48 Color-Alignment Measure Local color distribution of natural images tends to form a line Misalign by 1 pixel Disparity = 0Disparity = 1 [Omer 2004, Levin 2006]
49
49 Color-Alignment Measure Local color distribution of natural images tends to form a line Misalign by 3 pixels Disparity = 0Disparity = 1Disparity = 3 [Omer 2004, Levin 2006]
50
50 Color-Alignment Measure Local color distribution of natural images tends to form a line Misalign by 3 pixels Disparity = 0Disparity = 1Disparity = 3 [Omer 2004, Levin 2006]
51
51 Color-Alignment Measure Disparity = 0Disparity = 1Disparity = 3 Variances along the principal axes (eigenvalues) Variances along the RGB axes
52
52 Color-Alignment Measure Disparity = 0Disparity = 1Disparity = 3 Variances along the principal axes (eigenvalues) Variances along the RGB axes L = 0.003L = 0.11L = 0.39
53
53 Depth Estimation Solve for the disparity that makes the color-alignment measure minimum Captured image Pixel-wise estimates (intensity depth)
54
54 Depth Estimation Solve for the disparity that makes the color-alignment measure minimum With smoothness constraints –Graph-cut optimization After graph-cuts [Boykov 2001] Captured image Pixel-wise estimates (intensity depth)
55
55 Our Method Color-filtered aperture Depth estimation Matting
56
56 Matting Problem of estimating foreground opacity Input image Matte Foreground color Background color
57
57 Trimap Assigns each pixel to one of 3 labels –Strictly foreground ( = 1) –Strictly background ( = 0) –Unknown ( to be computed) Captured imageTrimap strictly back- ground strictly fore- ground k u n n o w n
58
58 Trimap Assigns each pixel to one of 3 labels –Strictly foreground ( = 1) –Strictly background ( = 0) –Unknown ( to be computed) Generated from the depth map Captured imageTrimap strictly back- ground strictly fore- ground k u n n o w n
59
59 Trimap-Based Matting Captured image Trimap
60
60 Trimap-Based Matting Errors remain where the foreground and background colors are similar Captured image Trimap [Levin 2006]
61
61 Trimap-Based Matting Errors remain where the foreground and background colors are similar –Contribution 2: matte error correction using color misalignment cues Captured image Trimap [Levin 2006]
62
62 Basic Idea Estimate foreground and background colors based on the current matte Captured image Current matte
63
63 Basic Idea Estimate foreground and background colors based on the current matte Captured image Current matte Estimated foreground color Estimated background color
64
64 Basic Idea Estimate foreground and background colors based on the current matte Detect inconsistent color misalignments Captured image Current matte Estimated foreground color Estimated background color
65
65 Synthetic Toy Example Ground truth matte Synthesized input image
66
66 Synthetic Toy Example Background Ground truth matte Synthesized input image Camera
67
67 Synthetic Toy Example Foreground Background Camera Ground truth matte Synthesized input image
68
68 Synthetic Toy Example Hard example –Similar foreground and background colors ForegroundBackground Foreground Background
69
69 Synthetic Toy Example Hard example –Similar foreground and background colors But solvable –Color misalignment cues from ‘x’ textures ForegroundBackground Foreground Background
70
70 Trimap-Based Matting Trimap Input image Foreground Background
71
71 Trimap-Based Matting Trimap Estimated matteInput image Foreground Background
72
72 Trimap-Based Matting Trimap Estimated matteInput image Ground truth Foreground Background
73
73 Matting Algorithm Input image Current matte Foreground Background
74
74 Matting Algorithm Input image Current matte Estimated foreground color Estimated background color Foreground Background
75
75 Matting Algorithm Input image Current matte Estimated foreground color Estimated background color Detected color-misalignment
76
76 Matting Algorithm Input image Current matte Estimated foreground color Estimated background color Detected color-misalignment Detected color-alignment
77
77 Matting Algorithm Input image Current matte Estimated foreground color Estimated background color Detected color-misalignment Detected color-alignment
78
78 Matting Algorithm Input image Final matte Foreground Background
79
79 Matting Algorithm Input image Final matte Ground truth Foreground Background
80
80 Outline Background Related Work Our Method Results Conclusion
81
81 Results of Depth & Matte Extraction Captured image DepthMatte
82
82 Comparison: Depth Estimation With the previous color-filter methods –Local estimation to show raw performance [Amari 1992] [Chang 2002] Captured Our method
83
83 Comparison: Matting With the trimap-based matting methods –The trimaps were generated by our method [Levin 2006] [Wang 2007]Our method Trimap
84
84 Comparison with Ground Truth Mattes Ground truth Composite image (color-aligned) Composite image (color-misaligned) Our method [Levin 2006][Wang 2007]
85
85 Image Editing Image composition Color-alignment reconstruction Novel view synthesis Refocusing Video matting
86
86 Example 1: Composition Captured image Composite Matte Different Background Different Background
87
87 Example 2: Color-Alignment Reconst. Matte Foreground color Background color Cancel the estimated misalignment Re-compose Cancel the estimated misalignment
88
88 Reconstructed Image
89
89 Captured Image
90
90 Examples 3 & 4: View/Focus Synthesis
91
91 Example 5: Video Matting
92
92 Outline Background Related Work Our Method Results Conclusion
93
93 Conclusion Automatic depth and matte extraction using a color-filtered aperture –Improved depth estimation –Novel matting algorithm
94
94 Conclusion Automatic depth and matte extraction using a color-filtered aperture –Improved depth estimation –Novel matting algorithm Easy-to-use computational photography –Put color filters in a camera lens –Take a single photo with a hand-held camera
95
95 Conclusion Automatic depth and matte extraction using a color-filtered aperture –Improved depth estimation –Novel matting algorithm Easy-to-use computational photography –Put color filters in a camera lens –Take a single photo with a hand-held camera Limitation –Entirely red objects cannot be handled
96
96 Thank You!! Any questions? Acknowledgments –Takeshi Naemura –Yusuke Iguchi –Takuya Saito –Johanna Wolf –Zoltan Szego –Paulo Silva –Saori Horiuchi
97
97 Other Possible Filter Arrangements Our prototype Combination with coded aperture Multi-aperture More light- efficient Even more light- efficient but with lower depth resolution Higher depth resolution Simple horizontal arrangement Arbitrary shapes
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.