數位三維視訊 楊 家 輝 Jar-Ferr Yang 電腦與通信工程研究所 電機工程學系 國立成功大學 Institute of Computer and Communication Engineering Department of Electrical Engineering National Cheng Kung University, Tainan, Taiwan Stereo Matching for Depth Estimation Digital 3D Video: Chapter 4

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-2 What is Stereo Matching? Department of Electrical Engineering, Institute of Computer and Communication Engineering Simple Concept of Stereo Matching : from two camera images to estimate disparity (depth) map with: Same object with same disparity Segmentation Calculate correspond pixels similarity (color & geographic distance) Occlusion handling Refinement

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-3 Relationship of Stereo Cameras 3 : Baseline : Depth : Focal length Left view Right view

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-4 Relationship Between Depth and Disparity 4 Right view Left camera Right camera Left view : Baseline : Depth : Focal length

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-5 Related Work Global Approaches –Energy minimization process (graphic cut, Brief Propagation, Dynamic Programming, Cooperative) –Per-processing –Accurate but slow Local Approaches –A local support region with winner-take-all decision –Fixed or Adaptive Support Weight –Fast but inaccurate.

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-6 Concept of Stereo Matching 6 Boundaries ambiguities Seimi-global stereo matching Left image Right image Depth map

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-7 An Example of Stereo Matching 7 Raw Matching Cost Cost Aggregation Left ImageRight Image Winner-takes-all Disparity Map Refinement

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-8 Raw Matching Cost (Census) Census Features: : bit-wise catenation : intensity of central pixel p : intensity of surrounding pixel q corresponding to pixel p in square N(p) : square window : auxiliary function Left Window Right Window Census Raw Matching Cost: Hamming Distance where Hamming denotes the number of positions at which the corresponding bits are different [2]. = 4

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-9 Cost Aggregation: Cross-based Method 9 pq : the maximum arm length : indicator function : color threshold : RGB color channel : the intensity of color band : the left, right, up and bottom arm length of the cross. : the number of support pixels in

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-10 Winner-takes-all (WTA) 10 : the disparity range

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-11 General Stereo Matching Techniques Stereo matching techniques could compute the depth from two or more images captured by different location cameras. Involving Three Primitive Problems: –Calibrating camera positions. –Finding all corresponding points (hardest part) –Computing depth or surfaces. It is a difficulty problem for general stereo matching algorithms

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-12 Stereo Vision Triangulate on two images of the same point to recover the depth: –Feature matching across views –Calibrated cameras LeftRight baseline Matching correlation windows across scan lines depth

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-13 Pixel Matching For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This leaves too much ambiguity, so: Improvement: match windows

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-14 Correspondence Using Correlation LeftRight SSD error disparity LeftRight scanline

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-15 Sum of Squared (Pixel) Differences LeftRight

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-16 Image Normalization Even when the cameras are identical models, there can be differences in gain and sensitivity. For these reason and more, it is a good idea to normalize the pixels in each window:

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-17 Block Images as Vectors LeftRight row 1 row 2 row 3 “Unwrap” image to form vector, using raster scan order Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length. row 2

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-18 Image Metrics Sum of Squared Differences (Normalized) Normalized Correlation

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-19 Window Size W = 3W = 20 Effect of window size Some approaches have been developed to use an adaptive window size (try multiple sizes and select best match)

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-20 Stereo Testing and Comparisons Ground truthScene D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two- Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-21 Scharstein and Szeliski Stereo Testing and Comparisons

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-22 Results with Window Correlation Window-based matching (best window size) Ground truth

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-23 Results with Better Method State of the art : Graph cutsGround truth

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-24 Stereo Correspondences …… Left scanlineRight scanline

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-25 Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-26 Search Over Correspondences Three cases: –Sequential – add cost of match (small if intensities agree) –Occluded – add cost of no match (large cost) –Disoccluded – add cost of no match (large cost) Left scanline Right scanline Occluded Pixels Disoccluded Pixels

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-27 Stereo Matching by Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-28 Dynamic Programming Efficient algorithm for solving sequential decision (optimal path) problems … How many paths through this trellis?

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-29 Dynamic Programming Suppose cost can be decomposed into stages: States:

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-30 Dynamic Programming Principle of Optimality for an n-stage assignment problem:

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-31 Dynamic Programming

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-32 Stereo Matching by Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-33 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching by Dynamic Programming

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-34 Scan across grid computing optimal cost for each node given its upper- left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching by Dynamic Programming

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-35 Simulation Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-36 Segmentation-based Stereo

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-37 Another Example

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-38 Results Using a Good Technique Right ImageLeft ImageDisparity

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-39 View Interpolation

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-40 Computing Correspondence Another approach is to match edges rather than windows of pixels: Which method is better? –Edges tend to fail in dense texture (outdoors) –Correlation tends to fail in smooth featureless areas

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-41 Summary of Stereo Matching Constraints: –Geometry, epipolar constraint. –Photometric: Brightness constancy, only partly true. –Ordering: only partly true. –Smoothness of objects: only partly true. Algorithms: –What you compare: points, regions, features? How you optimize: –Local greedy matches. –1D search. –2D search.

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-42 Local Approaches Flow Diagram of Local Methods

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-43 Edge Preserving filter: Remove noise and preserve structure/edge, like object consideration. Adaptive Support Weight Bilateral filter(BF) Guided filter(GF) Geodesic diffusion Arbitrary Support Region Local Approaches

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-44 Edge Preserving filter : Remove noise and preserve structure/edge, like object consideration. Adaptive Support Weight Bilateral filter(BF) Guided filter(GF) Geodesic diffusion Arbitrary Support Region Local Approaches

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-45 Local Stereo Matching Goal : To get a considerable high quality stereo matching but with low complexity and low memory, which is good for real-time applications Successive Weighted Summation (SWS) –Constant time filtering –Weighted aggregation

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-46 Cost Computation SAD = Sum of Absolute Differences What is Census? What is Hamming distance?

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-47 Census Transform X Census transform window :

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-48 Hamming Distance of Two Censuses Left image Right image Hamming Distance = XOR

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-49 Cost Aggregation

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-50 Cost Aggregation

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-51 (b)Horizontal effective weights (c)Vertical effective weights(d) 2D effective weights Cost Aggregation

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-52 (a) AW (b) Geodesic support (c) Arbitrary support region (d) Census with SAD Comparison With Other Methods

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-53 Refinement Using cross-check to detect reliable and occluded region detection ф is a constant (set to 0.1 throughout experiments)

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-54 Simulation Results (a) Linear mapping function for reliable pixels based on disparities (b) The resultant map for the left image

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-55 Disparity Variations 55 Before  After

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-56 (c) Detection of occluded and un-reliable regions (b) Without occlusion handling, bright regions correspond to small disparities Simulation Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-57 (c) Proposed occlusion handling (b) occlusion handling with no background favoring Simulation Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-58 Simulation Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-59 Selection of Weighting Parameter

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-60 Experimental Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-61 Experimental Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-62 Experimental Results Proposed = SAD + Census

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-63 Experimental Results Proposed method is the fastest method without any special hardware implementation among Top-10 local methods of the Middlebury test bench

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-64 Proposed O(1) AW Guided filter Geodesic support Arbitrary shaped cross filter Experimental Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-65 Experimental Results

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-66 Computational Time Analyses

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-67 Error Analyses

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-68 Full-Image Guided Filtering Full-ImageProposed InitializationAD + GradientSAD + Census Aggregation Refinement1.Cross checking (lowest disparity) 2.Weighted median filter 1. Cross checking (normalized disparity) 2. Median filter (background handling) Qingqing Yang, Dongxiao Li, Lianghao Wang, and Ming Zhang, “Full-Image Guided Filtering for Fast Stereo Matching”, Signal Processing Letters, IEEE

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-69 Comparison with Full-Image

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-70 Results Achieved by Full-Image

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-71 Full-Image Results SAD+ Census Results Ground Truth Proposed Versus Full-Image

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-72 Full-Image Results Ground Truth SAD+ Census Results Proposed Versus Full-Image

National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-73 Conclusion