1. Stereo Matching Segment-based Belief Propagation Iolanthe II racing in Waitemata Harbour

2. Segmentation #1 on the Middlebury rankings Klaus, Sormann and Karner, “Segment-based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure”, ICPR (3) 2006: 15-18 ICPR (3) 2006 Basic idea Depth (disparity) changes should occur at region boundaries in the image so Segment the image and match the patches

3. Algorithm Pair of rectified images Extract homogenous regions in reference image Apply local window based matching Extract set of disparity planes Approximate optimal disparity plane assignment Disparity map

4. Step 1 - Segmentation Decompose reference image  Regions of homogenous colour | grey-scale Assume Disparity values vary smoothly within these regions Note: Possibly sloping planes Fronto-planar assumption not required! Mean-shift colour segmentation used Gradient descent search for maxima in a density function over a high dimensional feature space Feature space: spatial coordinates + associated attributes (including edge information)

5. Step 2 – Local matching Local matching in pixel domain Window based correlation (SSD or SAD) Gradient or non-parametric matching algorithms possible  Better tolerate gain and offset changes KSK use self-adapting combination score C(x,y,d) = (1 -  ) C SAD (x,y,d) +  C grad (x,y,d) where C SAD (x,y,d) =  i,j  N | I L (i,j) – I R (i+d,j) | C grad (x,y,d) =  i,j  N |  x I L (i,j) -  x I R (i+d,j) | +  i,j  N |  y I L (i,j) -  y I R (i+d,j) | L  R Difference in Intensity L  R Difference in Horizontal Gradient L  R Difference in Vertical Gradient

6. Step 2 – Local matching KSK use self-adapting combination score C(x,y,d) = (1 -  ) C SAD (x,y,d) +  C grad (x,y,d) where C SAD (x,y,d) =  i,j  N | I L (i,j) – I R (i+d,j) | C grad (x,y,d) =  i,j  N |  x I L (i,j) -  x I R (i+d,j) | +  i,j  N |  y I L (i,j) -  y I R (i+d,j) | N(x,y) is a 3  3 window Nx(x,y) is a 3  2 window; Ny(x,y) is a 2  3 window Weighting,  chosen Winner-take-all – choose disparity with lowest cost + Maximising number of reliable correspondences filtered out by L  R crosscheck L  R Difference in Intensity L  R Difference in Horizontal Gradient L  R Difference in Vertical Gradient

7. Step 2 – Local matching Weighting,  chosen Winner-take-all – choose disparity with lowest cost + Maximising number of reliable correspondences filtered out by L  R crosscheck Normalized dissimilarity measure used Reliable correspondences used to estimate signal- noise ratio (SNR) Because of normalization, a fixed truncation threshhold is set just above the noise level  Robust matching score

8. Step 3 – Disparity plane estimation Allow sloping planes Abandon fronto-planar surface assumption! Disparity plane estimation Specify the plane with c 1, c 2, c 3 : d = c 1 x + c 2 y + c 3  Large number of possible planes  Use only reliable disparities Decompose fitting problem: Decompose fitting problem: Fit Horizontal and vertical slant separately Find  d/  x (  d/  y ) for each pixel Distribution of derivatives  d/  x (  d/  y ) is convolved with Gaussian kernel to determine mean  d/  x Slant in the centre of a segment then determined by finding distribution of centre disparities for each reliable point as before

9. Step 3 – Disparity plane estimation Allow sloping planes Abandon fronto-planar surface assumption! Estimate disparity planes Specify a plane with c 1, c 2, c 3 : d = c 1 x + c 2 y + c 3  Large number of possible planes  Use only reliable disparities Decompose fitting problem: Fit Horizontal and vertical slant separately Find  d/  x (  d/  y ) for each pixel Distribution of derivatives  d/  x (  d/  y ) is convolved with Gaussian kernel to determine mean  d/  x Slant in the centre of a segment then determined from distribution of centre disparities for each reliable point as before

10. Step 4 – Disparity plane refinement Increase the accuracy of the disparity plane Group regions that belong to the same disparity plane Calculate a matching cost For the plane, P, fitted to segment, S : C SEG (S,P) =  (x,y)  S C(x,y,d) Disparity plane with the minimum matching cost is assigned to each segment Segments assigned to that disparity plane are grouped Repeat for all grouped segments For each pixel in segment, S Mismatch cost for disparity, d

11. Step 5 – Disparity plane assignment Search for optimal segment  disparity place assignment Formulate as an energy minimization problem Find a labelling, f, which matches a segment, s  R, to a plane, f(s) in D ‘Energy’ for labelling f is E(f) = E data (f) + E smooth (f) where E data (f) =  s  R C SEG (s,f(s)) and E smooth (f) =  (  (si,sj)  SN|f(si)  f(sj) ) disc (s i,s j )  disc (s i,s j ) – discontinuity penalty – includes common border lengths and colour similarity S N set of adjacent segments

12. Step 5 Use loopy belief propagation to find optimal labeling with minimum energy Results: Matching – state-of-the-art! Top ‘average’ rank on the Middlebury set Sloping planes of ‘Venus’ are well handled Still has problems with edges!! See results in fig 2 of the paper Pixelization not handled? Computation cost? Not mentioned in paper! Algorithm contains many repeated steps Analysis of benefit of each one useful