Center for Machine Perception Department of Cybernetics Faculty of Electrical Engineering Czech Technical University in Prague Segmentation Based Multi-View Stereo Michal Jančošek, Tomáš Pajdla

/ Problem formulation : input Input Output

/ Problem formulation : output Input Output

/ Previous work  [Furukawa07] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis, CVPR 2007  prematching, growing, filtering  [Tao00] H. Tao and H.S. Sawhney. Global Matching Criterion and Color Segmentation Based Stereo, ACV 2000  using segmentation for hypothesizing the continuous parts of a scene  [Felzenszwalb04] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation, IJCV  color segmantation  [Jancosek08] Jancosek M. and Pajdla T. Effective seed generation for 3D reconstruction, CVWW 2008  optimal 3D segment orientation 34 4

/ Pipeline overview : Prematching Input Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

/ Pipeline overview : 3D segments Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

/ Pipeline overview : Filtering Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Output =

/ Pipeline overview : Final mesh construction Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output = Poisson Surface Reconstruction

/ Pipeline overview : Prematching Input Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

/ Prematching  Cameras are known (or computed [Martinec et al.])  Feature points detection (harris on a grid [Furukawa07])  Feature points are matched in each image pair (guided matching by epigeoms).  Matching is based on the NCC score of two 5x5 windows  Mutually best matches are selected  Tracks are constructed by grouping mutually best matches r = ² i ( ) = center of gravity of the points in S S

/ Pipeline overview : 3D segments Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

/ 3D segments  For each segment with some seed projections, we create an optimal 3D segment and set it as explored  Next we do a greedy searching of unexplored segments to explore more

/ Optimal 3D segment creation TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment We accept only 3D segments with the confidence above some threshold (0.6) We take only the best 3D segment according to confidence

/ Optimal 3D segment creation MNCC( )= Ф = 45 Ѳ = 90 Ф = 75 Ѳ = 120,  The goal is to find the global maximum of the criteria function k = 0

/ Optimal 3D segment creation  First, we estimate 3D segment orientation  3D segment orientation is estimated by gradient descent optimization from the best sample out of 10x10 regular orientation samples [Jancosek08]  Next, we find the optimum position of 3D segment on the ray from reference camera center for the estimated orientation

/ Greedy searching of unexplored segments TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment

/ Greedy searching of unexplored segments TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment

/ Pipeline overview : Filtering Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Output =

/ Filtering TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment The FP 3D segments are rejected when they are not supported by another 3D segments

/ Pipeline overview : Final mesh construction Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output = Poisson Surface Reconstruction

/ Results : Strecha’s evaluation

/ Results : Strecha’s evaluation

/ Results : Strecha’s evaluation

/ Results : Strecha’s evaluation

/ Results : Strecha’s evaluation

/ Results : Strecha’s evaluation

/ Results : Homogenous regions

/ Results : Homogenous regions

/ Results : Homogenous regions

/ Results : Homogenous regions

/ Results : Homogenous regions

/ Results : Homogenous regions

/ Conclusions  Advantages  Complete models  Lack of texture is explained by planes  Speed  Possible to implement on GPU  Disadvantages  Low accuracy  Future work  MRF on volume around 3D segments

/ THANK YOU FOR YOUR ATTENTION