1. Center for Machine Perception Department of Cybernetics Faculty of Electrical Engineering Czech Technical University in Prague Segmentation Based Multi-View Stereo Michal Jančošek, Tomáš Pajdla jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz

2. / Problem formulation : input 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 2 Input Output

3. / Problem formulation : output 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 3 Input Output

4. / Previous work  [Furukawa07] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis, CVPR 2007  prematching, growing, filtering  [Tao00] H. Tao and H.S. Sawhney. Global Matching Criterion and Color Segmentation Based Stereo, ACV 2000  using segmentation for hypothesizing the continuous parts of a scene  [Felzenszwalb04] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation, IJCV 2004.  color segmantation  [Jancosek08] Jancosek M. and Pajdla T. Effective seed generation for 3D reconstruction, CVWW 2008  optimal 3D segment orientation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 4

5. / Pipeline overview : Prematching 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 5 Input Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

6. / Pipeline overview : 3D segments 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 6 Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

7. / Pipeline overview : Filtering 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 7 Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Output =

8. / Pipeline overview : Final mesh construction 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 8 Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output = Poisson Surface Reconstruction

9. / Pipeline overview : Prematching 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 9 Input Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

10. / Prematching  Cameras are known (or computed [Martinec et al.])  Feature points detection (harris on a grid [Furukawa07])  Feature points are matched in each image pair (guided matching by epigeoms).  Matching is based on the NCC score of two 5x5 windows  Mutually best matches are selected  Tracks are constructed by grouping mutually best matches 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 10 r = ² i ( ) = center of gravity of the points in S S

11. / Pipeline overview : 3D segments 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 11 Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output =

12. / 3D segments 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 12  For each segment with some seed projections, we create an optimal 3D segment and set it as explored  Next we do a greedy searching of unexplored segments to explore more

13. / Optimal 3D segment creation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 13 TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment We accept only 3D segments with the confidence above some threshold (0.6) We take only the best 3D segment according to confidence

14. / Optimal 3D segment creation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 14 MNCC( )= Ф = 45 Ѳ = 90 Ф = 75 Ѳ = 120,  The goal is to find the global maximum of the criteria function k = 0

15. / Optimal 3D segment creation  First, we estimate 3D segment orientation  3D segment orientation is estimated by gradient descent optimization from the best sample out of 10x10 regular orientation samples [Jancosek08]  Next, we find the optimum position of 3D segment on the ray from reference camera center for the estimated orientation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 15

16. / Greedy searching of unexplored segments 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 16 TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment

17. / Greedy searching of unexplored segments 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 17 TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment

18. / Pipeline overview : Filtering 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 18 Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Output =

19. / Filtering 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 19 TPFP TP TP – true positive FP – false positive - explored segment - unexplored segment The FP 3D segments are rejected when they are not supported by another 3D segments

20. / Pipeline overview : Final mesh construction 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 20 Input Prematching (3D seeds) Prematching (3D seeds) Hypothesizing (3D segments) Hypothesizing (3D segments) Final mesh construction Filtering (3D patches) Filtering (3D patches) Output = Poisson Surface Reconstruction

21. / Results : Strecha’s evaluation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 21

22. / Results : Strecha’s evaluation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 22

23. / Results : Strecha’s evaluation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 23

24. / Results : Strecha’s evaluation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 24

25. / Results : Strecha’s evaluation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 25

26. / Results : Strecha’s evaluation 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 26

27. / Results : Homogenous regions 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 27

28. / Results : Homogenous regions 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 28

29. / Results : Homogenous regions 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 29

30. / Results : Homogenous regions 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 30

31. / Results : Homogenous regions 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 31

32. / Results : Homogenous regions 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 32

33. / Conclusions  Advantages  Complete models  Lack of texture is explained by planes  Speed  Possible to implement on GPU  Disadvantages  Low accuracy  Future work  MRF on volume around 3D segments 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 33

34. / 34 jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz 34 THANK YOU FOR YOUR ATTENTION