1. Multi-view Stereo via Volumetric Graph-cuts
George Vogiatzis
Roberto Cipolla Cambridge Univ. Engineering Dept.
Philip H. S. Torr
Department of Computing Oxford Brookes University

2. Multi-view Dense Stereo
Calibrated images of Lambertian scene
3D model of scene

3. Multi-view Dense Stereo
Two main approaches
Volumetric
Disparity (depth) map
Volumetric

4. Dense Stereo reconstruction problem:
Two main approaches
Volumetric
Disparity (depth) map
Disparity-map

5. Shape representation Disparity-maps
MRF formulation – good optimisation techniques exist (Graph-cuts, Loopy BP)
MRF smoothness is viewpoint dependent
Disparity is unique per pixel – only functions represented

6. Shape representation Volumetric – e.g. Level-sets, Space carving etc.
Able to cope with non-functions
Levelsets: Local optimization
Space carving: no simple way to impose surface smoothness

7. Our approach Cast volumetric methods in MRF framework
Use approximate surface containing the real scene surface
E.g. visual hull
Benefits:
General surfaces can be represented
No depth map merging required
Optimisation is tractable (MRF solvers)
Smoothness is viewpoint independent

8. Volumetric Graph cuts for segmentation
Boykov and Jolly ICCV 2001
Volume is discretized
A binary MRF is defined on the voxels
Voxels are labelled as OBJECT and BACKGROUND
Labelling cost set by OBJECT / BACKGROUND intensity statistics
Compatibility cost set by intensity gradient

9. Volumetric Graph cuts for stereo
Challenges:
What do the two labels represent
How to define cost of setting them
How to deal with occlusion
Interactions between distant voxels

10. Volumetric Graph cuts (x) 1. Outer surface
2. Inner surface (at constant offset)
(x)
3. Discretize middle volume
4. Assign photoconsistency cost to voxels

11. Volumetric Graph cuts
Source
Sink

12. Volumetric Graph cuts S cut  3D Surface S Cost of a cut   (x) dS
Source
[Boykov and Kolmogorov ICCV 2001] S S
Sink

13. Volumetric Graph cuts
Minimum cut  Minimal 3D Surface under photo-consistency metric
Source
[Boykov and Kolmogorov ICCV 2001]
Sink

14. Photo-consistency Occlusion 1. Get nearest point on outer surface
2. Use outer surface for occlusions
2. Discard occluded views

15. Photo-consistency
Occlusion
Self occlusion

16. Photo-consistency
Occlusion
Self occlusion

17. Photo-consistency Occlusion
threshold on angle between normal and viewing direction
threshold= ~60 N

18. Photo-consistency Score Normalised cross correlation
Use all remaining cameras pair wise
Average all NCC scores
Score

19. Photo-consistency Score  = 1 - exp( -tan2[(C-1)/4] / 2 )
Average NCC = C
Voxel score
 = 1 - exp( -tan2[(C-1)/4] / 2 )
Score
0    1
 = 0.05 in all experiments

20. Example

21. Example - Visual Hull

22. Example - Slice

23. Example - Slice with graphcut

24. Example – 3D

25. Protrusion problem ‘Balooning’ force
favouring bigger volumes that fill the visual hull
L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

26.  (x) dS -   dV Protrusion problem ‘Balooning’ force
favouring bigger volumes that fill the visual hull
L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

27. Protrusion problem

28. Protrusion problem

29. Graph wij = 4/3h2 * (i+j)/2 wb wb = h3 wij i j h SOURCE
[Boykov and Kolmogorov ICCV 2001]
wb = h3
wij i j h

30. Results
Model House

31. Results
Model House – Visual Hull

32. Results
Model House

33. Results
Stone carving

34. Results
Haniwa

35. Summary Questions ? Novel formulation for multiview stereo
Volumetric scene representation
Computationally tractable global optimisation using Graph-cuts.
Visual hull for occlusions and geometric constraint
Occlusions approximately modelled
Questions ?