1. http://www.VRVis.at/ A Globally Optimal Algorithm for Robust TV-L 1 Range Image Integration Christopher Zach VRVis Research Center Thomas Pock, Horst Bischof Institute for Computer Graphics and Vision TU Graz

2. Christopher Zach http://www.VRVis.at/ 2Robust Range Image Integration – 11/20/2016 Dense 3D Reconstruction  E.g. site reconstruction from images  Standard SfM

3. Christopher Zach http://www.VRVis.at/ 3Robust Range Image Integration – 11/20/2016 Dense 3D Reconstruction  Multi-view stereo for dense depth  Only statue model is desired  No fg/bg segmentation  “Clutter” in the depth maps

4. Christopher Zach http://www.VRVis.at/ 4Robust Range Image Integration – 11/20/2016 Depth Map/Range Image Fusion  “Robust” GPU-based VRIP  Proposed method: ☺ ?

5. Christopher Zach http://www.VRVis.at/ 5Robust Range Image Integration – 11/20/2016 Outline  Introduction to TV and L 1  TV-L 1 Approach to Range Image Integration  Results  Conclusion/Outlook

6. Christopher Zach http://www.VRVis.at/ 6Robust Range Image Integration – 11/20/2016 Classification [1] Lempitsky & Boykov, “Global optimization for shape fitting” [2] Vogiatzis et al., “Multi-view stereo via volumetric graph cuts” [3] Hornung & Kobbelt, “Robust reconstruction of watertight models from non- uniformely sampled point clouds” [4] Faugeras & Keriven, “Variational principles, surface evolution, PDEs, level set methods and the stereo problem” [5] Kazhdan et al., “Poisson surface reconstruction”

7. Christopher Zach http://www.VRVis.at/ 7Robust Range Image Integration – 11/20/2016 TV + L 1 Fact Sheet  Total Variation:  L 1 data fidelity:  TV-L 1 energy:  E is convex, but not strictly convex (minimum not unique)  For binary f: optimize for real-valued u, then apply thresholding  e.g. iso-surface extraction in our domain  More regularization: small features disappear instead of increased blurring  [6] Chan & Esedoglu, “Aspects of TV Regularized L 1 Function Approximation”

8. Christopher Zach http://www.VRVis.at/ 8Robust Range Image Integration – 11/20/2016 Binary TV-L 1 Example

9. Christopher Zach http://www.VRVis.at/ 9Robust Range Image Integration – 11/20/2016 Multiple Inputs to Approximate  Proposed TV-L 1 energy for multiple sources f i  Simultaneously approximating all f i  and minimize area of respective iso-surface while  Inputs may have missing data  Spatially regularized median  λ may be spatially varying  e.g. based on sampling density

10. Christopher Zach http://www.VRVis.at/ 10Robust Range Image Integration – 11/20/2016 Application to Depth Map Fusion  Convert source depth maps to directional signed + truncated 3D distance fields  Perform integration using the TV-L 1 approach  Extract the final iso-surface from the fused result

11. Christopher Zach http://www.VRVis.at/ 11Robust Range Image Integration – 11/20/2016 How to Optimize E  Direct optimization of E is difficult (or slow)  Strictly convex energy functional  u approximates v using the ROF energy  v approximates u and the given data point-wise

12. Christopher Zach http://www.VRVis.at/ 12Robust Range Image Integration – 11/20/2016 Determine u for fixed v  Problem:  This is the Rudin-Osher-Fatemi energy [7]  We employ the algorithm by Chambolle [8]:  [7] Rudin et al., “Nonlinear Total Variation based Noise Removal Algorithms”  [8] Chambolle, “An Algorithm for Total Variation Minimization and Applications” |p| ≤ 1

13. Christopher Zach http://www.VRVis.at/ 13Robust Range Image Integration – 11/20/2016 Determine v for fixed u  Problem:  This can be solved separately for every pixel/voxel  Linear-time search through intervals  If only missing values at x:  See paper for details

14. Christopher Zach http://www.VRVis.at/ 14Robust Range Image Integration – 11/20/2016 Implementation Notes  Signed distance generation on the GPU  Encoding of input volumes  “Run-length” compression of voxel data  number of empty/occluded/missing entries  sorted sequence of distance values  e.g. 54 views, 360x400x240 voxel space:  380 MB instead of 6.9 GB  Fusion accelerated by this scheme  Coarse-to-fine for faster fill-in

15. Christopher Zach http://www.VRVis.at/ 15Robust Range Image Integration – 11/20/2016 Middlebury Multi-View Eval. Depth maps from GPU-based plane-sweep

16. Christopher Zach http://www.VRVis.at/ 16Robust Range Image Integration – 11/20/2016 Varying λ

17. Christopher Zach http://www.VRVis.at/ 17Robust Range Image Integration – 11/20/2016 Reducing # of Depth Maps 40 depth maps λ = 0.2 20 depth maps λ = 0.4 10 depth maps λ = 0.8 6 depth maps λ = 1.2

18. Christopher Zach http://www.VRVis.at/ 18Robust Range Image Integration – 11/20/2016 Line-of-Sight Handling

19. Christopher Zach http://www.VRVis.at/ 19Robust Range Image Integration – 11/20/2016 More Results

20. Christopher Zach http://www.VRVis.at/ 20Robust Range Image Integration – 11/20/2016 Conclusion & Future Work  Pros:  Low quality depth maps yield high quality 3D meshes  Easy to implement (numerical part is ~150 basic C++ LoC)  Run-time in the magnitude of minutes  Parallelized implementations on multi-core CPUs possible  Yields global optimum  Todos:  Large scenery/high details  e.g. sector-based fusion  Incremental model update  Photo-flux for direct multi-view stereo

21. Christopher Zach http://www.VRVis.at/ 21Robust Range Image Integration – 11/20/2016 Outlook  Initial implementation on the GPU (Geforce 8800 GTX) 22 s 24 s 22 s

22. Christopher Zach http://www.VRVis.at/ 22Robust Range Image Integration – 11/20/2016 Thank you for your attention!