1. Seamless image-based texture atlases using multi-band blending Cédric AllèneJean-Philippe PonsRenaud Keriven Université Paris-Est – Ecole des Ponts – CERTIS

2. Seamless image-based 3D textures Outline 1) Context: 3D reconstruction 2) Image regions choice 3) Multi-view blending 4) Atlas building and results

3. Seamless image-based 3D textures - 1 - Context: 3D reconstruction

4. Seamless image-based 3D textures 1 – Context: 3D reconstruction 3D reconstruction from multi-view First step: Get a set of pictures or the model to reconstruct.

5. Seamless image-based 3D textures 1 – Context: 3D reconstruction 3D reconstruction from multi-view Second step: Autocalibration from your set of pictures.

6. Seamless image-based 3D textures 1 – Context: 3D reconstruction 3D reconstruction from multi-view Third step: 3D mesh building.

7. Seamless image-based 3D textures 1 – Context: 3D reconstruction 3D reconstruction from multi-view Fourth step: Add colored textures on the mesh from multi-view images. Problems:  Which image should the texture be from?  How to minimize color discontinuities at the frontiers of regions from different images? This is here our method comes into play!

8. Seamless image-based 3D textures - 2 - Image regions choice

9. Seamless image-based 3D textures Surface partition We note:  the input calibrated images, and  the projection from 3D space to image We have to assign each face of the mesh in to one of the input views in which it is visible. We obtain a labeling vector What do we want?  good visual detail, and  color continuity at region boundaries 2 – Image regions choice

10. Seamless image-based 3D textures Good visual detail: Color continuity at region boundaries: where is a dissimilarity term between the neighbour faces and with labels and. Energy terms 2 – Image regions choice

11. Seamless image-based 3D textures Energy minimization Global energy to minimize: Since is a sum of regular functions, we can use the MRF/min-cut minimization method to solve our problem. Problem: Even if minimized, color discontinuities remain between regions of different labels… 2 – Image regions choice

12. Seamless image-based 3D textures - 3 - Multi-view blending

13. Seamless image-based 3D textures Multi-frequencies blending 3 – Multi-view blending Method proposed in: P. J. Burt and E. H. Adelson. A multiresolution spline with application to image mosaics. ACM Trans. on Graphics, 2(4), 1983. Two main functions:  Reduce function (RED):  Input: image I  Output: image I’ which dimensions are half of those of input image I reduced through a gaussian kernel  Expand function (EXP):  Input: image I  Output: image I’ which dimensions are twice those of input image I expanded through the same gaussian kernel

14. Seamless image-based 3D textures Gaussian & laplacian pyramids 3 – Multi-view blending  Gaussian pyramid (left), obtained by successive reductions of input image.  Laplacian pyramid (right), obtained by substractions of two gaussian pyramid levels. As a consequence, the laplacian pyramid is a multi-frequencies decomposition.

15. Seamless image-based 3D textures Summing laplacian pyramid 3 – Multi-view blending Important remark: Summing through successive expansions the laplacian pyramid allows to get back the gaussian pyramid and, so, the original image.

16. Seamless image-based 3D textures ? 2D blending Second step: Create a new laplacian pyramid from the image ones through the gaussian pyramids of their masks. 3 – Multi-view blending 2D blending: Merging in a “natural” way the pictures following their respective masks. First step: Build the laplacian pyramid for each image, and Build the gaussian pyramid for each mask. Third step: Get the final blended image by summing the newly created laplacian pyramid.

17. Seamless image-based 3D textures Multi-view blending Straightforward from 2D blending using the projection functions for each image. So we obtain the final color at point x of the surface: with 3 – Multi-view blending

18. Seamless image-based 3D textures - 4 - Atlas building and results

19. Seamless image-based 3D textures Results – Aiguille du Midi (France) 4 – Atlas building and results "Naive" approach Optimized patchwork Optimized patchwork + blending Successive approaches Copyright Bernard Vallet (www.bvallet.com)

20. Seamless image-based 3D textures Atlas – Aiguille du Midi (France) 4 – Atlas building and results

21. Seamless image-based 3D textures Results – Ettlingen castle (Germany) 4 – Atlas building and results Optimized patchwork + blending Successive approaches Courtesy Christoph Strecha, EPFL (http://cvlab.epfl.ch/strecha/multiview/) "Naive" approach Optimized patchwork

22. Seamless image-based 3D textures Atlas – Ettlingen castle (Germany) 4 – Atlas building and results

23. Seamless image-based 3D textures Conclusion Seamless image-based 3D textures:  Graph-based combinatorial optimization for texture partition  Multi-view blending on the partition Advantages:  Good results Drawbacks:  Computing time dependant of the number of levels in the pyramids Future work:  GPU acceleration of the blending part 4 – Atlas building and results

24. Seamless image-based 3D textures Thank you for your attention! Any question?