Lapped textures Emil Praun Adam Finkelstein Hugues Hoppe 2000/07/28 Lapped Textures Emil Praun Adam Finkelstein Hugues Hoppe Princeton University Microsoft Research SIGGRAPH 2000

? Goal mesh geometry textured surface “example” image Lapped textures 2000/07/28 Goal mesh geometry ? The goal of our project is to texture 3D objects by example: given a triangle mesh and a 2D image [animation], we want to cover the object with texture that looks like the provided example. textured surface “example” image SIGGRAPH 2000

Goal Little user effort No apparent seams No obvious periodicity Lapped textures 2000/07/28 Goal Little user effort No apparent seams No obvious periodicity Low distortion Local texture control Anisotropy There are a number of properties that such a texture creation method should have: it should require relatively little user effort, the resulting object should be seamlessly covered, there should be no obvious periodicity, as one notices when tiling the plane with a given image, The mapping of the texture onto curved regions should be achieved without large distortion. Finally it would be nice if the user could locally control the texture appearance, like relative size and, for anisotropic textures, the orientation. SIGGRAPH 2000

Previous 2D Texture Synthesis Lapped textures 2000/07/28 Previous 2D Texture Synthesis Histogram equalization [Heeger ’96] Laplacian block shuffling [de Bonet ’97] Pixel template matching [Efros ’99] [Wei ’00] Most work in synthesizing 2D textures by example has utilized randomness combined with constrained matching. These methods are usually fairly slow to compute and while they may be extended to 3D, they would be even slower. SIGGRAPH 2000

Previous 2D Texture Synthesis Lapped textures 2000/07/28 Previous 2D Texture Synthesis Histogram equalization [Heeger ’96] Laplacian block shuffling [de Bonet ’97] Pixel template matching [Efros ’99] [Wei ’00] Random pasting of image blocks [Xu ’00] Our work is largely inspired by that of Xu et all, whose method I will describe briefly. They start with a small toroidal image, which they tile to produce an image of the desired size.[animation] They select rectangular blocks of texture which they copy and paste across the image with a small amount of blending at the edges. [animation] Here is an example of a box that was moved. [animation] And here is another example. SIGGRAPH 2000

Previous 3D Texturing Volumetric textures: Lapped textures 2000/07/28 Previous 3D Texturing Volumetric textures: Noise functions [Perlin ’85, Worley ’96] Solid textures by example [Ghazanfarpour ’96] Synthesizing texture on a surface: Reaction-diffusion [Turk ’91, Witkin ’91] Cellular textures [Fleischer ’95] Covering surface with triangular tiles [Neyret ’99] Volumetric textures are good for many applications, but not all surface textures can be modeled with solid textures. In particular, it is hard to model anisotropy, when the texture preferred direction is supposed to change across a surface: it is hard to model the bark on a very twisted tree w/ solid textures. Reaction-diffusion and cellular texture methods work directly on surface of arbitrary topology in 3D. However, they are slow, and being parametric, it is hard to control the final appearance of the texture. Neyret & Cani’s scheme, presented last year, does not handle anisotropic textures. SIGGRAPH 2000

Approach texture patch surface Lapped textures 2000/07/28 Our method is very similar to the work of Xu et al in 2D: it works by cutting regions of the input image, that we call texture patches, texture patch surface SIGGRAPH 2000

Key Idea: Patch Pasting Lapped textures 2000/07/28 Key Idea: Patch Pasting And repeatedly pasting them onto the 3D surface,[animation] until we cover it.[animation] Note that we do not produce a tiling of the surface, since we place the patches in overlapping fashion. This is where the name of our method comes from. texture patch “lapped textures” surface SIGGRAPH 2000

PROCESS Lapped textures 2000/07/28 I am now going to present the process. SIGGRAPH 2000

Algorithm texture patch surface Lapped textures 2000/07/28 The overview of the algorithm is as follows: We start by cutting the texture patches. Quite often, a single patch suffices. For anisotropic textures only we specify a direction field over the mesh. {Next Slides:} We select an uncovered mesh point, Grow a surface patch around it, Parametrize it over the texture, And using this parametrization, we paste a copy of the texture patch onto the surface. We update face coverage information, and then repeat the process starting with another uncovered mesh point. We stop when we have covered the mesh. texture patch surface SIGGRAPH 2000

Algorithm texture patch surface Lapped textures 2000/07/28 We select an uncovered mesh point, Grow a surface patch around it, Parametrize it over the texture, And using this parametrization, we paste a copy of the texture patch onto the surface. texture patch surface SIGGRAPH 2000

Algorithm texture patch surface Lapped textures 2000/07/28 We update face coverage information, and then repeat the process starting with another uncovered mesh point. texture patch surface SIGGRAPH 2000

Algorithm texture patch surface Lapped textures 2000/07/28 We stop when we have covered the mesh. texture patch surface SIGGRAPH 2000

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering In order to implement this algorithm we had to address these six issues. SIGGRAPH 2000

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering The first step in our process is the texture patch creation. SIGGRAPH 2000

Texture Patch Creation Lapped textures 2000/07/28 Texture Patch Creation For very structured textures, the final result looks better if the patch boundaries do not cut across semantic features in the input texture, like bricks, or strands in this woven mat image. Since semantic classification of textures is not a solved problem, we let the user cut the patches. The patch consists of the input image together with a transparency mask: the patch is opaque at its center but has a gradual blend region near its boundaries. This helps hide the seams between different patches on the textured mesh. SIGGRAPH 2000

Less Structure  Splotch Lapped textures 2000/07/28 Less Structure  Splotch For less structured images, the patch boundaries do not have to align with specific texture features, so we can use a generic shape, such as an irregular blob.[animation] We prefer wavy boundaries since the human eye is better at detecting straight-line discontinuities rather than wavy ones. SIGGRAPH 2000

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering The next step is to establish a direction field over the mesh. SIGGRAPH 2000

Direction Field: User-specified Lapped textures 2000/07/28 Direction Field: User-specified This is very important for anisotropic textures. Quite a number of real-world textures are anisotropic: they don’t look the same if you rotate them. Our algorithm needs to know how to align these texture with respect to the surface. For this purpose, the user specifies the desired direction for a small number of faces. The direction is interpolated for the remaining faces. SIGGRAPH 2000

Direction Field: Local to Patch Lapped textures 2000/07/28 Direction Field: Local to Patch For anisotropic textures, there is no need for a globally consistent direction field, however we need local ones in order to compute the parametrization for each patch. A random tangential direction is chosen for the center of the patch and subsequently propagated to all the faces that are added to the patch. SIGGRAPH 2000

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering This brings us to the next point, surface patch growth. SIGGRAPH 2000

Patch Growth Lapped textures 2000/07/28 We start with a seed point on the mesh and the face containing it. The seed point is mapped to the texture patch center. {Next slides:} As we add faces to the patch, we make an initial guess for their parametrization. We need this initial guess in order to know how large we should grow the patch. When growing the patch we avoid introducing faces that would cause unacceptable texture distortion. Also, the patch is kept homeomorphic to a disk, so we do not allow any holes or cylindrical patches. SIGGRAPH 2000

Patch Growth Lapped textures 2000/07/28 As we add faces to the patch, we make an initial guess for their parametrization. SIGGRAPH 2000

Patch Growth

Patch Growth Lapped textures 2000/07/28 We need this initial guess in order to know how large we should grow the patch. SIGGRAPH 2000

Patch Growth Lapped textures 2000/07/28 When growing the patch we avoid introducing faces that would cause unacceptable texture distortion. SIGGRAPH 2000

Lapped textures 2000/07/28 Patch Growth SIGGRAPH 2000

Patch Growth Lapped textures 2000/07/28 Also, the patch is kept homeomorphic to a disk, so we do not allow any holes or cylindrical patches. SIGGRAPH 2000

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering Once we have a surface patch, we optimize its parametrization. SIGGRAPH 2000

Align Patch to Direction Field Lapped textures 2000/07/28 Align Patch to Direction Field The goal is to align the texture principal direction, such as the grooves in a tree bark texture, with the specified direction field. texture patch surface SIGGRAPH 2000

Tangential Vector Field Lapped textures 2000/07/28 Tangential Vector Field Every triangle has a local coordinate frame, composed of the given direction field, the triangle normal, and their cross product. The red vector represent the local “right” direction for every triangle while the green one is the local “up” direction. SIGGRAPH 2000

Optimizing the Parametrization Lapped textures 2000/07/28 Optimizing the Parametrization The optimization tries to match these direction to the corresponding axes unit vectors in texture space, in a least squares sense. We construct a sparse linear system and solve for the texture coordinates at the vertices. Our only constraints are that all the red vectors should match the unit OX vector and all the green vectors should match the unit OY vector. There is no explicit fairness functional, the fairness comes from the underlying direction field on the mesh. Least squares best match to unit axes Sparse linear system. No explicit fairness functional SIGGRAPH 2000

Result of Optimization Lapped textures 2000/07/28 Result of Optimization Here you can see the result of the optimization. Note that since the method locally warps the texture everywhere to align to the globally consistent direction field, derivative continuity at patch boundaries is very good. This would not happen with methods that use a singe direction constraint for a whole patch (for instance at its center). If you look at the texture squares, they are aligned everywhere and they follow the direction filed streamlines. Such a streamline is shown in yellow here. SIGGRAPH 2000

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering In order to know when to stop pasting we need to keep track of what parts of the surface have been covered. SIGGRAPH 2000

Coverage estimation Render patch triangles Flag covered triangles Lapped textures 2000/07/28 Coverage estimation Render patch triangles Flag covered triangles Remember 1 pixel per uncovered triangle After each paste operation, we update the coverage information for the affected faces. We render these faces in an off-screen buffer. We composite all the previous paste operations that affect them. off-screen buffer SIGGRAPH 2000

Coverage estimation Render patch triangles Flag covered triangles Remember 1 pixel per uncovered triangle off-screen buffer

Coverage estimation Render patch triangles Flag covered triangles Remember 1 pixel per uncovered triangle off-screen buffer

Issues Texture patch creation Specifying direction field Lapped textures 2000/07/28 Issues Texture patch creation Specifying direction field Surface patch growth Patch parametrization Face coverage estimation Texture storage and rendering The final issue is texture storage and rendering. SIGGRAPH 2000

Texture Storage and Rendering Lapped textures 2000/07/28 Texture Storage and Rendering Method 1: Texture Atlas Pre-composite into a global texture map. -- OR -- Method 2: Runtime pasting Composite at run-time using hardware Once we have the patch parameterizations, we can store and create the texture in two ways: The traditional representation, that people normally use, is to create a global texture map for the whole mesh and pre-composite all the paste operations in it. Such a global texture map is called a texture atlas. We introduce an alternate method which consists of just storing the computed parameterizations and compositing all the patches at runtime. SIGGRAPH 2000

Method 1: Texture Atlas Patches of triangles with similar normals Lapped textures 2000/07/28 Method 1: Texture Atlas In order to create a texture atlas,we group faces with similar normals together to form patches which we relax using the described optimization. To have a single unified texture for the whole mesh we need to arrange the 2D images of these patches inside a unit square. Finding the optimum arrangement is not tractable for a few hundred arbitrary polygons. There exist a few heuristics but they don’t perform as well as a trained human. In our prototype implementation we let the user arrange the pieces by hand. Of course, the atlas is independent of the texture so this only needs to happen once for every mesh that need to be textured. Patches of triangles with similar normals 2D packing problem for arbitrary polygons SIGGRAPH 2000

Method 2: Runtime Pasting Lapped textures 2000/07/28 Method 2: Runtime Pasting Store vertex coordinates for each patch Composite at run-time using hardware May render triangles several times For the runtime pasting approach, we store the texture coordinates at the vertices for every paste operation. We then draw all the patches, compositing them using the frame buffer. Since triangles may be covered by different patches, they may be drawn more than once. This problem may be alleviated using multitexturing hardware. SIGGRAPH 2000

Atlas vs. Runtime Pasting Lapped textures 2000/07/28 Atlas vs. Runtime Pasting Atlas Faster rendering, more portable Easy to paint unique details (eyes, nose on bunny) Sampling artifacts; user effort Pasting Increases model complexity ( 1.6 –3) Huge effective resolution Reuse splotch parameterization for many textures Here are the pros and cons of the atlas approach versus runtime pasting. Meshes textured with an atlas render slightly faster and may be more portable. The suffer from resampling artifacts since there is an extra resampling step: pixels move from the original texture to the texture atlas and then to the screen. Also, since the atlas image has to represent the whole surface, its resolution may not be enough to capture all the detail present in the original image. User effort may be needed to create a good texture atlas. Runtime pasting has the drawback of slightly increasing the model complexity. However, it offers a huge effective resolution by storing a small texture and displaying it many times. For textures that accommodate a generic patch boundary, the computed parametrization can be reused allowing the user to browse through several textures. All the results that we present use runtime pasting. SIGGRAPH 2000

RESULTS

(completely automatic: no direction field) Lapped textures 2000/07/28 Results: Splotches These results were produced in a completely automatic fashion, since these textures are isotropic and also don’t require customized patch boundaries. (completely automatic: no direction field) SIGGRAPH 2000

Results: Anisotropic Lapped textures 2000/07/28 Here are some examples of structured textures with user-defined patch boundaries. SIGGRAPH 2000

Controlling Direction and Scale Lapped textures 2000/07/28 Controlling Direction and Scale The stripes on the body of the bunny differ in orientation and size, while the ones on the head and on the ears are roughly the same. SIGGRAPH 2000

256 x 256 texture 15,000 faces 25 frames per sec! Lapped textures 2000/07/28 256 x 256 texture (282 times) 15,000 faces The rock bunny from the proceedings cover stores only a 256 by 256 texture image, instanced 282 times, to texture a model with 15000 faces. It renders at 25 frames per second at a resolution of 1024 squared. 25 frames per sec! SIGGRAPH 2000

direction field singularities Lapped textures 2000/07/28 Limitations Like many texturing algorithms, our method has some limitations. It does not work well with textures that have strong low frequency components, since the patch overlap might juxtapose image regions with different overall color, revealing the patch boundaries. Also, since we place patches in random location, the texture features at the boundaries might not line up exactly with the features of texture that has already been placed. In this brick example we see fragmented bricks between the two patches. In covering an arbitrary topology mesh with a consistent direction field, we may need to introduce singularity points or discontinuity lines. low-frequency components boundary mismatches direction field singularities SIGGRAPH 2000

Video

Interactive Paint Demo

Timings Texture patch creation: 1 min Human effort Lapped textures 2000/07/28 Timings Texture patch creation: 1 min Specifying direction field: 15 min Surface patch growth Patch parameterization Face coverage estimation Rendering: 25fps @ 10242 Human effort Preprocessing: 20sec – 6 min Here are the timings for the different stages of the process. Note that human effort is only needed for some textures: stochastic textures don’t need customized patches while isotropic textures don’t need a globally consistent direction field. Preprocessing took between 20 sec and 6 min, but it was in most cases under 1 min. The models render at 25 frames per second at full screen resolution on a GeForce graphics card. Pentium III 733MHz, GeForce graphics SIGGRAPH 2000

Conclusions Effective texture-by-example through: Lapped textures 2000/07/28 Conclusions Effective texture-by-example through:  Overlapping texture patches  Minimal edge blending Aligning to direction field  fast optimization Runtime pasting  high effective resolution In conclusion, we have shown how to do effective texturing of meshes using the concept of placing overlapping texture patches. Hiding the patch seams is done by alpha blending over a few pixels region near the boundaries. The point of this talk is: even though it may seem surprising, since one would expect to see these boundaries, THE METHOD WORKS. THE FINAL RESULTS LOOK VERY GOOD FOR A LARGE VARIETY OF TEXTURES AND MESHES. We have also shown how the parametrization based on direction field alignment is fast, since it only requires solving a sparse linear system. It is also robust and does not use an explicit fairness functional, the fairness coming from the underlying direction field. Finally, runtime pasting is an efficient method for storing lapped textures, offering high effective resolution, and opening the way for many interesting applications. This brings us to future work: SIGGRAPH 2000

Future Work Other texture types: Greater automation Lapped textures 2000/07/28 Future Work Other texture types: Animated “Thick” (volumetric) textures  fur NPR rendering Greater automation Fine-tuning patch placement We are investigating the use of the lapped textures paradigm for: Animated (video) textures: instead of browsing through a set of texture images, we could run a small movie inside the texture patch. Volumetric textures, that represent the mesh as a series of concentric shells, are useful for rendering such phenomena as fur. We are also investigating the use of lapped textures for the parametrization of the strokes that are drawn in a non-photorealistic rendering style. A possible direction for future work might be to alleviate the problem of boundary feature mismatch, as in the brick case: by aligning a patch to the previously-placed texture we might improve the match. The method may also benefit from an automated patch boundary selection mechanism for structured textures. Such a method may use snakes or intelligent scissoring. The direction field may also be created automatically, using curvature information from the mesh. SIGGRAPH 2000