Surface parametrizations 2001/05/18 Geometry Images Xianfeng Gu Harvard University Steven Gortler Harvard University Hugues Hoppe Microsoft Research Good afternoon. This research project has been done in collaboration with my advisor Steven Gortler at Harvard University, and Hugues Hoppe from Microsoft Research. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Irregular meshes Vertex 1 x1 y1 z1 Vertex 2 x2 y2 z2 … Face 2 1 3 Face 4 2 3 … The standard geometric representation used in today’s graphics hardware is the irregular mesh. A mesh consists of an array of vertices, and an array of triangles, where each triangle refers to three vertex indices. <click> Here is such a mesh rendered with flat shading. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Texture mapping Vertex 1 x1 y1 z1 Vertex 2 x2 y2 z2 … s1 t1 s2 t2 Face 2 1 3 Face 4 2 3 … To represent more detail over meshes, it is common to use texture mapping. The mesh vertices are assigned texture coordinates, defining a parametrization of the mesh onto an S,T domain. <click> An image is placed on this domain, <click> which is then mapped back to the surface. In this case, the texture image represents surface normals, <click> which can be used in per-pixel shading. t normal map s IMA Workshop on Computer Graphics

Complicated rendering process Surface parametrizations 2001/05/18 Complicated rendering process Vertex 1 x1 y1 z1 Vertex 2 x2 y2 z2 … s1 t1 s2 t2 Face 2 1 3 Face 4 2 3 … random access! random access! This traditional rendering process is actually quite complicated. First, the hardware must access vertices in random order. This cost is mitigated to some degree through vertex caching. Once vertices are loaded, their texture coordinates in turn refer to some arbitrary area of the texture image. To mitigate this second cost, texture caching is used. The fact that this pipeline has been made efficient is a remarkable engineering feat. In this talk, we’ll introduce a representation that removes both of these expensive access patterns. ~40M Δ/sec IMA Workshop on Computer Graphics

Semi-regular representations Surface parametrizations 2001/05/18 Semi-regular representations [Eck et al 1995] [Lee et al 1998] [Khodakovsky 2000] [Guskov et al 2000] … Previous work in remeshing has gone part of the way in creating more regular geometric representations. But, such methods still use an irregular base mesh, which leads to a sampling that is only semi-regular. This presents challenges for hardware implementation. only semi-regular irregular vertex indices IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Geometry Image completely regular sampling 3D geometry Our approach is to represent an arbitrary surface using a completely regular grid of samples on a square domain, which we call a geometry image. <click> Here is an example. As you can see, it looks just like an ordinary image. It has 257-by-257 pixels. The only difference is that it has 12 bits/channel instead of the usual 8. Why do we call this a geometry image? Because its RGB colors encode XYZ positions, and in fact <click> entirely describe this 3D shape. As we will see later in the talk, the regularity of this representation can permit image processing techniques, such as compression, to be applied to geometry directly. geometry image 257 x 257; 12 bits/channel IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Basic idea cut parametrize Here is an outline of the basic steps involved in creating a geometry image. First, we slice open the mesh along an appropriate set of cut paths, creating a surface with the topology of a disk. <click> Second, we parametrize this cut surface onto the square domain of the geometry image. To illustrate this parametrization, we’ll show a demo. <demo> Here we see a morph between the cut bunny and its embedding in the square. You can see how the cut in blue is being morphed into the boundary of the square. Because this is so much fun, I’ll show the morph one more time. demo IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Basic idea cut sample Next, we overlay a uniform 2D sampling grid over the domain, <click> and map these samples through the parametrization back onto the surface. Note that it is important to create a good cut and a good parametrization, so that the samples are evenly distributed over the surface. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Basic idea cut store Next, we compute the XYZ positions of the surface samples, as well as other surface attributes, and store these in the geometry image. Here we are visualizing the XYZ data as RGB colors. <click> Given a geometry image, it is straightforward to render it as shaded 3D geometry. We’ll next discuss each of these steps in further detail. render [r,g,b] = [x,y,z] IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 How to cut ? 2D surface disk First, let’s consider how we can cut a surface onto a disk. Suppose we are given a sphere in 3D. <click> We can slice it open along a path from one pole to the other, as show here in red. <click> This results in the surface disk shown on the right. sphere in 3D IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 How to cut ? 2D surface disk We can also introduce additional cuts, <click> as shown here, <click> to obtain this surface disk shown now on the right. <click> In general, for a genus-0 closed surfaces, any tree of edges constitutes a valid cut. <pause> sphere in 3D Genus-0 surface  any tree of edges IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 How to cut ? What about surfaces of higher genus? <click> For this torus, which has genus 1, a valid cut must include two loops that generate its fundamental group. <click> More generally, for a genus-g surface, the cut must include 2g independent loops. <click> In addition to these two loops, the cut may also contain an arbitrary forest of edges. torus (genus 1) Genus-g surface  2g generator loops minimum IMA Workshop on Computer Graphics

Surface cutting algorithm Surface parametrizations 2001/05/18 Surface cutting algorithm (1) Find topologically-sufficient cut: 2g loops [Dey and Schipper 1995] [Erickson and Har-Peled 2002] (2) Allow better parametrization: additional cut paths [Sheffer 2002] Our cutting algorithm has two phases. In the first phase, we find an initial topologically-sufficient cut with 2g loops. This problem has been studied in the computational topology literature. We use a retraction-based algorithm similar to that of Dey and Schipper. In the second phase, we introduce additional cuts to improve the quality of the parametrization. A related approach is described by Sheffer, but our method more robustly identifies surface extremities. IMA Workshop on Computer Graphics

Step 1: Find topologically-sufficient cut Surface parametrizations 2001/05/18 Step 1: Find topologically-sufficient cut (a) retract 2-simplices (b) retract 1-simplices To create a topologically-sufficient cut, we first remove a seed face. We then apply a 2-phase retraction algorithm. <click> In the first phase, we iteratively remove any edge adjacent to a single face, as well as that face, <click> as shown here. Note that the retraction leaves behind dangling edges. <7 clicks> This sequence illustrates the process. Since we have removed a topological disk, the remaining edges must form a topologically-sufficient cut. However, the shape of the removed disk is quite convoluted, with every vertex on its boundary. <click> Therefore, in the second phase, we iteratively retract each dangling edge, <click> as shown here. <6 clicks> This sequence illustrates the second phase. Since the resulting cut may be serrated, we straighten the loops using shortest path computations, <click>, as shown here. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Results of Step 1 Here are some examples of topologically-sufficient cuts determined by Step 1. Note that the cuts are shown in blue where visible, and in red where occluded. The genus-6 Buddha has 12 loops, and the 3-holed torus has 6 loops. For a genus-0 surface, the result of retraction is a single vertex. In this case, we re-introduce two neighboring edges in order to allow parametrization into a disk. genus 6 genus 3 genus 0 IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Step 2: Augment cut Make the cut pass through “extrema” (note: not local phenomena). Approach: parametrize and look for “bad” areas. To avoid under-sampling, we have found that it is important for the cut to pass through the various extrema of the mesh. Extrema are not a local phenomena and are tricky to both define and discover. In our approach, given a current cut, we find extrema by tentatively parametrizing the mesh, and then finding the region that will be most under-sampled. The cut is then augmented to reach these extrema. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Step 2: Augment cut We will use this genus-0 example to describe the details of the cut augmentation step. <click> Here, the result of step 1 is simply two edges. <click> We can therefore open and parametrize the mesh into a quadrilateral. Note that the entire hand is hidden in the central dense mass of vertices. Given this mapping, we locate the point where the mapping will have the greatest undersampling, <click> shown in yellow at the tip of middle finger. <click> We extend the cut to include this point by finding the shortest path from it to the cut, <click> and re-parametrize using the resulting cut mesh. We repeat this process, as long as we can further improve the quality of the parametrization. <6 clicks> …iterate while parametrization improves IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Results of Steps 1 & 2 Here are some examples showing the results of our complete cutting algorithm. Observe that the cut reaches the fingertips of the tyrannosaur and the lip of the dragon. <pause> genus 1 genus 0 IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Parametrize boundary a a’ a’ a We now have computed the cut for the mesh. Our next task is to map the open mesh onto the unit square in 2D. We first determine the parametrization of the boundary. We must ensure that both sides of the cut will be sampled identically, so that the seams of the geometry image meet exactly, without cracks. To achieve this, we enforce two boundary constraints. First, each cut path is allocated the same boundary length as its mate, as shown here in blue. Second, the endpoints of the cut paths, shown in orange, are forced to lie at grid points in 2D. These constraints are generally easy to satisfy. Constraints: cut-path mates identical length endpoints at grid points  no cracks IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Parametrize interior Once the boundary parametrization is fixed, we then proceed to parametrize the interior. To do this, we find the parameterization that minimizes the geometric-stretch metric as introduced in Sander et al 2001. <click> As shown here, this metric attempts to minimize undersampling, and thus uniformly distributes samples over the surface. In fact, as described in Sander et al 2002, this parameterization minimizes geometric reconstruction error with respect to a point-sampled approximation. Geometric-stretch metric minimizes undersampling [Sander et al 2001] optimizes point-sampled approx. [Sander et al 2002] IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Stretch parametrization Previous metrics (Floater, harmonic, uniform, …) This slide shows the importance of using the stretch-minimizing parametrization, shown in the top row. As shown on the bottom, if we were to use previous metrics found in the literature, regions of the surface would be under-sampled, resulting in a poor reconstruction. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Sample As mentioned previously, once we have a parametrization, we sample the surface at domain grid points to create the geometry image. <pause> geometry image IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Rendering To render geometry images on current hardware, we span each quad of samples with two triangles. <click> This is done by splitting along the shorter of its two diagonals. <click> Here is a rendering of a very coarse 65-by-65 geometry image. <pause> (65x65 geometry image) IMA Workshop on Computer Graphics

Rendering with attributes Surface parametrizations 2001/05/18 Rendering with attributes geometry image 2572 x 12b/ch Here is another example using a 257-by-257 geometry image, shown flat-shaded. <click> To exploit the texture-mapping capabilities of today’s hardware, we sample a normal-map image using the exact same parametrization used to construct the geometry image. This normal-map image is sampled at a higher resolution, here 512-by-512. <click> The normal-mapped rendering shown here is thus created completely from these two images. normal-map image 5122 x 8b/ch rendering IMA Workshop on Computer Graphics

Advantages for hardware rendering Surface parametrizations 2001/05/18 Advantages for hardware rendering Regular sampling  no vertex indices. Unified parametrization  no texture coordinates.  Raster-scan traversal of source data: geometry & attribute samples in lockstep. Summary: compact, regular, no indirection Here are some advantages of geometry images for hardware-rendering. First, because the geometry image is a regular grid, vertex indices are absent. Second, the geometry image and attribute images (such as color-maps or normal-maps) share a unified parametrization, so texture coordinates are also unnecessary. As a result, the rendering process can be made very simple by just traversing the geometry and attribute images in raster-scan order, considering geometry and attribute samples in lockstep. In summary, for rendering, geometry images are compact, provide regular access, and eliminate expensive indirection. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Normal-Mapped Demo geometry image 129x129; 12b/ch Now I’ll show a demo. We started with the Buddha model of half-a-million triangles, and converted it into these two images. The geometry image is sampled at a coarse resolution of 129-by-129, while the texture map is sampled at a resolution of 512-by-512. <start demo> <make sure left window selected> In the left window, we see the irregular mesh, simplified to 50,000 faces for the purpose of this demo. <press DE> These blue edges show the cut. <press Ds> Here they are without the mesh. <press DsDe> <Alt-tab to right window> On the right, we see the rendered geometry image. <press De> Here it is flat-shaded. <pause> <press Dt> Here it is normal-mapped (or texture mapped with pre-shaded colors). Note how it faithfully reproduces the original mesh. normal map 512x512; 8b/ch demo IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Pre-shaded Demo geometry image 129x129; 12b/ch Now I’ll show a demo. We started with the Buddha model of 500,000 triangles, and converted it into these two images. The geometry image is sampled at a coarse resolution of 129-by-129, while the texture map is sampled at a resolution of 512-by-512. <start demo> <make sure left window selected> In the left window, we see the irregular mesh, simplified to 50,000 faces for the purpose of this demo. <press DE> These blue edges show the cut. <press Ds> Here they are without the mesh. <press DsDe> <Alt-tab to right window> On the right, we see the rendered geometry image. <press De> Here it is flat-shaded. <pause> <press Dt> Here it is normal-mapped (or texture mapped with pre-shaded colors). Note how it faithfully reproduces the original mesh. color map 512x512; 8b/ch demo IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Results 257x257 Here is another example. In this case, the geometry image is 257-by-257, and we use a normal-map at 512-by-512 resolution. <pause> normal-map 512x512 IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Results 257x257 In this example, the texture-map is an ordinary color-map with pre-shaded colors. <pause> color image 512x512 IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Mip-mapping 257x257 129x129 65x65 Level-of-detail rendering can be directly implemented by mip-mapping the geometry image. <click> Here we show the resulting renderings at 3 different resolution levels. In order to avoid cracks at multiple levels of details, we use geometry images with resolution two-to-the-n-plus-1, and apply simple sub-sampling. <click> Also, we construct the parametrization boundary constraints for the grid sampling on the coarsest mipmap level, in this case 65-by-65, so that the cut is crack-free for all finer resolution. boundary constraints set for size 65x65 IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Hierarchical culling view-frustum culling geometry image Hierarchical culling is a common optimization for real-time rendering. It is interesting to note that it can be implemented over a geometry image using a simple quad-tree structure. Quad-tree nodes over the geometry image can be used for view-frustum culling. And similarly, quad-tree nodes over the normal-map can be used for backface culling. It’s elegant that both these operations can be done in a unified setting. backface culling normal-map image IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Compression Image wavelet-coder Geometry images tend to be rather smooth, and therefore provide opportunity for compression. <click> Here, we use a standard wavelet image coder. To show an extreme example, we have selected an aggressive compression factor of 200. <click> The resulting geometry is shown on the right. Since the wavelet coding is lossy, cut-path mates may no longer join seamlessly along the cut. <click> To address this problem, we also encode the topology of the cut into a data sideband, which only requires about 12 bytes. <click> During decompression, we use this topological information to geometrically fuse the cut. 295 KB  1.5 KB fused cut + topological sideband (12 B) IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Compression results 295 KB  In the previous example, we showed a compression ratio of 200-to-1. This sequence shows results of less-severe compression. Note the reasonable quality at 12 kilobytes. <pause> 1.5 KB 3 KB 12 KB 49 KB IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Rate distortion This rate-distortion graph shows reconstruction accuracy as a function of compression rate for the bunny model. The blue curve shows our results for a wavelet-compressed geometry image. For comparison, the red curve shows results for the Khodakovsky coder, which is one the best known algorithms for lossy geometry compression. We were reasonably happy that we could be so competitive without using any specialized compression techniques. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Some artifacts Our geometry images do suffer from some artifacts. Here are some examples in the remeshing of the Buddha model. First, sharp features, such as creases, currently suffer from aliasing. We hope to improve this by optimizing sample placement. Also, there can be regions where the parameterization is quite anisotropic aliasing anisotropic sampling IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Summary Simple rendering: compact, no indirection, raster-scan stream. Mipmapped geometry Hierarchical culling Compressible In summary, the geometry image is a completely regular representation for arbitrary surfaces. It allows for a very simple rendering process, in which ordinary images are just streamed in raster-scan order into the rendering pipeline. Treating geometry as an ordinary imaged signal allows mip-mapping to be applied directly to geometry. Also, hierarchical culling can use a simple, implicit structure over a 2D grid. Finally, we were able to obtain very promising compression results using off-the-shelf image coders. IMA Workshop on Computer Graphics

Surface parametrizations 2001/05/18 Future work Better cutting algorithms Feature-sensitive remeshing Tangent-frame compression Bilinear and bicubic rendering Build hardware There are many areas for future work. We want to pursue better algorithms for surface cutting. Feature-sensitive remeshing should improve reconstruction results for meshes with sharp geometry. Improved compression could be obtained by using methods specialized for surfaces, such as tangent-frame coding. We would like to explore new rendering schemes using higher-order surfaces. Finally, it would be nice to realize this in hardware. Thank you. IMA Workshop on Computer Graphics