Synthesis of Compact Textures for real-time Terrain Rendering Nader Salman 22 juin 2007 Encadrant : Sylvain Lefebvre

Motivations Need for large textures –Simple repetition: low quality Texture synthesis from example –Pre-computation: Need to store large result Limited memory! –Image Compression: Rendering: Hinders sequential coding Block based: Fixed rate (÷ 6) –Our approach : Synthesize compact textures, that can be efficiently decoded at rendering time [Garber 1981, Popat & Picard 1993, Efros & Leung 1999, Wei & Levoy 2000, Ashikhmin 2001, Hertzmann et al 2001, Tong et al 2002 …]

Our approach Inspired by patch-based synthesis –Patch-based Synthesis [Praun et al 2000, Liang et al 2001, Efros and Freeman 2001, Kwatra et al 2003]

Our approach Inspired by patch-based synthesis –Record patch positioning and stitching (instead of final image) –Decode at rendering time Main difficulties: –Adapt synthesis for compact encoding –Prepare for efficient decoding on GPU

Exemplar Positioning 12 Cuts 3 Stitch Our approach

A first example Memory size: –Full image: 805 MB ( result size: 16K x16K ) –Our encoding: 26 MB ( patch size: 128x128 ) Exemplar (1024x1024) Synthesis Result Compact representation

Overview Introduction Our approach –Patch positioning –Optimal cuts –Seam removal Results and performance Future Work –Multiple materials –Slope –GPU Decoding

Our approachExemplar Positioning 1 2 Cuts 3 Stitch 1. Positioning Exemplar Scan exemplar Choose best fit patch Place patch Memorize position ? Patch position Compact representation

Exemplar Positioning 1 2 Cuts 3 Stitch Our approach

Why do we cut?

Graph cut A B Construct graph such that: Graph Mincut  Best Seam A ∞ ∞ ∞ B ∞ ∞ ∞ Potential Seam Cost Constraint A B Seam A B [Kwatra 2003]

Graph cut Problem with encoding: –Cut can be arbitrary  Expensive to store! Our approach –Constrain cuts to be 1D functions –Still search for optimal!

Our approachExemplar Positioning 12 Cuts 3 Stitch 2. Optimal Cuts Current output A B B Constrained arcs Bidirectional arcs One D. Seam Horizontal Overlap Image Vertical Overlap Image Compact representations

Exemplar Positioning 12 Cuts 3 Stitch Our approach

Why stitching ? Seams might still be visible in some cases  Avoid visible discontinuities

Our approach 3. Stitching New patchExisting texture Poisson Image Editing Current position Mask [Perez et al. 2003]

Encoding of Poisson field (1) Before PoissonAfter Poisson Deformation Field (x8 for visualisation)

Encoding of Poisson field (2) Problem: –Deformation is an image as big as output! Approx ?

Compression: –Per patch Approximate as sum of 2 fields Each is a ‘flattened 1D pyramid’ –Optimize with least square argmin || Ax - b || 2 –Fast reconstruction on the GPU 1D pyramid = 1D MIP-map Encoding of Poisson field (3) High Frequency Smooth

Encoding of Poisson field (4) pSNR = 43,7 dB Output with PoissonApproximated Output High Frequencies

Output with Poisson pSNR = 50,2 dB pSNR = 43,2 dB pSNR = 46,12dB Approx. outputs

Overview Introduction Our approach –Patch positioning –Optimal cuts –Seam removal Results and performance Upcoming Work –Multiple materials –Slope –GPU Decoding

Algorithm performance Patch size (pixels) Output size (MB) Patch Size 16x16 Output 2,4 GB Patch Size 32x32 Output 542 MB Patch Size 128x128 Output 90 MB Input size: 1024 x 1024 Output size: 32K x 32K 32K x 32K  3GB uncompressed

Quick comparison Our approach [Kwatra 2003]

Overview Introduction Our approach –Patch positioning –Optimal cuts –Seam removal Results and performance Upcoming Work –Multiple materials –Slope –GPU Decoding

Conclusion Need for detailed large textures –Limited memory Contributions –High quality texture synthesis Results comparable to existing schemes –Directly into compact representation Reduced storage –Decoded at rendering time only Never stored ‘uncompressed’ First step in more general approach –Data amplification: Details at runtime from compact representation

Thank you

Compression value Output size 4k x 4k 8k x 8k16k x 16k 131k x 131k Uncompressed50 MB201 MB805 MB51 GB Our technique4 MB9 MB26 MB1.4 GB Exemplar size: 1024 x 1024 pixels Patch Size: 128 x 128 pixels Ratio improves as output gets larger Varies in this case from 12 up to 36

Compression Defauts des compression JPEG/PNG –Pas effectuable au rendu –Necessite de decompresser sur GPU Division en blocs –Compression maximale facteur 6 –Entraine des pertes significatives Or nous voulons –Facteur plus important que 6 –Pouvoir etre realise au rendu sur GPU

Calculating the compact size Output side size : S Exemplar side size : E Patch side size : P Overlap size : O Number of patches per side : n = S ÷ (P – O) Number of patches : nPatches = ceil (n) * ceil(n) Total size : total = nPatches*2 + 2*p*nPatches + e*e*3 + 2*(2*P-1)*2*nPatches

Run-time synthesis Tiles Combined to form a texture Synthesized at rendering time –Pixel-based Fine-scale variety Fast and parallel  Unique material  Complex to implement –Tile-based On-demand evaluation  Variety limited by number of tiles  Too many tiles if multiple materials  Distinctive features reveal tiling structure [Lefebvre and Hoppe 2005] [Cohen etal. 2003, Lefebvre and Neyret 2003, Wei 2004] Example

Run-time synthesis Tiles Combined to form a texture Pixel-based Fine-scale variety Fast  Still not fast enough  Unique material  Complex to implement Tile-based On-demand evaluation  Variety limited by number of tiles  Distinctive features reveal tiling structure ExemplarSynthesized [Lefebvre and Hoppe 2005] [Wei 2004]

[Perez et al 2003] Poisson image editing f* Ω g Poisson Image Editing f © Disney/Pixar 2003

Oblivion ©2006 Bethesda Softworks Bad transitions Distortion Repetitions

Multiple materials User guided Edge map Color or edge depending on material (In progress)

Steep patches Difficulty: –Adapt each step for steep patches (In progress)

Input size: 1024 x 1024 px Output size: 32K x 32K px Uncompressed output : 32K x 32K x 3 = 3 GB !!  Our algorithm : 90 MB !! Algorithm performance (1)

Contributions High quality texture synthesis –Results comparable to existing schemes Directly into compact representation –Reduced storage Decoded at rendering time only –Never stored ‘uncompressed’