1. Synthesis of Compact Textures for real-time Terrain Rendering Nader Salman 22 juin 2007 Encadrant : Sylvain Lefebvre

2. Motivations Need for large textures –Simple repetition: low quality Texture synthesis from example –Pre-computation: Need to store large result Limited memory! –Image Compression: Rendering: Hinders sequential coding Block based: Fixed rate (÷ 6) –Our approach : Synthesize compact textures, that can be efficiently decoded at rendering time [Garber 1981, Popat & Picard 1993, Efros & Leung 1999, Wei & Levoy 2000, Ashikhmin 2001, Hertzmann et al 2001, Tong et al 2002 …]

3. Our approach Inspired by patch-based synthesis –Patch-based Synthesis [Praun et al 2000, Liang et al 2001, Efros and Freeman 2001, Kwatra et al 2003]

4. Our approach Inspired by patch-based synthesis –Record patch positioning and stitching (instead of final image) –Decode at rendering time Main difficulties: –Adapt synthesis for compact encoding –Prepare for efficient decoding on GPU

5. Exemplar Positioning 12 Cuts 3 Stitch Our approach

6. A first example Memory size: –Full image: 805 MB ( result size: 16K x16K ) –Our encoding: 26 MB ( patch size: 128x128 ) Exemplar (1024x1024) Synthesis Result Compact representation

7. Overview Introduction Our approach –Patch positioning –Optimal cuts –Seam removal Results and performance Future Work –Multiple materials –Slope –GPU Decoding

8. Our approachExemplar Positioning 1 2 Cuts 3 Stitch 1. Positioning Exemplar Scan exemplar Choose best fit patch Place patch Memorize position ? Patch position Compact representation

9. Exemplar Positioning 1 2 Cuts 3 Stitch Our approach

10. Why do we cut?

11. Graph cut 147 258 369 147 258 369 A B Construct graph such that: Graph Mincut  Best Seam A ∞ ∞ ∞ B ∞ ∞ ∞ 14 25 36 7 8 9 Potential Seam Cost Constraint 20 100 30 10 15 85 40 35 14 25 36 7 8 9 A B 20 30 10 15 Seam 1 25 36 47 8 9 A B [Kwatra 2003]

12. Graph cut Problem with encoding: –Cut can be arbitrary  Expensive to store! Our approach –Constrain cuts to be 1D functions –Still search for optimal! 0 0 2 2 2 2 2 1 1 1 1 0

13. Our approachExemplar Positioning 12 Cuts 3 Stitch 2. Optimal Cuts Current output A B B Constrained arcs Bidirectional arcs One D. Seam Horizontal Overlap Image 00022221100 Vertical Overlap Image 22211011 Compact representations

14. Exemplar Positioning 12 Cuts 3 Stitch Our approach

15. Why stitching ? Seams might still be visible in some cases  Avoid visible discontinuities

16. Our approach 3. Stitching New patchExisting texture Poisson Image Editing Current position Mask [Perez et al. 2003]

17. Encoding of Poisson field (1) Before PoissonAfter Poisson Deformation Field (x8 for visualisation)

18. Encoding of Poisson field (2) Problem: –Deformation is an image as big as output! Approx ?

19. Compression: –Per patch Approximate as sum of 2 fields Each is a ‘flattened 1D pyramid’ –Optimize with least square argmin || Ax - b || 2 –Fast reconstruction on the GPU 1D pyramid = 1D MIP-map Encoding of Poisson field (3) High Frequency Smooth

20. Encoding of Poisson field (4) pSNR = 43,7 dB Output with PoissonApproximated Output High Frequencies

21. Output with Poisson pSNR = 50,2 dB pSNR = 43,2 dB pSNR = 46,12dB Approx. outputs

22. Overview Introduction Our approach –Patch positioning –Optimal cuts –Seam removal Results and performance Upcoming Work –Multiple materials –Slope –GPU Decoding

23. Algorithm performance Patch size (pixels) Output size (MB) Patch Size 16x16 Output 2,4 GB Patch Size 32x32 Output 542 MB Patch Size 128x128 Output 90 MB Input size: 1024 x 1024 Output size: 32K x 32K 32K x 32K  3GB uncompressed

24. Quick comparison Our approach [Kwatra 2003]

25. Overview Introduction Our approach –Patch positioning –Optimal cuts –Seam removal Results and performance Upcoming Work –Multiple materials –Slope –GPU Decoding

26. Conclusion Need for detailed large textures –Limited memory Contributions –High quality texture synthesis Results comparable to existing schemes –Directly into compact representation Reduced storage –Decoded at rendering time only Never stored ‘uncompressed’ First step in more general approach –Data amplification: Details at runtime from compact representation

27. Thank you

67. Compression value Output size 4k x 4k 8k x 8k16k x 16k 131k x 131k Uncompressed50 MB201 MB805 MB51 GB Our technique4 MB9 MB26 MB1.4 GB Exemplar size: 1024 x 1024 pixels Patch Size: 128 x 128 pixels Ratio improves as output gets larger Varies in this case from 12 up to 36

68. Compression Defauts des compression JPEG/PNG –Pas effectuable au rendu –Necessite de decompresser sur GPU Division en blocs –Compression maximale facteur 6 –Entraine des pertes significatives Or nous voulons –Facteur plus important que 6 –Pouvoir etre realise au rendu sur GPU

69. Calculating the compact size Output side size : S Exemplar side size : E Patch side size : P Overlap size : O Number of patches per side : n = S ÷ (P – O) Number of patches : nPatches = ceil (n) * ceil(n) Total size : total = nPatches*2 + 2*p*nPatches + e*e*3 + 2*(2*P-1)*2*nPatches

70. Run-time synthesis Tiles Combined to form a texture Synthesized at rendering time –Pixel-based Fine-scale variety Fast and parallel  Unique material  Complex to implement –Tile-based On-demand evaluation  Variety limited by number of tiles  Too many tiles if multiple materials  Distinctive features reveal tiling structure [Lefebvre and Hoppe 2005] [Cohen etal. 2003, Lefebvre and Neyret 2003, Wei 2004] Example

71. Run-time synthesis Tiles Combined to form a texture Pixel-based Fine-scale variety Fast  Still not fast enough  Unique material  Complex to implement Tile-based On-demand evaluation  Variety limited by number of tiles  Distinctive features reveal tiling structure ExemplarSynthesized [Lefebvre and Hoppe 2005] [Wei 2004]

72. [Perez et al 2003] Poisson image editing f* Ω g Poisson Image Editing f © Disney/Pixar 2003

73. Oblivion ©2006 Bethesda Softworks Bad transitions Distortion Repetitions

74. Multiple materials User guided Edge map Color or edge depending on material (In progress)

75. Steep patches Difficulty: –Adapt each step for steep patches (In progress)

76. Input size: 1024 x 1024 px Output size: 32K x 32K px Uncompressed output : 32K x 32K x 3 = 3 GB !!  Our algorithm : 90 MB !! Algorithm performance (1)

77. Contributions High quality texture synthesis –Results comparable to existing schemes Directly into compact representation –Reduced storage Decoded at rendering time only –Never stored ‘uncompressed’