1. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Hybrid Texture Synthesis Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt

2. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 2D Texture Synthesis n x m Input Texture N x M Output Texture ► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.

3. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Our Inspiration ► Existing (and Impressive) Technology Pixel-Based: ─Non-parametric Sampling [Efros and Leung 1999] ─Tree-structured Vector Quantization [Wei and Levoy 2000] ─Synthesizing Natural Textures [Ashikhmin 2001] ─Image Analogies [Hertzmann et al. 2001] Patch-Based: ─Patch-Based Sampling [Guo et al. 2001] ─Image Quilting [Efros and Freeman 2001] ─Hierarchical Pattern Mapping [Soler et al. 2002]

4. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Our Inspiration ► Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring

5. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Our Inspiration ► Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring Wei/Levoy Algorithm: Can occur when using structured Textures with rich histograms. This is due to the L 2 norm.

6. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Our Inspiration ► Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring Patch-Based Sampling Algorithm: Can occur when texture features are not well-aligned across patch boundaries

7. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Our Inspiration ► Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring ► Is it possible to reduce all deficiencies for an even larger class of textures ?

8. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput Step n in Algorithm

9. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput Build L-shaped Neighborhood

10. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput Find Matching Neighborhood(s) in Input

11. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput Select Best Neighborhood from all Candidates

12. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput Copy Selected Pixel to Output

13. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput Repeat for all Pixels in this Row...

14. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis InputOutput...and all other Output Rows

15. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Pixel-Based Texture Synthesis ► Main difference between Efros/Leung (EL), Wei/Levoy (WL) and Ashikhmin (AS): Per- Pixel Search Strategy. ► Ashikhmin’s insight: “WL tends to produce textures with smoothed out edges in some cases.” ► Ashikhmin’s solution: consider only coherent pixels in the search, which encourages region growing. ► Problem: inconsistencies can still occur for highly-structured textures.

16. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Step n in Algorithm

17. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Patch to be Synthesized in this Step

18. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Build Neighborhood (Image Mask)

19. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Find Matching Neighborhood(s) in Input

20. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Select Best Neighborhood from all Candidates

21. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Copy Selected Patch to Output

22. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Apply some Overlap Repair Strategy...

23. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput... And we are done with this patch

24. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis InputOutput Repeat for all Patches in Output

25. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis Generalization of Pixel-Based Synthesis ► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy. ► IQ performs a minimum-error-boundary- cut (MEBC) through the error surface (least cost path; Dijkstra). ► PBS feathers the overlap region (alpha- blending). ► Pro: patch-based algorithms are good at maintaining global structure. ► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).

26. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis Generalization of Pixel-Based Synthesis ► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy. ► IQ performs a minimum-error-boundary- cut (MEBC) through the error surface (least cost path; Dijkstra). ► PBS feathers the overlap region (alpha- blending). ► Pro: patch-based algorithms are good at maintaining global structure. ► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).

27. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch-Based Texture Synthesis Generalization of Pixel-Based Synthesis ► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy. ► IQ performs a minimum-error-boundary- cut (MEBC) through the error surface (least cost path; Dijkstra). ► PBS feathers the overlap region (alpha- blending). ► Pro: patch-based algorithms are good at maintaining global structure. ► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).

28. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Hybrid Texture Synthesis Idea ► Combine best aspects from other approaches while avoiding (or improving over) known pitfalls: Patch-based algorithms ─ Are good at preserving global structure ─ Can introduce artifacts along patch boundaries Pixel-based algorithms ─ Preserve local coherence (MRF model: local and stationary) ─ Possibly fail to preserve global structure ─ This is especially problematic for textures with rich histograms and many high frequency features, due to the smoothing nature of the distance metric (L 2 norm)

29. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Hybrid Texture Synthesis (HTS) Algorithm ► Hybrid, two-fold approach to texture synthesis Adaptive patch sampling [Soler et al. 2002] ─ Start with a uniform quadrilateral grid of patches (the output) ─ Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoff  max ) Overlap re-synthesis (novel overlap repair strategy) ─ Mark invalid pixels in the overlap region ─ Validity map is defined by the error surface E(x)=(x src – x dst ) 2 and a user-defined tolerance parameter  max ─ Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy

30. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Adaptive Patch Sampling Principle ► Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002] ► Analogy: Assembling a quadtree puzzle  p >  max  Split Patch Synthesize White Patch  p <  max

31. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Adaptive Patch Sampling Trade-off ► Trade-off between preserving global structure and avoiding detail artifacts  max = 1.0  max = 0.01  max = 0.04

32. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Adaptive Patch Sampling Trade-off ► Trade-off between preserving global structure and avoiding detail artifacts  max = 1.0  max = 0.01  max = 0.04

33. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Adaptive Patch Sampling Trade-off ► Trade-off between preserving global structure and avoiding detail artifacts  max = 1.0  max = 0.01  max = 0.04

34. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Adaptive Patch Sampling Trade-off ► Trade-off between preserving global structure and avoiding detail artifacts  max = 1.0  max = 0.01  max = 0.04

35. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Synthesizing a single patch Synthesize Black Patch i

36. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Extracting the image mask Synthesize Black Patch i ► Grow patch by overlap (toroidally)

37. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Extracting the image mask Image Mask (I i )Synthesize Black Patch iBinary Support (J i ) ► Extract image mask (I i ) and binary support (J i ), and circularly shift to upper left corner

38. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Computing the error image Image Mask (I i )Input Texture (T)Binary Support (J i ) ► Compute error E i (x 0 ) between I i and T for each circular shift x 0 =(  x,  y) of the input texture T

39. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Computing the error image Error Image (E i )Input Texture (T)

40. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Selecting a Patch Error Image (E)Input Texture (T)Selected Patch (P)

41. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 In Search of Good Patches Selecting a Patch Error Image (E)Input Texture (T)Selected Patch (P) ► Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.

42. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Overlap error and pixel invalidation Patch and Image MaskSelected Patch (P)

43. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Overlap error and pixel invalidation Image Mask (I)Selected Patch (P)

44. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Overlap error and pixel invalidation Image Mask (I)Selected Patch (P) Error Surface (S)

45. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Overlap error and pixel invalidation Patch and Image MaskP with Invalid Pixels (blue) Error Surface (S) ► Trade-off: user-defined  max is the pixel error tolerance. Setting to 1 results in pure feathering.

46. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Compositing Intermediate ResultPatch with Invalid Pixels P OVER R i-1

47. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Compositing Composited ResultPatch with Invalid Pixels ► Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region ► Solution: alternative ordering (Pixel Traversal Map)

48. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited ResultPatch with Invalid Pixels Pixel Traversal Map (M) ► Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).

49. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited ResultPatch with Invalid Pixels Pixel Traversal Map (M) ► Pixel Traversal Map: step 1

50. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited ResultPatch with Invalid Pixels Pixel Traversal Map (M) ► Pixel Traversal Map: step 2...

51. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited ResultPatch with Invalid Pixels Pixel Traversal Map (M) ► Pixel Traversal Map:... step n

52. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited ResultPatch with Invalid Pixels Pixel Traversal Map (M) ► Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.

53. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Pixel Traversal Map (M) ► Synthesize red pixel from valid (or already re- synthesized) pixels Intermediate Result

54. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Image Mask (I j ) Binary Support (J j ) ► Extract image mask (I j ) and binary support (J j ) for best- pixel search in the input texture. Complexity: O(n log n) per pixel. Intermediate Result

55. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate ResultPixel Traversal Map (M)

56. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate Result Shifted Pixel Traversal Map (M)

57. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate Result with Traversal Map Pixel Traversal Map (M)

58. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Step 1 Pixel Traversal Map (M)

59. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Step 2 Pixel Traversal Map (M)

60. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Step 3 Pixel Traversal Map (M)

61. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Result Pixel Traversal Map (M)

62. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

63. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

64. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

65. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

66. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

67. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

68. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

69. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

70. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

71. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

72. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

73. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

74. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

75. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Overlap Repair Comparisons No RepairHTSPBSIQ

76. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

77. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

78. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

79. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

80. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

81. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

82. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input Efros/LeungWei/Levoy IQPBSHTS

83. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input PBSHTS PBSHTS Input

84. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input PBSHTS PBSHTS Input

85. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results Synthesis Comparisons Input PBSHTS PBSHTS Input

86. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Results

87. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Conclusions and Future Work ► Improve Computational Complexity Pixel neighborhoods of patch and pixel stage are not known a priori, so precomputation is not straightforward We can solve this in the pixel stage by employing k- coherence search [Tong et al. 2002] [Ashikhmin 2001] ► Improve Error Metric Still using the L 2 norm due to its simplicity Develop a metric which takes feature mismatch into account Texton approach looks promising [Zhang et al. 2003]

88. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Questions ? ► Contact Information Andrew Nealen andy@nealen.com Marc Alexa alexa@informatik.tu-darmstadt.de http://www.dgm.informatik.tu-darmstadt.de Matlab code and extended Thesis: http://www.nealen.com