Download presentation
Presentation is loading. Please wait.
1
Overview of Texture Synthesis Ganesh Ramanarayanan Cornell Graphics Seminar
2
The Problem small texture ??? big texture
3
A Simple Solution … small texture ??? big texture
4
??? big texture A Simple Solution … Tiling small texture
5
A Simple Solution … Tiling Tiling is obvious! big texture? small texture
6
Ideal Solution?
7
The Problem … in Words Given texture I, generate a texture J which – Looks like the same texture – Has no obvious copying or tiling from I – Difference between I and J should be the same as the way I “ differs from itself ” [DeBonet97] Things to watch for: – ‘ Looks the same ’ : what is the texture model? – ‘ Obvious copying ’ : how is it avoided? – Underlined text: indicates algorithm parameter
8
Classes of Algorithms Multiresolution pyramids – [HeegerBergen95] Pixel-by-pixel synthesis – [EfrosLeung99] Multiresolution pixel-by-pixel – [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01] Patch quilting – [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching – [WuYu04], [Liu et.al. 04]
9
Heeger Bergen 1995 Seminal paper that introduced texture synthesis to the graphics community Algorithm: – Initialize J to noise – Create multiresolution pyramids for I and J – Match the histograms of J ’ s pyramid levels with I ’ s pyramid levels – Loop until convergence – Can be generalized to 3D
10
Heeger Bergen 1995 - Algorithm Image pyramids – Gaussian – Laplacian Steerable pyramids [SimoncelliFreeman95] – b): multiple scales of oriented filters – c): a sample image – d): results of filters in b) applied to c)
11
Heeger Bergen 1995 - Results IJ SuccessesFailures IJ
12
Heeger Bergen 1995 - Results
13
Heeger Bergen 1995 - Verdict Texture model: – Histograms of responses to various filters Avoiding copying: – Inherent in algorithm No user intervention required Captures stochastic textures well Does not capture structure – Lack of inter-scale constraints
14
De Bonet 1997 Propagate constraints downwards by matching statistics all the way up the pyramid Feature vector: multiscale collection of filter responses for a given pixel Algorithm: – Initialize J to empty image – Create multiresolution pyramids for I and J – For each pixel in level of J, randomly choose pixel from corresponding level of I that has similar feature vector
15
De Bonet 1997 - Algorithm 6 feature vectors shown Notice how they share parent information
16
De Bonet 1997 - Results
17
De Bonet 1997 - Verdict Texture model: – Feature vector containing multiscale responses to various filters Avoiding copying: – Random choice of pixels with ‘ close ’ feature vectors, but copying still frequent on small scale Individual per-filter thresholds are cumbersome Feature vectors used in later synthesis work
18
Efros Leung 1999 Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood Algorithm: – Initialize J to empty – Synthesize new pixels in a spiral pattern – Select new pixel from I based on conditional probabilities given the current neighborhood in J and similar neighborhoods in I
19
Efros Leung 1999 - Algorithm IJ ‘ causal ’ neighborhood
20
Efros Leung 1999 - Results
21
Efros Leung 1999 – Verdict Texture model: – MRF Avoiding copying: – MRF Neighborhood size = largest feature size Markov model is surprisingly good – “ I spent an interesting evening recently with a grain of salt. ” Search is very slow with large neighborhoods
22
Wei Levoy 2000 Adds tree-structured vector quantization speedup and multiresolution matching Algorithm: – Initialize J to noise – Synthesize new pixels in scanline order – Select new pixel from I that has the closest matching neighborhood to J – In multiresolution case, look instead at neighborhood of feature vectors
23
Wei Levoy 2000 - Algorithm Scanline synthesis with neighborhoods – a) input texture – b) algorithm start – c) midway point – d) ending
24
Wei Levoy 2000 - Results
25
Wei Levoy 2000 - Verdict Texture model: – Multiscale filters and MRF Avoiding copying: – Not explicitly, but in practice inherently avoided; TSVQ adds additional unpredictability Results comparable to [EfrosLeung99] TSVQ speedup comes at a price (quality loss)
26
Ashikhmin 2001 Adds more verbatim image copying and user- constrained synthesis Algorithm: – New array J* stores location in I of J ’ s pixels – Initialize J to empty, J* to random – Synthesize new pixels in scanline order – Select new pixel by copying from patches of I, as determined by locations in J* – In user-constrained case, also include error comparison with user-specified target image T
27
Ashikhmin 2001 - Algorithm J* x=3 y=4 x=4 y=4 x=12 y=11 x=17 y=16 Candidates: -x=4, y=5 -x=18, y=16 -x=11, y=12 x=18 y=16
28
Ashikhmin 2001 - Results
29
Ashikhmin 2001 - Verdict Texture model: – MRF Avoiding copying: – Actually, here it is encouraged on a small scale, but in practice it doesn ’ t occur on a large scale Recognized that copying preserves fine detail Features can get ‘ cut off ’ because of the abrupt jumping from patch to patch
30
Hertzmann et. al. 2001 Framework for solving image analogies: learning an image filter f by example – Given A, f(A), and B, generate f(B) To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J Algorithm: – Bring image histograms into correspondence – Combine [WeiLevoy00] and [Ashikhmin01] Each search method is weighted appropriately and then the best choice is used
31
Hertzmann et. al. 2001 - Algorithm A provides set of constraints Histogram matching critical because of color differences between A and B sets of images
32
Hertzmann et. al. 2001 - Results IJIJ
34
Hertzmann et. al. 2001 - Verdict Texture model: – Multiscale filters and MRF Avoiding copying: – See [WeiLevoy00] and [Ashikhmin01] Unique approach to image manipulation and texture synthesis Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]
35
Efros Freeman 2001 Inspired by chaos mosaics [Xu et. al. 00] Make an ‘ image quilt ’ by pasting together patches of the input and fixing seams Algorithm: – Initialize J to empty – Copy new patches of certain size in scanline order, with fixed overlap width – Randomly select new patch that has error in overlap region less than given threshold – Copy pixels based on minimal error boundary cut
36
Efros Freeman 2001 - Algorithm Input texture block B1 B2 Neighboring blocks constrained by overlap B1B2 Minimal error boundary cut
37
Efros Freeman 2001 - Results
38
Efros Freeman 2001 - Verdict Texture model: – ?? Avoiding copying: – Randomized patch selection, but still noticeable Patch size is a hard parameter to understand Results are surprisingly good given algorithm Multiscale goes on a brief hiatus
39
Kwatra et. al. 2003 Generalizes seam computation in overlap regions as a graph cut problem – Based on [Boykov et. al. 99] (with Ramin Zabih) Algorithm: – Initialize J to empty – Copy pieces of I to J using a variety of methods – Formulate flow graph in overlap region based on errors and compute minimum cut – Copy sink-side pixels to J – Variety of strategies to further hide seams
40
Kwatra et. al. 2003 - Algorithm (assume cut region is 3x3 for simplicity)
41
Kwatra et. al. 2003 - Results
43
Kwatra et. al. 2003 - Verdict Texture model: – MRF Avoiding copying: – Even with a multitude of patch selection methods, still noticeable when it happens repeatedly Paper presents a bag of synthesis tricks without much intuition for when to use what Graph cut formalization is useful and powerful
44
Wu Yu 2004 Synthesizes a feature image J* in parallel Tries to match features in overlap region Algorithm: – Compute feature image I* – Initialize J* to empty – Synthesize J and J* in parallel – When searching for new patch, add in weighted feature matching error – Prior to copying, warp selected patch to match features exactly
45
Wu Yu 2004 - Algorithm Match feature pixels based on – Tangent orientation – Distance – Bijectivity Example: – L 1 out matches L 1 in – L 2 out matches L 3 in – Warping will move matching features closer to each other Feature correspondence
46
Wu Yu 2004: Results
47
Wu Yu 2004: Verdict Texture model: – Feature lines and MRF Avoiding copying: – No; in fact, mirroring is often very noticeable Unclear how to combine this metric with others Insight to use warping is valuable, although the next paper does a much better job with it
48
Liu et. al. 2004 Regular textures: can be tiled Near-regular textures: statistical distortion of a regular texture Algorithm (simplified): – Mark lattice points in I – Transform I into regular texture R – Express transformation as deformation texture I* – Synthesize larger deformation J* – Synthesize J by combining J* with R
49
Liu et. al. 2004 - Algorithm RI Deformation Texture I*
50
Liu et. al. 2004 - Results
51
original texture transfer
52
Liu et. al. 2004 - Verdict Texture model: – Regular texture with modulated geometry and color Avoiding copying: – Depends on the synthesis algorithm Proposes specific model to handle specific types of textures Interestingly, this method depends on previous texture synthesis methods
53
Classes of Algorithms - Recap Multiresolution pyramids – [HeegerBergen95] Pixel-by-pixel synthesis – [EfrosLeung99] Multiresolution pixel-by-pixel – [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01] Patch quilting – [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching – [WuYu04], [Liu et.al. 04]
54
Evolution of Texture Model Responses to image filters / decompositions Markov Random Fields Copies of small pieces of image Feature data Various combinations of the above
55
What ’ s Next? Texture synthesis is really hard Rather than creating a silver bullet, focus on algorithms for more specific problems Superresolution: + = ???
56
What ’ s Next? Texture synthesis is really hard Rather than creating a silver bullet, focus on algorithms for more specific problems Superresolution: + = [Hertzmann et. al. 01]
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.