Pyramid-Based Texture Analysis / Synthesis Presented By : Keren Kahanov

Texture Synthesis Generation of much texture as desired so that any object can be covered Input texture Output size Output texture The output texture is synthesized image that match the texture appearance of the given digitized image

General Techniques Texture mapping – repeating the texture Can lead to unacceptable artifacts in visible seams , visible repetition Distortion in map to 3-D – often no natural map from planar texture image to the geometry/ topology of the surface. (There are some partial solutions , but no universal solution for mapping an image onto arbitrarily shaped surface.)

General Techniques (cont.) Create (paint) textures by hand directly onto the 3-d surface model Very labor intensive Requires considerable artistic skills Computer-synthesized textures as much texture can generated as needed Some of the synthesis techniques produce textures that tile seamlessly.

Synthetic textures The distortion problem has been solved in two different ways: Some techniques work by synthesizing texture directly on the object surface. The second solution is to use solid textures (no mapping no distortion)

Heeger & Bergen Pyramid-Based Texture Analysis / Synthesis Our lecture today A technique for synthesizing an image (or solid texture) that matches the appearance of a given texture sample. The output image (at any size) looks like the original image in its color and texture properties. The texture is synthesized so that it tiles seamlessly.

Heeger & Bergen (cont.) First order statistics (histograms) Focuses on the synthesis of stochastic textures deterministic textures: characterized by a set of primitives and a placement rule (e.g., a tile floor ,bricks). stochastic textures : do not have easily identifiable primitives (e.g., granite, bark, sand). Many real-world textures have some mixture of these two (e.g. woven fabric, plowed fields).

Heeger & Bergen’s Algorithm The texture-matching algorithm relies on two main components: a self-inverting steerable pyramid a histogram matching process.

Pyramids - Reminder Reminder from “image processing” course by Dr. Hagit Hel-Or

Pyramids – Reminder (cont.)

(1) Steerable Pyramid a linear multi-scale, multi-orientation image decomposition developed in order to overcome the limitations of orthogonal separable wavelet decompositions The steerable pyramid, unlike wavelet, is non-orthogonal and overcomplete Overcomplete = the number of pixels in the pyramid is much greater than the number of pixels in the input image

Steerable Pyramid (cont.) The basis functions of the steerable pyramid are directional derivative operators, that come in different sizes and orientations translation-invariant (i.e., the subbands are aliasing-free, or equivariant with respect to translation) Translating the image the transform translates energy in each band is preserved (not in wavelet)

Aliasing Aliasing Properly sampled In steerable pyramid only the low-pass band is subsampled This is done to minimize the amount of aliasing within each subband. Avoiding aliasing is critical because the algorithm treats each subband independently. Aliasing Properly sampled

Steerable Pyramid (cont.) rotation-invariant = steerable (i.e., the subbands are steerable, or equivariant with respect to rotation) rotating the image the transform rotates A set of filters form a steerable basis if (1) they are rotated copies of each other, and (2) a copy of the filter at any orientation may be computed as a linear combination of the basis

Steerable filters example 2nd derivative of Gaussian Linear combination filters: Original Image: Same Linear combination outputs:

Steerable Pyramid (cont.) A self-inverting pyramid: it provides a complete representation of an image original image can be reconstructed by applying the same filters used to generate the pyramid A steerable pyramid can capture the variation of a texture in both intensity and orientation

Steerable Pyramid Build Original image Steerable Pyramid Build Needs 7 filters:

Steerable Pyramid Build (cont.) Original image 4 oriented filters The steerable pyramid

Steerable Pyramid- Fourier domain The steerable pyramid performs a polar-separable decomposition in the frequency domain, thus allowing independent representation of scale and orientation 1 4 3 2 1 4 2 3 3 2 4 1 wavelet

(2) Histogram Matching A generalization of histogram equalization. modify an image (im1) to have the same histogram as another image (im2). Using two lookup tables: the cumulative distribution function (cdf) of one image. The inverse cumulative distribution function (inverse cdf) of the other image

Cdf / inverse Cdf The cdf maps from the interval [0,256] to the interval [0,1]. (cdf = accumulated histogram) The inverse cdf maps back from [0,1] to [0,256]. constructed by resampling (with linear interpolation) the cdf so that its samples are evenly spaced on the [0,1] interval. Note! the values in the noise and target images need not lie in the range[0,256]

Histogram Matching Algorithm

התאמת ההיסטוגרמה של תמונה זו לזו של תמונה זו Histogram matched תוצאה סופית

Image 1 histogram matched to image 2 histogram

Enough with all the talks about Pyramids & Histograms !!! It’s time to learn the real thing – Heeger & Bergen’s Algorithm for Texture Analysis / Synthesis

Heeger & Bergen’s Algorithm

The Algorithm - steps Input1 = sample texture image Input2 = sample of uniform random noise ( noise image) Its dimensions = desired size of output image Match noise image’s histogram to the input sample’s histogram. ( modified noise image) The texture sample is decomposed into an oriented bandpass pyramid

The Algorithm – steps (cont.) Loop for several iterations: Make pyramid from the modified noise image Histogram matching process is employed between corresponding subbands of the two pyramids. The noise pyramid is collapsed The histogram matching process is again applied to the new noise image and the texture sample After a few iterations, the output image, which began as random noise, begins to resemble the input texture.

When are two textures similar? Textures appear visually similar if… they have similar distributions of filter outputs*. (Filters : orientation and spatial-frequency) FROM: “Statistical Texture Synthesis” Presented by:Zeev Litichever & Tal Hassner , Weizmann Institute

Laplassian \ steerable pyramid FROM: “Statistical Texture Synthesis” Presented by:Zeev Litichever & Tal Hassner , Weizmann Institute

Using histograms for synthesis Output image with similar distributions to sample. Iteratively coarse noise image to have sample histogram via histogram matching. Output pyramid Noise image of output size: Apply filters And create histograms FROM: “Statistical Texture Synthesis” Presented by:Zeev Litichever & Tal Hassner , Weizmann Institute

Results Input1 = texture image Input2 = noise image Output

More Results In each pair : left= input (original) , right =output (synthesized)

Edge Handling Proper edge handling in the convolution operations is important. synthesis pyramid: use circular convolution. Given that the synthesis starts with a random noise image, circular convolution guarantees that the resulting synthetic texture will tile seamlessly. (synthesis pyramid = noise pyramid)

Edge Handling (cont.) analysis pyramid: circular convolution would typically result in spuriously large filter responses at the image borders. This would, in turn, introduce artifacts in the synthesized texture. (analysis pyramid = original image pyramid) A reasonable border handler for the analysis pyramid is to pad the image with a reflected copy of itself. Reflecting at the border usually avoids spurious responses.

Color Heeger and Bergen’s Algorithm can be applied for both : gray level texture images color texture images

Color Space RGB RGB components of a typical image are not independent cannot directly apply the algorithm to each channel individually. R G correlation Looking for B ? B is on a vacation , in another dimension (helped to simplify the graph)

Color Space Transform First, transform the color space of the input texture to independent channels SVD algorithm : The decorrelation transform is :

Color Space Transform (cont.) Diagonal matrix G No correlation in transformed color space R

New Color Space each independent channel can be processed separately. After processing, the channels are transformed back into RGB values, using yielding the synthesized texture.

Solid Textures The algorithm can also be used to make isotropic 3-d solid textures. Same as before, except for the choice of pyramid: use a 2-d Laplacian pyramid for analysis and a 3-d Laplacian pyramid for synthesis.

Solid Textures - examples

Texture Mixtures Texture mixtures synthesized by choosing the color palette from one image and the pattern from a second image.

Texture Mixtures (cont.) tiling different textures as input to Heeger and Bergen’s algorithm Output: combine the properties of multiple textures.

Algorithm – no. iterations An Implementation of Heeger and Bergen’s Texture Analysis/Synthesis Algorithm by Thomas F. El-Maraghi The human observer would find it difficult, or even impossible, to discern any change after the 4th iteration.

No. of orientation The effect of changing the number of oriented subbands in the pyramid on the synthesized texture. the synthesized texture looks more like the original as the number of orientations is increased. original 1 orientation 2 orientations 4 orientations

Homogenous / Inhomogoneous input Homogoneous input texture success Inhomogoneous input texture produces blotchy synthetic texture failure

Failures

More Failures

summery In their paper entitled “Pyramid-Based Texture Analysis/Synthesis” , Heeger and Bergen present an algorithm for generating synthetic copies of stochastic textures. The algorithm is based on a self-inverting oriented bandpass pyramid and a histogram matching technique The process is tantalizing both for the results it produces and for its intrinsic simplicity. Given a texture sample, either grayscale or color, the algorithm can synthesize as much of it as desired.

Upsides Fast Simple Produces tileable textures No parameters (requires only the “target” texture as input) You just crop a textured region from a digitized image and run a program to produce as much of that texture as you want. Produces tileable textures Works well on stochastic textures

Downsides Doesn’t work well other textures Not guaranteed to converge For example : inhomogeneous texture, quasi-periodic textures and on random mosaic textures… Not guaranteed to converge

References Based on articles : Pyramid-Based Texture Analysis/Synthesisby David J. Heeger & James R. Bergen An Implementation of Heeger and Bergen’s Texture Analysis/Synthesis Algorithm by Thomas F. El-Maraghi More Referances: THE STEERABLE PYRAMID: A FLEXIBLE ARCHITECTURE FOR MULTI-SCALE DERIVATIVE COMPUTATION by Eero P Simoncelli & William T Freeman “image processing” course ,Haifa University, by Dr. Hagit Hel-Or “Advanced Topics in Computer Vision“ course, Spring 2003, Weizmann Institute http://people.csail.mit.edu/billf/steer.html http://www.cns.nyu.edu/~eero/steerpyr/ http://noobeed.com/nb_ex_image_histmatch.htm www.dfanning.com/documents/programs.html#HISTOMATCH

THE END