Image Quilting and Apples By Zach Broderick A presentation of the paper “Image Quilting for Texture Synthesis and Transfer” by Alexei A. Efros and William T. Freeman

Texture Synthesis (Images taken from paper) The concept of image-based texture synthesis is to create an image of unlimited size from a smaller source image. We want this new image to “look the same” as the source image, as well as maintain its resolution, not repeat, and be seamless even if the edges of the original don’t line up. (Images taken from paper)

Output texture is: Infinitely Large Output Sample of Texture Same resolution Non-repeating Seamless Must be perceived as “same”

Previous Methods Tiling (bad seams, repetitive) Stochastic (doesn’t work for structured) Pixel-based (SLOW, poor for structured) Random Patches/Blending (poor seams)

Image Quilting Copy patches (like the random patch method) based on neighborhood (like pixel method) Faster (predetermined pixels/less search space), maintains structure of regular textures Seams smoothed with graph cut through least error path (Image from Wikipedia)

Image Quilting (Diagram from paper) The idea of “perceiving” an image to be the same as another is related to certain statistics of the image, namely Markov Fields, which the paper does not get into. These statistics must be transferred from the source to destination images in order to maintain “sameness”. Previous methods used a variety of less successful techniques, such as copying the source image pixel by pixel based on the surrounding neighbor pixels. This was slow and had trouble with more structured textures, though it did okay with stochastic textures. There were also fractal based methods, as well as copying random blocks and doing a second pass to smooth the seams. A faster way that maintains the integrity of the original image better is “quilting”, of which this paper is one of the first to introduce back in 2001. Quilting involves copying blocks of the source image to the destination image with a slight overlap in pixels. Block selection is made based on least mean squared error, and not the entire block is copied—a least-error path is carved through the overlap region in order to minimize the appearance of seams, as seen above. (Diagram from paper)

An In-Depth Look at the Image Quilting Algorithm: Overview Copy tiles from the source image to the destination image in a left to right, top to bottom fashion, with an overlap of about 1/6 the tile size on the top and left regions. In selecting each of these tiles, search the source image space S for the tile with the least mean squared error in the overlap region (the first tile can be selected at random). When copying each tile, calculate the minimum error path cut in each of the overlap regions, and only copy the portion of the tile within the cut.

Tiles are copied from left to right, top to bottom (though this is not a requirement). Notice the overlap in the top and left regions of the tile. It is from this overlap that we calculate the mean squared error and make the graph cut in order to minimize the appearance of seams. Copying a whole tile maintains the structure of the original image.

Error Calculation First, we search the source space, which is every possible tile that can be fit in the source image (which is a lot, so it is slow). For each of these possible tiles, we compute the total error and choose a tile at random from the group with the least errors. Error is computed by subtracting the red, green, and blue color values (squared) of each of the pixels in the overlap region.

Path Cut In order to obtain more optimal seams, we will not copy the entire source tile, but rather cut a path through the overlap region along the minimum error line and only copy the pixels within those cuts.

Path Cut Cont’d… We start with a potential path for each column in the left overlap (or row in the top, in this case they are both 4). Starting with the first pixel, we calculate the error as before. Then we move up (or left) a row (or column), and compare the errors of the pixels above the previous one to the left, center, and right. We choose the one with the lowest error and add that one to our path, and repeat for the next row (or column). For the corner, we simply merge the horizontal and vertical paths.

Texture Transfer Algorithm can be generalized to provide texture transfer capabilities (image from paper)

Problems Repetition often occurs (see raspberries) Occasional glaring seams (mutant tomatoes)

Demonstration

Questions?