1. Graphcut Texture: Image and Video Synthesis Using Graph Cuts
Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk, Aaron Bobick
Georgia Institute of Technology
SIGGRAPH 2003
林柏志
2008/11/25

2. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Outline
Introduction
Patch Fitting using Graph Cuts
Patch Placement & Matching
Extensions & Refinements
Discussion
Summary
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

3. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Introduction
Texture:
An infinite pattern that can be modeled by a stationary stochastic process.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

4. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Introduction
Goal of texture synthesis:
be restated as the solution for the nodes of the network, that maximizes the total likelihood.
Probabilistic inference in graphical models is cyclic network problem (NP Hard)
approximation to the optimal solution
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

5. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Introduction
Two decision for each patch:
the offset of the patch (offset searching)
the patch seam (seam finding)
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

6. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Introduction
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

7. Patch Fitting using Graph Cuts
Determine an optimal portion of selected patch is computed and only these pixels are copied to the output image
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

8. Patch Fitting using Graph Cuts
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

9. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Graph Cuts
Let s and t be two adjacent pixel positions in the overlap region.
Let A(s) and B(s) be the pixel colors at the position s in the old and new patches.
Define the matching quality cost M between the two adjacent pixels s and t that copy from patches A and B.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

10. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Graph Cuts
The arcs connecting the adjacent pixel nodes s and t are labelled with the matching quality cost M(s, t,A,B).
Add arcs that have infinitely high costs between some of the pixels and the nodes A or B are constraint arc.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

11. Accounting for Old Seams
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

12. Accounting for Old Seams
Seam node M(s, t,As,At ) where s and t are the two pixels that straddle the seam.
If the arc between a seam node and the new patch node B is cut, this means that the old seam remains in the output image.
If such an arc is not cut, this means that the seam has been overwritten by new pixels.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

13. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Surrounded Regions
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

14. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Surrounded Regions
All of the pixels in a border surrounding the placement region are constrained to be reflected in the arcs from the border pixels to node A.
Placed a single constraint arc from one interior pixel to node B in order to force at least one pixel to be copied from patch B.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

15. Patch Placement & Matching
Algorithms for picking candidate patches.
Restrict the patch selection to previously unused offsets.
Use the cost of existing seams to quantify the error in a particular region of the image, and pick the region with the largest error as an error-region.
To pick those patch locations that completely overlap the error-region.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

16. Patch Placement & Matching
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

17. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

18. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Selection methods
Random placement
Entire patch matching
Sub-patch matching
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

19. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Random placement
The new patch is translated to a random offset location.
This is the fastest synthesis method and gives good results for random textures.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

20. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Entire patch matching
To account for partial overlaps between the input and the output.
Normalize the sum-of-squared-differences (SSD) cost with the area of the overlapping region.
This method gives the best results for structured and semi-structured textures
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

21. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Entire patch matching
C(t) is the cost at translation t of the input.
I and O are the input and output images.
At is the portion of the translated input overlapping the output.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

22. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Entire patch matching
σ is the SD. of the pixel values in the input image.
k controls the randomness in patch selection.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

23. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Sub-patch matching
Pick output-sub-patch which is the same or slightly larger than the error-region.
Look for a sub-patch in the input texture that matches this output-sub-patch well.
It is also the best method for stochastic textures.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

24. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Sub-patch matching
So is the output-sub-patch
Patch is picked stochastically using a probability function.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

25. Extensions & Refinements
Adapting the Cost Function
Feathering and multi-resolution splining
FFT-Based Acceleration
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

26. Adapting the Cost Function
Discontinuities or seam boundaries are more prominent in low frequency regions rather than high frequency ones.
Take this into account by computing the gradient of the image along each direction.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

27. Adapting the Cost Function
GdA and GdB are the gradients in the patches A and B along the direction d.
M’ penalizes seams going through low frequency regions more than those going through high frequency regions.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

28. Feathering and multi-resolution splining
Feathering is done using a Gaussian kernel.
Use multi-resolution splining [Burt and Adelson 1983] of patches across seams.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

29. FFT-Based Acceleration
The SSD-based algorithms can be computationally expensive.
Computing the cost C(t) for all valid translations is O(n2)
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

30. FFT-Based Acceleration
The first two terms are sum of squares can be computed efficiently in O(n) time using summed-area tables [Crow 1984].
The third term is a convolution can be computed in O(nlog(n)).
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

31. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Discussion with DP.
Imposes an artificial grid structure on the pixels
potentially mean missing out on good seams that cannot be modeled within the imposed structure
Memoryless optimization procedure
cannot explicitly improve existing seams
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts

32. Graphcut Texture: Image and Video Synthesis Using Gragph Cuts
Summary
Benefits:
No restrictions on shape of the region where seam will be created.
Consideration of old seam costs.
Easy generalization to creation of seam surfaces.
A simple method for adding constraints.
Graphcut Texture: Image and Video Synthesis Using Gragph Cuts