1. Graphcut Textures:Image and Video Synthesis Using Graph Cuts
Vivek Kwatra ,Arno Schodl ,Irfan Essa ,
Greg Turk ,Aaron Bobick

2. Outline Texture Synthesis Graphcut Texture Video Textures
What is texture?
How to synthesis?
Graphcut Texture
Main idea, Contribution
Patch placement and matching techniques
Patch fitting
Refinements and extensions
Video Textures

3. Texture
Generate a large image from smaller image or longer video from smaller one
How to do it?
Copy patches (or pixels ) from input to output
Problems
Artifacts ( e.g., boundaries of patches)

4. Solution
Copy patches from input to output with overlap

5. Definitions Where to position the input texture called Offset
Which part of the input texture to transfer called Seam

6. M(s,t,A,B) = ||A(s) – B(s)|| + || A(t) – B(t)||
Cutting the graph
cut 8 1 4 7 8
cut
existing
pixels A
new patch B
new patch
corresponding
graph
old
pixels 8 8 2 5 8 8 8 3 6 9
overlap
Use Max Flow/ Min Cut algorithm to find the best cut where the cost between two adjacent nodes s and t :
M(s,t,A,B) = ||A(s) – B(s)|| + || A(t) – B(t)||
A(s) and B(s) are pixel colors in position s in the old and new patch respectively

7. Cutting the graph 8 8 8 8 8 8 8 8 old pixels existing pixels 1 4 7
corresponding
graph
cut
new patch 2 5 8 8 3 6 9 8
new patch
overlap 8 8
old cut
existing
pixels 1 5 9 13
cut
overlap 2 6 10 14
old
pixels
new patch 3 7 11 15
corresponding
graph
new cut 4 8 12 16 8

8. Seam nodes
Problem: What should we do if we lay down a patch in a spot that already has existing seams? How should we take preexisting seams into account when making a cut?
old cut
new cut
old cut
new cut 1 4 7 8 8
existing
pixels A
new patch B 8 8
new patch
corresponding
graph
old
pixels 2 5 8 8 8
overlap 3 6 9

9. Seam nodes
Solution: add a seam node between two pixels and connect the seam node with an arc to the new patch. The weight of the arc will be the old cost between the pixels.
old cut
new cut
old cut
new cut 1 4 7 8 S1 8
existing
pixels A
new patch B S2
corresponding
graph
old
pixels 8 8
new patch 2 5 8 S3 8 8
overlap 3 6 9 S4
Then add arcs from pixels to seam node with new costs
M(1,2, A, B)
M(1,2, B, A) B A 1 S1 4 7 8 8

12. The Synthesis Process
Step1:Patch placement and Matching ( Choose candidate patches or offset)
Random Placement
Entire Patch Matching
Sub-patch Matching
Step2:Patch Fitting ( Choose optimal portion or seam)
Only those pixels are copied that are chosen by graph-cut algorithm
Cost of graph-cut is a measure of similarity

13. Stemp1. Patch Placement Random Placement
Entire input image is translated to random location in the output image
Good results for random textures
Patch Matching ( Entire or Sub )
Used when we already have some patches in the output image (refinement).
Every seam has a cost (min graph cut cost)
Uses error region

14. Sub-Patch Matching

15. Entire Patch Matching

16. Error Region Error = seam cost Choose a pixel with largest error
Sum of costs along minimum cut patch
Choose a pixel with largest error
Select a region around that pixel, called error region
Patch Matching ( entire or sub ) will select those patches that completely cover our error region

17. Patch Placement Entire Patch Matching Matching criteria:
Search for translated input versions and choose that gives best match
Matching criteria:
Normalized SSD:
C(t): The smaller, the better (means similar)

18. Patch Placement
Compute cost for all possible offsets. Cost is inversely proportional to similarity
Choose the cost that has the highest probability of resulting in a similar region
A low value of k leads to picking of only those patch locations that have a very good match with the output whereas a high value of k leads to more random patch selection.
Good results for structured and semi-structured texture

19. Patch Placement Sub-Patch Matching Matching criteria:
Pick a small sub-patch from output
Search for output-patch in input texture or look for translations of input sub-patch
Matching criteria:
Use the same probability function
Best results for unstructured regions or video textures ( fire, waves, smoke, etc.)

20. Patch Fitting Make graph for overlap region Every pixel is a node
Edge weights:
Associate weight with each edge
Find minimum graph cut

21. Refinements
Modified Matching cost function

22. Refinements Search across all translations is costly (use FFT)
speed up
150x100x30 video sequence
Search for a new patch reduces from around 10 minutes (using naïve search) to 5 seconds (using FFT-based search)

23. Extensions Translation to Transformation
Rotation, scaling, affine or projection
Interactive merging and blending
Many source images
User specifies position & constraints pixels
Algorithm finds best seam
SIGGRAPH banner

24. Interactive merging and blending

25. Interactive merging and blending

26. Extensions Goal is to loop the video forever Video texture
One way is to find the pair of similar looking frames and use them to repeat the video
Video synthesis using graphcut
Find time of transition on pixel-wise basis

27. Video Texture – GraphCut
Finding Seams for Video Transitions
Find good transition between pair of images
Take a window around transition (60 frames)
Construct the graph by connecting a pixel to its neighbors in space and time
Min cut will give you time of transition on per pixel basis
Use translation in time and space both

28. Video Synthesis

29. Video Synthesis Temporal Constraints for Video
To loop the video, add k frames in start and end of video (10 frames), constraint these frames to stay the same, graph is generated and best seam is found, then k frames are removed.

30. Video Synthesis
We fix the first k and last k frames of the output sequence to be the same k frames of the input.
The pixels in these frames are now constrained to stay the same.
This is ensured by adding links of infinite cost between these pixels and the patches they are constrained to copy from, during graph construction
Use graphcut to find the best seam given that these pixels don’t change.
Once the output has been generated, we remove the first k frames from it.
This ensures a video loop since the kth frame of the output is the same as its last frame before this removal operation