Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra

Image vs. Texture

Example-based Texture Synthesis Input Example

Related Work Local region-growing method -Pixel-based -Patch-based-Patch-based Global optimization-based method Heeger and Bergen sig95,Pyramid-based texture synthesis Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field ….. Physical Simulation et al

Example-based Texture Synthesis 1.Dynamic texturing (animation/video) 2.Greater control: external knobs, e.g., flow, shape, illumination 3.Texture quality evaluation and refinement

Motivation + Input Texture Input Flow Field Output Sequence

Limitations of previous work Pixel-based can not maintain global structure of the texture inconsistencies Patch-based have no control over individual pixel values Global methods: pixel-to-pixel interactions insufficient to capture large scale structures complex formulations difficult to optimize

Texture Energy Pixel-by-pixel comparison of source and target not possible Compare texture “elements” –Local pixel neighborhoods Want each target neighborhood to be similar to some source neighborhood

Texture Energy X (target frame) Z (source)

Energy for Single Neighborhood p (pixel) Z (source) X (target frame)

Energy for Single Neighborhood X p (neighborhood) Z (source) X (target frame)

Energy for Single Neighborhood X p (neighborhood) Z (source) X (target frame)

Energy for Single Neighborhood (nearest neighbor) Z p X p (neighborhood) Z (source) X (target frame)

Energy for Single Neighborhood X Z (nearest neighbor) Z p X p (neighborhood) Texture Energy (single neighborhood)

Texture Energy for Entire Image X Z

X Z  individual neighborhood energy

4.Repeat iteratively until set {Z P } stops changing 3.Update X to be texture that minimizes the energy Texture Synthesis 1.Initialize texture X 2.Find closest input neighborhood Z P corresponding to each output neighborhood X P

Texture Synthesis

Gradient-based Energy D is the differentiation operator and µ is a relative weighting coefficient

Muti-level Synthesis Multi-resolution Multi-scale/multiple neighborhood sizes Reasons  a good initialization of the texture  avoid undesirable local minima  avoid inconsistencies  capture large scale structures

Random Initialization Multiple –Resolution Levels –Neighborhood Sizes Progressively refined output Texture Energy Graph

Controllable Synthesis

Flow consistency –Perceived motion similar to flow Texture similarity –Shape, size, orientation of texture elements similar to source Flow-guided Texture Animation Target Flow Flowing Target Source Texture Texture Similarity Flow Consistency

Naive Approach Target Flow Ignore Texture Similarity Warp each frame via flow field X1X1 XnXn Target Frames X0X0 Warp Flowing Target Flow Consistency Source Texture Texture Similarity Source Texture Texture Similarity

Naive Approach Texture structure not maintained Target Flow Flowing Target Flow Consistency Source Texture Texture Similarity

Warp Instead of… Warp XnXn X0X0 X1X1 Target Frames

Warp Instead of… Warp XnXn X0X0 X1X1 Target Frames

Warp + Correct “Correct” Operation Target Frames XX– Warp W Correct Warped Frame Synthesized Frame

Link to original goals “Correct” Operation X Target Frames X– W Correct Warp

Link to original goals “Correct” Operation X Target Frames X– W Correct Flow Consistency Warp

Link to original goals “Correct” Operation X Target Frames X– W Correct Source Flow Consistency Texture Similarity Warp

Energy = Flow Energy + Texture Energy Optimization of Energy X Target Frames X– W Correct Source Flow Consistency Texture Similarity Warp

Flow Energy 2D flow field frame iframe i+1 pixel (p,i)pixel (q,i+1)

Results: Texture Synthesis

InputWei-Levoy [Wei’00] Image Quilting [Efros’01] Graph-cuts [Kwatra’03] Texture Optimization [Kwatra’05]

Time Cost Static texture synthesis –5-10 iterations per resolution level Flowing texture synthesis –20-60 seconds per frame ResolutionOne iterationTotal time 64x642.5 sec20 sec 128x12810 sec2 min 256x25625 sec8.5 min

Limitations Optimization finds local minima Blurry / misaligned texture elements Texture scale should be compatible with flow field

Thanks a lot!

Pixel-based Methods Compare local causal neighbourhoods Efros and Leung (ICCV ’ 99) Wei and Levoy (Siggraph 2000) Ashikhmin (I3D 2001) InputOutput

Patch-based Methods Copy patches of pixels rather than single pixels Copy patches of pixels rather than single pixels Chaos Mosaic, Xu et al, 1997 Patch-Based Sampling, Liang et al(ACM 2001) Image Quilting, Efros and Williams(Siggraph 2001)

Back