1. Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra

2. Image vs. Texture

3. Example-based Texture Synthesis Input Example

4. Related Work Local region-growing method -Pixel-based -Patch-based-Patch-based Global optimization-based method Heeger and Bergen sig95,Pyramid-based texture synthesis Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field ….. Physical Simulation et al

5. Example-based Texture Synthesis 1.Dynamic texturing (animation/video) 2.Greater control: external knobs, e.g., flow, shape, illumination 3.Texture quality evaluation and refinement

6. Motivation + Input Texture Input Flow Field Output Sequence

7. Limitations of previous work Pixel-based can not maintain global structure of the texture inconsistencies Patch-based have no control over individual pixel values Global methods: pixel-to-pixel interactions insufficient to capture large scale structures complex formulations difficult to optimize

8. Texture Energy Pixel-by-pixel comparison of source and target not possible Compare texture “elements” –Local pixel neighborhoods Want each target neighborhood to be similar to some source neighborhood

9. Texture Energy X (target frame) Z (source)

10. Energy for Single Neighborhood p (pixel) Z (source) X (target frame)

11. Energy for Single Neighborhood X p (neighborhood) Z (source) X (target frame)

12. Energy for Single Neighborhood X p (neighborhood) Z (source) X (target frame)

13. Energy for Single Neighborhood (nearest neighbor) Z p X p (neighborhood) Z (source) X (target frame)

14. Energy for Single Neighborhood X Z (nearest neighbor) Z p X p (neighborhood) Texture Energy (single neighborhood)

15. Texture Energy for Entire Image X Z

16. X Z  individual neighborhood energy

17. 4.Repeat iteratively until set {Z P } stops changing 3.Update X to be texture that minimizes the energy Texture Synthesis 1.Initialize texture X 2.Find closest input neighborhood Z P corresponding to each output neighborhood X P

18. Texture Synthesis

19. Gradient-based Energy D is the differentiation operator and µ is a relative weighting coefficient

20. Muti-level Synthesis Multi-resolution Multi-scale/multiple neighborhood sizes Reasons  a good initialization of the texture  avoid undesirable local minima  avoid inconsistencies  capture large scale structures

21. Random Initialization Multiple –Resolution Levels –Neighborhood Sizes Progressively refined output Texture Energy Graph

22. Controllable Synthesis

23. Flow consistency –Perceived motion similar to flow Texture similarity –Shape, size, orientation of texture elements similar to source Flow-guided Texture Animation Target Flow Flowing Target Source Texture Texture Similarity Flow Consistency

24. Naive Approach Target Flow Ignore Texture Similarity Warp each frame via flow field X1X1 XnXn Target Frames X0X0 Warp Flowing Target Flow Consistency Source Texture Texture Similarity Source Texture Texture Similarity

25. Naive Approach Texture structure not maintained Target Flow Flowing Target Flow Consistency Source Texture Texture Similarity

26. Warp Instead of… Warp XnXn X0X0 X1X1 Target Frames

27. Warp Instead of… Warp XnXn X0X0 X1X1 Target Frames

28. Warp + Correct “Correct” Operation Target Frames XX– Warp W Correct Warped Frame Synthesized Frame

29. Link to original goals “Correct” Operation X Target Frames X– W Correct Warp

30. Link to original goals “Correct” Operation X Target Frames X– W Correct Flow Consistency Warp

31. Link to original goals “Correct” Operation X Target Frames X– W Correct Source Flow Consistency Texture Similarity Warp

32. Energy = Flow Energy + Texture Energy Optimization of Energy X Target Frames X– W Correct Source Flow Consistency Texture Similarity Warp

33. Flow Energy 2D flow field frame iframe i+1 pixel (p,i)pixel (q,i+1)

34. Results: Texture Synthesis

36. InputWei-Levoy [Wei’00] Image Quilting [Efros’01] Graph-cuts [Kwatra’03] Texture Optimization [Kwatra’05]

37. Time Cost Static texture synthesis –5-10 iterations per resolution level Flowing texture synthesis –20-60 seconds per frame ResolutionOne iterationTotal time 64x642.5 sec20 sec 128x12810 sec2 min 256x25625 sec8.5 min

38. Limitations Optimization finds local minima Blurry / misaligned texture elements Texture scale should be compatible with flow field

39. Thanks a lot!

40. Pixel-based Methods Compare local causal neighbourhoods Efros and Leung (ICCV ’ 99) Wei and Levoy (Siggraph 2000) Ashikhmin (I3D 2001) InputOutput

41. Patch-based Methods Copy patches of pixels rather than single pixels Copy patches of pixels rather than single pixels Chaos Mosaic, Xu et al, 1997 Patch-Based Sampling, Liang et al(ACM 2001) Image Quilting, Efros and Williams(Siggraph 2001)

42. Back