2. Inverse Texture Synthesis
Li-Yi Wei1 Jianwei Han2 Kun Zhou1,2
Hujun Bao2 Baining Guo1 Harry Shum1
1Microsoft 2Zhejiang University

3. Example-based texture synthesis
For a small input texture
produce an arbitrarily large output with similar look
Why? may not possible to obtain large input
texture synthesis
input
output

4. Inverse texture synthesis
From a large input texture
produce a small output that best summarizes input
inverse texture synthesis
output
input

5. Why? Textures are getting large Advances in scanning technology
High dimensionality: time-varying, BRDF
Expensive to store, transmit, compute
So why we need this?
Textures are getting larger, making them more expensive to store, transmit, and compute.
Yale University
MSR Asia
Columbia University

6. Overview inverse texture synthesis input output (large) (small)
texturing
(slow)
(fast)
Inverse texture synthesis can solve these problems.
Given a large input, the algorithm will produce a small compaction that retains vital information of the input.
The compaction can be used for texturing with similar visual results from the original input, with faster computation and smaller memory consumption.
similar quality

7. Related work: image compression
pixel-wise
identical
compress
decompress
inverse synth
texture synth
input
perceptual
similar

8. Related work: epitome Epitome [Jojic et al. 2003]
Jigsaw [Kannan et al. 2007]
Major source of inspiration for us
For general images, not just textures
We provide better quality
Bidirectional similarity [Simakov et al. 2008]
Factoring repeated content [Wang et al. 2008]

9. Related work: manual crop
stationary
globally
varying
original
manual crop
our result

10. Globally-varying textures
Markov Random Field (MRF) textures
local & stationary
Globally-varying textures
local, but not necessarily stationary
MRF
globally varying

11. Globally varying textures Previous work
MRF input → globally varying output
texture-by-numbers in Image analogies [Hertzmann et al. 2001]
progressively variant textures [Zhang et al. 2003]
texture design and morphing [Matusik et al. 2005]
Globally varying input
appearance manifold [Wang et al. 2006]
spatially & time varying BRDF [Gu et al. 2006]
context-aware texture [Lu et al. 2007]

12. Globally varying textures Definition
texture + control maps
Examples of control maps
user-specified colors [Hertzmann et al. 2001]
spatially-varying parameters [Gu et al. 2006]
weathering degree-map [Wang et al. 2006]
context information [Lu et al. 2007]
texture (paint crack)
control map (paint thickness)

13. Globally varying textures
Including time-varying textures as well
Large data size!
time-varying BRDF
[Gu et al. 2006]
512 x 512 x 33, 288 MB
context-aware texture
[Lu et al. 2007]
1226 x 978 x 50, 35 MB

14. Inverse texture synthesis
Compacting globally varying textures
including both texture + control map
output compaction
inverse synthesis
texture
control
texture
control map
input

15. Compaction as summary of original
Re-synthesis with user control map
faster
slower
forward
synthesis
user control +
compaction
re-synthesis
from
compaction
re-synthesis
from
original

16. Inverse Texture Synthesis
Applicable to MRF textures
no control map
homogenized result
manual cropping ?
original
re-synthesis

17. Inverse Texture Synthesis
Manual cropping unsuitable for globally variant texture
no matter where you put the window
our compaction
manual crop
black
purple
orange
original
from compaction
from manual crop

18. forward term [Kwatra et al. 2005]
Basic formulation
Inspired by texture optimization [Kwatra et al. 2005]
inverse term (New!)
forward term [Kwatra et al. 2005] xp Zp
best match zq xq
best match
Z (output)
X (input)

19. Energy plot
energy
original
compaction size

20. Why both terms? inverse term preserves all input features
forward
inverse term preserves all input features
forward term avoids artifacts in compaction
f-only
missing feature
both
i-only
garbage
both
i-only
discontinuity
both

21. Comparing with epitome [Jojic et al. 2003]
Similar to our method but only inverse term
blur, discontinuity
epitome
epitome
our
our
original
original

22. Comparing with epitome [Jojic et al. 2003] Re-synthesis
our
epitome
our
original
original

23. Solver How to solve this? Texture optimization [Kwatra et al. 2005]
Discrete solver [Han et al. 2006]

24. Optimization [Kwatra et al. 2005]
NO inverse term
forward term [Kwatra et al. 2005]
E-step
fix xq
argminz E(x,z)
least square
M-step
fix Z
argminxq |xq-zq|2
search
fix xq xq Zq zq xq Z
argminxq |xq-zq|2 X

25. forward term [Kwatra et al. 2005]
Our solver
inverse term
forward term [Kwatra et al. 2005]
E-step
fix xq
argminz E(x,z)
least square
M-step (forward)
fix Z
argminxq |xq-zq|2
search xp xp zp xq Zq
discrete solver [Han et al. 2006]
M-step (inverse)
fix xp
argminzp |xp-zp|2
discrete solver zq xq Z
argminxq |xq-zq|2
discrete solver X

26. Results

27. Results output control compaction 1282 original (paint crack)
799 x 546
from orig.
84555 sec
from comp.
1131 sec

28. Compaction size on quality
799 x 546
5122
2562
1282

29. Compaction size on quality
799 x 546
(original)
5122
2562
1282

30. Re-synthesis without control map
stationary only
comp.
original
re-synthesis

31. GPU synthesis – small texture better Extension from [Lefebvre & Hoppe 2005]
3 fps, original
6 fps, compact
cheese
mold
1214 x 1212
compaction
1282
3.5 fps, original
7.0 fps, compact
dirt
271x481
original

32. Limitation: Correlation between texture & control
original
reconstruction
compaction

33. Why little squares. Instead of, e. g
Why little squares? Instead of, e.g. a set of texton [Leung & Malik 2001]
Most general
compaction utilizable for any synthesis algorithm
GPU synthesis
Looks nice
I like little squares
visual summary & visualization

34. Orientation field for anisotropic textures
Orientation field w as part of energy function
E(x, z) → E(x, z; w)
Good orientation field yields better solution
comp.
no w
comp.
with w
original
orientation field

35. Results: orientation field just mention this in a quick pass
Paris
et al. 04
original
manual +
interp
ours

36. Results: orientation field
Paris
et al. 04
original
manual +
interp
ours

37. Results: orientation field
Paris
et al. 04
original
manual +
interp
ours

38. Results: orientation field
Paris
et al. 04
original
manual +
interp
ours

39. Future work Higher dimensional textures
e.g. video
General images, not just textures
Bidirectional similarity [Simakov et al. CVPR 2008]
Image compression

40. Acknowledgements
Yale graphics group Columbia graphics group Sylvain Lefebvre Hughes Hoppe Matusik et al Mayang.com Jiaping Wang Xin Tong Jian Sun Frank Yu
Bennett Wilburn Eric Stollnitz Dwight Daniels Reviewers Dinesh Manocha Ming Lin Chas Boyd Brandon Lloyd Avneesh Sud Billy Chen

41. Thank You!