1. Nonparametric Bayesian Texture Learning and Synthesis
Leo Zhu
Yuanhao Chen
William Freeman
Antonio Torralba
Presented at NIPS 2009
Discussion and Slides by Eric Wang
August 13, 2010
Wang 2010

2. Outline Introduction Image Patches and Features HDP-2DHMM Inference
Texture Synthesis
Image Segmentation and Synthesis
Conclusion
Wang 2010

3. Introduction
The authors consider texture learning and synthesis with spatial dependence.
Two basic approaches to this problem have been considered
Represent texture using textons and spatial layout. This approach is sensitive to parameter settings, has low rendering quality, and is slow.
Patch based approaches offer improved rendering quality and are faster, but do not have semantic understanding and texture modeling.
In this paper, the authors adopt a patch based approach, and augment it with nonparametric Bayesian modeling and statistical learning. A spatial HMM (2D-HMM) whose states (texton vocabulary) are generated from an HDP.
Once the parameters of the HDP-2DHMM are learned, large textures can be synthesized according to the spatial transition matrix and dictionary of textures.
Wang 2010

4. Introduction
Wang 2010

5. Image Patches and Features
Each patch is characterized by a set of filter responses that correspond to values b of image filter response h at location l.
Specifically, each 24x24 patch is divided into 6x6 cells (l=1,…,36), each of which is size 4x4 pixels. For each pixel in cell l, the response to h=37 image filter responses are computed.
For each 4x4 cell, the responses to each filter are averaged across all pixels and quantized (binned) into 1 of b=6 bins. Therefore, each patch is represented by 37x36x6=7992 dimensional feature responses.
The authors point out that, unlike previous work that used Kmeans to first form a visual vocabulary, their approach directly applies a nonparametric model
Wang 2010

6. HDP-2DHMM
Let L(i), T(i), R(i), D(i) denote the four neighbors of node i.
In the above graphical model, only L(i) and T(i) dependencies are shown.
indexes the state of node i, and associates it with the cluster label of a texton.
and are the emission and transition parameters, respeectively
Wang 2010

7. HDP-2DHMM
Wang 2010

8. HDP-2DHMM
Wang 2010

9. HDP-2DHMM
Wang 2010

10. HDP-2DHMM
Wang 2010

11. HDP-2DHMM
Wang 2010

12. HDP-2DHMM
Wang 2010

13. HDP-2DHMM
Wang 2010

14. HDP-2DHMM
Wang 2010

15. HDP-2DHMM
The authors also allow the textons to be slightly shifted from their original locations by introducing two hidden variables that indicate the displacements of textons relative to location i.
For simplicity, the shift variables are given a uniform prior over a small shift neighborhood (10% in either direction).
Wang 2010

16. Inference
Collapsed Gibbs sampling is used to learn the parameters of HDP-2DHMM. The inference alternates between sampling the state indicators z and the global state probabilities .
The state indicators z are sampled as
The global state probabilities are sampled as in HDP, and the equation not given.
Wang 2010

17. Texture Synthesis
The primary objective of this paper was to synthesize textures from smaller texture observations.
The authors note, however, that HDP-2DHMM is a generative model for image features and not actual image patches. To do this, they integrate image quilting, and define a dictionary of image patches (from the original image) from which the new image is synthesized (with much lower synthesis cost over standard image quilting).
Wang 2010

18. Synthesis Examples

19. Synthesis Examples
Wang 2010

20. Synthesis Examples
Wang 2010

21. Image Segmentation and Synthesis
Wang 2010

22. Conclusion
A novel nonparametric Bayesian model called HD-2DHMM for texture learning and synthesis was presented in this paper.
The main contributions of this paper are that it learns the textons and spatial HMM structure jointly.
Inference is performed using a collapsed Gibbs Sampler
The clustering nature of the model allows for faster image quilting and synthesis with good qualitative results.
Image segmentation and synthesis results were also presented.
Wang 2010