Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are directly copied from the paper and Hanna Wallach’s slides

outline Introduction Finite belief network Infinite belief network Inference Experiment

Introduction Main contribution: combine deep belief network and nonparametric bayesian together. Main idea: use IBP to learn the structure of the network Structure of the network include: Depth Width Connectivity

Single layer network Use Binary matrix to represent the network. Black refer to 1(two unit were connected) White refer to 0 (two unit were not connected) IBP can be used as the prior for infinite columns binary matrix Z

Review IBP 1.First customer tries dishes. 2. Nth customer tries Tasked dishes K with probability new dishes

Multi-layer network

Cascading IBP Also parameterize by Each dishes in the restaurant is also a customer in another Indian buffet process Each matrix is exchangeable both rows and columns This chain can reach the state with probability one ( number of unit in layer m) Properties: For unit in layer m+1 Expected number of parents: Expected number of children:

Sample from the CIBP prior

model m refer to the layers and increase upto M. weightsbias Place layer wise Gaussian prior on weights and bias, Gamma prior on noise precision

Inference Weights, bias, noise variance can be sampled with Gibbs sampler.

Inference( sample Z) Two step: Sample existing dishes MH-sample Add a new unit and, and insert connection to this unit with For a exist unit remove the connection to this unit with MH ratio

Experiment result Olivetti faces Remove bottom halves of the test image.

Experiment result MNIST Digits

Experiment result Frey Faces