Stick-breaking Construction for the Indian Buffet Process Duke University Machine Learning Group Presented by Kai Ni July 27, 2007 Yee Whye The, Dilan Gorur, and Zoubin Ghahramani, AISTATS 2007

Outline Introduction Indian Buffet Process (IBP) Stick-breaking construction for IBP Slice samplers Results

Introduction Indian Buffet Process (IBP)  A distribution over binary matrices consisting of N rows (objects) and an unbounded number of columns (features);  1/0 in entry (i,k) indicates feature k present/absent from object i. An example  Objects are movies – “ Terminator 2 ”, “ Shrek ” and “ Shanghai Knights ” ;  Features are – “ action ”, “ comedy ”, “ stars Jackie Chan ” ;  The matrix can be [101; 010; 110].

Relationship to CRP IBP and CRP are both tools for defining nonparametric Bayesian models with latent variables. CRP – Each object belongs to only one of infinitely many latent classes. IBP – Each object can possess potentially any combination of infinitely many latent features. Previous Gibbs sampler for IBP is based on CRP. In this paper the author derives a stick-breaking representation for the IBP, and develop efficient slice samplers.

Indiant Buffet Process Let Z be a random binary N x K matrix, and denote entry (I,k) in Z by z ik. For each feature k let u k be the prior probability that feature k is present in an object. Let be the strength parameter of the IBP, the full model is: If we integrated out u k and taking the limit of K -> infinity, we obtain the IBP in the situation similar to CRP.

Gibbs sampler for IBP For new features:

Stick-breaking construction for IBP

Derivation pdf for each u cdf for each u cdf for u (1) pdf for u (1)

Relation to DP

Stick-breaking for IBP (2) In truncated stick-breaking for IBP, let K* be the truncation level. We set u (k) =0 for k>K*, and z ik =0 for k>K*.

Slice Sampler Using Adaptive rejection sampling (ARS) to deal with the truncation level. Introduce an auxiliary slice variable s with

1. Update s: if new s makes K* becomes larger, we iteratively draw u (k) until u (K*’) > s. 2. Update Z: given s, we only need to update zik for each i and k<=K*. 3. Update for k = 1, …, K*. 4. Update u (k) for k = 1, …, K*. Sampling 12K*K*’ Decreasing u (k) Old sNew s 0Range of uniform dist. for s

Change of Representations IBP – ignoring the ordering on features; Stick-breaking IBP – enforcing an ordering with decreasing weights. Stick-breaking -> IBP: Drop the stick lengths and the inactive features, leaving only the K + active feature columns along with the corresponding parameters. IBP -> stick-breaking: Draw both the stick lengths and order the features in decreasing stick lengths, introducing K o inactive features until

Semi-ordered Stick-breaking u k + on active features are unordered and draw from a CRP similar distribution: The stick length on inactive feature is similar to the stick- breaking IBP The auxiliary variable s determines how many inactive features need to add. (unordered 1~K + ) KoKo s 0Range of uniform dist. for s Min(u (k) )

Results Used the conjugate linear-Gaussian binary latent feature model for comparing the performance of the different samplers. Each data point is modeled using a spherical Gaussian with mean z i,: A and variance

Demonstration Apply semi-ordered slice sampler to 1000 examples of handwritten images of 3’s in the MNIST dataset.

Conclusion The author derived novel stick-breaking representations of the Indian buffet process. Based on these representations, new MCMC samplers are proposed that are easy to implement and work on more general models than Gibbs sampling.