1. Stick-Breaking Constructions
Patrick Dallaire
June 10th, 2011

2. Outline
Introduction of the Stick-Breaking process

3. Outline Introduction of the Stick-Breaking process
Presentation of fundamental representation

4. Outline Introduction of the Stick-Breaking process
Presentation of fundamental representation
The Dirichlet process
The Pitman-Yor process
The Indian buffet process

5. Outline Introduction of the Stick-Breaking process
Presentation of fundamental representation
The Dirichlet process
The Pitman-Yor process
The Indian buffet process
Definition of the Beta process

6. Outline Introduction of the Stick-Breaking process
Presentation of fundamental representation
The Dirichlet process
The Pitman-Yor process
The Indian buffet process
Definition of the Beta process
A Stick-Breaking construction of Beta process

7. Outline Introduction of the Stick-Breaking process
Presentation of fundamental representation
The Dirichlet process
The Pitman-Yor process
The Indian buffet process
Definition of the Beta process
A Stick-Breaking construction of Beta process
Conclusion and current work

8. The Stick-Breaking process

9. The Stick-Breaking process
Assume a stick of unit length

10. The Stick-Breaking process
Assume a stick of unit length

11. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

12. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

13. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

14. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

15. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

16. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

17. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

18. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

19. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

20. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

21. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut

22. The Stick-Breaking process
Assume a stick of unit length
At each iteration, a part of the remaining stick is broken by sampling the proportion to cut
How should we sample these proportions?

23. Beta random proportions
Let be the proportion to cut at iteration

24. Beta random proportions
Let be the proportion to cut at iteration
The remaining length can be expressed as

25. Beta random proportions
Let be the proportion to cut at iteration
The remaining length can be expressed as
Thus, the broken part is defined by

26. Beta random proportions
Let be the proportion to cut at iteration
The remaining length can be expressed as
Thus, the broken part is defined by
We first consider the case where

27. Beta distribution The Beta distribution is a density function on
Parameters and control its shape

28. The Dirichlet process

29. The Dirichlet process
Dirichlet processes are often used to produce infinite mixture models

30. The Dirichlet process
Dirichlet processes are often used to produce infinite mixture models
Each observation belongs to one of the infinitely many components

31. The Dirichlet process
Dirichlet processes are often used to produce infinite mixture models
Each observation belongs to one of the infinitely many components
The model ensures that only a finite number of components have appreciable weight

32. The Dirichlet process
A Dirichlet process, , can be constructed according to a Stick-Breaking process
Where is the base distribution and is a unit mass at

33. Construction demo

34. Construction demo

35. Construction demo

36. Construction demo

37. Construction demo

38. Construction demo

39. Construction demo

40. Construction demo

41. Construction demo

42. Construction demo

43. Construction demo

44. Construction demo

45. Construction demo

46. Construction demo

47. Construction demo

48. Construction demo

49. The Pitman-Yor process

50. The Pitman-Yor process
A Pitman-Yor process, , can be constructed according to a Stick-Breaking process
Where and

51. Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes

52. Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
The parameter determines initial shapes of the Beta distribution

53. Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
The parameter determines initial shapes of the Beta distribution
When , there is no changes over time and its called a Dirichlet process

54. Evolution of the Beta cuts
The parameter controls the speed at which the Beta distribution changes
The parameter determines initial shapes of the Beta distribution
When , there is no changes over time and its called a Dirichlet process
MATLAB DEMO

55. The Indian Buffet process

56. The Indian Buffet process
The Indian Buffet process was initially used to represent latent features

57. The Indian Buffet process
The Indian Buffet process was initially used to represent latent features
Observations are generated according to a set of unknown hidden features

58. The Indian Buffet process
The Indian Buffet process was initially used to represent latent features
Observations are generated according to a set of unknown hidden features
The model ensure that only a finite number of features have appreciable probability

59. The Indian Buffet process
Recall the basic Stick-Breaking process

60. The Indian Buffet process
Recall the basic Stick-Breaking process

61. The Indian Buffet process
Recall the basic Stick-Breaking process
Here, we only consider the remaining parts

62. The Indian Buffet process
Recall the basic Stick-Breaking process
Here, we only consider the remaining parts

63. The Indian Buffet process
Recall the basic Stick-Breaking process
Here, we only consider the remaining parts
Each value corresponds to a feature probability of appearance

64. Summary

65. Summary
The Dirichlet process induces a probability over infinitely many classes

66. Summary
The Dirichlet process induces a probability over infinitely many classes
This is the underlying de Finetti mixing distribution of the Chinese restaurant process

67. De Finetti theorem
It states that the distribution of any infinitely exchangeable sequence can be written
where is the de Finetti mixing distribution

68. Summary
The Dirichlet process induces a probability over infinitely many classes
This is the underlying de Finetti mixing distribution of the Chinese restaurant process
The Indian Buffet process induces a probability over infinitely many features

69. Summary
The Dirichlet process induces a probability over infinitely many classes
This is the underlying de Finetti mixing distribution of the Chinese restaurant process
The Indian Buffet process induces a probability over infinitely many features
Its underlying de Finetti mixing distribution is the Beta process

70. The Beta process

71. The Beta process
This process

72. Beta with Stick-Breaking
The Beta distribution has a Stick-Breaking representation which allows to sample from

73. Beta with Stick-Breaking
The Beta distribution has a Stick-Breaking representation which allows to sample from
The construction is

74. Beta with Stick-Breaking

75. Beta with Stick-Breaking

76. Beta with Stick-Breaking

77. Beta with Stick-Breaking

78. Beta with Stick-Breaking

79. Beta with Stick-Breaking

80. Beta with Stick-Breaking

81. Beta with Stick-Breaking

82. Beta with Stick-Breaking

83. Beta with Stick-Breaking

84. Beta with Stick-Breaking

85. Beta with Stick-Breaking

86. Beta with Stick-Breaking

87. Beta with Stick-Breaking

88. Beta with Stick-Breaking

89. Beta with Stick-Breaking
The Beta distribution has a Stick-Breaking representation which allows to sample from
The construction is

90. The Beta process A Beta process is defined as
as , and is a Beta process

91. Stick-Breaking the Beta process
The Stick-Breaking construction of the Beta process is such that

92. Stick-Breaking the Beta process
Expending the first terms

93. Conclusion
We briefly described various Stick-Breaking constructions for Bayesian nonparametric priors
These constructions help to understand the properties of each process
It also unveils connections among existing priors
The Stick-Breaking process might help to construct new priors

94. Current work
Applying a Stick-Breaking process to select the number of support points in a Gaussian process
Defining a stochastic process for unbounded random directed acyclic graph
Finding its underlying Stick-Breaking representation