1. Dirichlet Distribution
M. Farrow, MAS3301 Bayesian Statistics, Newcastle University
B. A. Frigyik, A. Kapila, and M. a R. Gupta. Introduction to the Dirichlet Distribution and Related Processes, University of Washington Department of Electrical Engineering, 2012.
Feb 26, 2015
Hee-Gook Jun

2. Outline Bernoulli Distribution Binomial Distribution
Multinomial Distribution
Beta Distribution
Dirichlet Distribution

3. Distribution Function vs. Linear Function
𝒇 𝒙 =𝒂𝒙+𝒃
Distribution function
Random variable: x
Parameter: a, b
Linear function
Variable: x
Constant: a, b
Binomial dist. function: 𝑿~𝑩 𝒑 =𝒇(𝒙;𝒑)
Gaussian dist. function: 𝑿~𝑵 𝝁, 𝝈 𝟐 =𝐟(𝐱;𝝁, 𝝈 𝟐 )
𝝁↓ 𝝈 𝟐 ↑
𝝁 ↑ 𝝈 𝟐 ↓

4. Bernoulli Distribution
Random variable: X = {0,1}
Parameter: 0 < p < 1
Sample space (support): x ∈ {0,1}
(when p is 0.5)
0.5
pmf
If success, 𝑓(𝑥; 𝑝) = 𝑝
If fail, 𝑓(𝑥; 𝑝) = 1 − 𝑝 1
𝑓(𝑥; 𝑝) = 𝑝 𝑥 (1−𝑝) 1−𝑥
cdf
𝑓𝑜𝑟 𝑥< −𝑝 𝑓𝑜𝑟 0≤𝑥≤ 𝑓𝑜𝑟 𝑥 ≥1
F 𝑥;𝑝 =𝑃 𝑋≤𝑥 = 1 1

5. Binomial Distribution
Random variable: X = # of successes in n trials
Parameter
n: # of trials
p: success probability in each trial
Sample space: x ∈ {0,…,n}
𝑓 𝑥;𝑛, 𝑝 =𝑃 𝑋=𝑥 = 𝑛 𝑥 𝑝 𝑥 (1−𝑝) 𝑛−𝑥
pmf
F 𝑘;𝑛,𝑝 =𝑃 𝑋≤𝑘 = 𝑖=0 𝑘 𝑛 𝑖 𝑝 𝑖 (1−𝑝) 𝑛−𝑖
cdf에서 수식 헷갈릴거 같아 x대신 k 씀. 의미는 그대로임
cdf
𝑿 𝟏 , 𝑿 𝟐 ,…, 𝑿 𝒏 ~𝑩𝒆𝒓 𝜽 ⇔ 𝒀~𝑩𝒊𝒏 𝒏,𝜽

6. Binomial Distribution: the parameter
Distribution’s shape is changed by the p
p < 0.5 skewed left
p = 0.5 symmetric
p > 0.5 skewed right
p 마다 pdf 형태 달라진다 → p 도 분포 가질 수 있다!

7. Beta Distribution: distribution of probability
Beta dist. provide a family of conjugate prior probability distributions in Bayesian inference
The domain of the beta dist. can be viewed as a probability, and in fact beta dist. is often used to describe the distribution of a probability value p
𝑿~𝑩𝒊𝒏 𝒏,𝒑 P ~𝑩𝒆𝒕𝒂(𝜶,𝜷)
𝑿~𝑩𝒊𝒏 𝒏,𝜽 Θ ~𝑩𝒆𝒕𝒂(𝜶,𝜷)
𝒇 𝒙; 𝒏,𝜽 𝒇(𝜽; 𝜶,𝜷)

8. Parameters of Gaussian Distribution
𝑓 𝑥;𝜇, 𝜎 2 = 1 𝜎 2𝜋 𝑒 − 𝑥−𝜇 𝜎 2

9. Parameters of Binomial Distribution
𝑓 𝑥;𝑛, 𝑝 = 𝑛 𝑥 𝑝 𝑥 (1−𝑝) 𝑛−𝑥

10. Parameters of Beta Distribution
Θ ~𝐵𝑒𝑡𝑎(𝛼,𝛽)

11. Beta Distribution: Gamma Function and Beta Function
Extension of the factorial function
Γ(𝑛) = (n−1)!
Beta function
Binomial coefficient after adjusting indices
Beta a, b = Γ(a)Γ(b) Γ(a+b)
𝑓 𝜃; 𝛼,𝛽 = Γ(𝛼+𝛽) Γ(𝛼)Γ(𝛽) 𝜃 𝛼−1 (1−𝜃) 𝛽−1
Θ ~𝐵𝑒𝑡𝑎(𝛼,𝛽)
𝑛 𝑥 = 𝑛! 𝑛−𝑥 !𝑥! =…= 1 𝑛+1 𝐵𝑒𝑡𝑎(𝑛−𝑥+1,𝑥+1)
𝑛 𝑥 𝑝 𝑥 (1−𝑝) 𝑛−𝑥

12. Bayesian inference Posterior probability
Consequence of two antecedents
Prior probability
Likelihood function
P 𝐴|𝐵 = P B A 𝑃(𝐴) 𝑃(𝐵)
posterior
likelihood x prior
𝑝 𝜃|𝑥 ∝ p x 𝜃 ×𝑝(𝜃)

13. Bayesian inference: Intuition
What we want to know
What we should know
Background knowledge
𝑝 𝜃|𝑥
posterior
𝑝 𝜃|𝑥 = 𝑝 𝑥 𝜃 𝑝(𝜃)
likelihood x prior
𝑝 𝜃|𝑥 = 𝑝 𝑥 𝜃 𝑝(𝜃)
Maximum Likelihood
Assumption

14. Conjugate prior for a binomial likelihood
Posterior distributions are in the same family as the prior probability distribution
Beta distribution
Conjugate prior for the binomial dist. for binomial likelihood
𝑝 𝜃|𝑥 ∝ p x 𝜃 𝑝(𝜃)
likelihood (Binomial Dist.)
prior (Beta Dist.)
Γ(𝛼+𝛽) Γ(𝛼)Γ(𝛽) 𝜃 𝛼−1 (1−𝜃) 𝛽−1
𝑛 𝑥 𝜃 𝑥 (1−𝜃) 𝑛−𝑥
Γ(𝛼+𝛽+n) Γ(𝛼+𝑥)Γ(𝛽+𝑛−𝑥) 𝜃 𝛼+𝑥−1 (1−𝜃) 𝛽+𝑛−𝑥−1
Posterior (Beta Dist.)

15. Binomial Distribution vs. Multinomial Distribution
Multinomial dist. is a generalization of the binomial dist.
𝑓 𝑥;𝑛, 𝑝 = 𝑛 𝑥 𝑝 𝑥 (1−𝑝) 𝑛−𝑥
Binomial distribution
𝑝 5 (1−𝑝) 10−5
T F T T F F F T F T
𝑓 𝑥 1 , …,𝑥 𝑘 ;𝑛, 𝑝 1 , …,𝑝 𝑘 = 𝑛! 𝑥 1 ! …𝑥 𝑘 ! 𝑝 1 𝑥 1 … 𝑝 𝑘 𝑥 𝑘
Multinomial distribution
10! 5!4!2!1! 𝑝 𝑝 𝑝 𝑝 4 1
⇧ ⇩ ⇧ ⇧ ⇩ ⇦ ⇧ ⇩ ⇨ ⇧ ⇦ ⇩

16. Beta Distribution vs. Dirichlet Distribution
Dirichlet dist. is a multivariate generalization of the Beta dist.
Very often used as prior distributions in Bayesian Statistics
Conjugate prior of the Multinomial dist.
𝑿~𝑩𝒊𝒏 𝒏,𝜽
Θ ~𝑩𝒆𝒕𝒂(𝜶,𝜷)
𝑿 𝟏 ,…, 𝑿 𝒌 ~𝑴𝒖𝒍𝒕𝒊 𝒏, 𝜽 𝟏 ,…, 𝜽 𝒌
Θ 𝟏 ,…, Θ 𝒌 ~𝑫𝒊𝒓 𝜶 𝟏 ,…, 𝜶 𝒌

17. Beta Distribution vs. Dirichlet Distribution Cont.
𝑓 𝜃; 𝛼,𝛽 = Γ(𝛼+𝛽) Γ(𝛼)Γ(𝛽) 𝜃 𝛼−1 (1−𝜃) 𝛽−1
Beta distribution
𝑓 𝜃 1 , …,𝜃 𝑘 ; 𝛼 1 , …,𝛼 𝑘 = Γ( 𝛼 1 +…+ 𝛼 𝑘 ) Γ( 𝛼 1 )...Γ( 𝛼 𝑘 ) 𝜃 1 𝛼 1 … 𝜃 𝑘 𝛼 𝑘
Dirichlet distribution
Beta
Dirichlet