1. Distributions and Concepts in Probability Theory
10701 Recitation
Pengtao Xie
11/13/2018

2. Outline Important Distributions Exponential Family Conjugate Prior
Biased and Unbiased Estimators
11/13/2018

3. Outline Important Distributions Exponential Family Conjugate Prior
Biased and Unbiased Estimators
11/13/2018

4. Distributions Usage in machine learning Expectation and variance
Bernoulli, Beta, multinomial, Dirichlet, Gaussian
11/13/2018

5. Usage of distributions in ML
Gaussian: Least Square Regression, Mixture of Gaussians, Kalman Filtering, Gaussian Markov Random Field, Gaussian Process
Multinomial: Hidden Markov Model, Mixture of Gaussians, Latent Dirichlet Allocation, Naive Bayes classifier
Dirichlet: Latent Dirichlet Allocation, Dirichlet Process
Bernoulli: Logistic Regression, switching variables in Graphical Models
Beta: Beta Process
11/13/2018

6. Expectation and variance
Expectation: the average value of a random variable under its probability distribution
Variance: a measure of how much variability there is in x around its mean value
11/13/2018

7. Bernoulli Distribution Dirichlet Distribution
Distributions
Random Variable
Discrete
Random Variable
Continuous
Random Variable
Two
Outcomes
Multiple Outcomes
Bernoulli Distribution
Multinomial
Distribution
Gaussian
Distribution
Conjugate
Conjugate
Beta
Distribution
Dirichlet Distribution
Conjugate
11/13/2018

8. Bernoulli Probability mass function Expectation
Model binary variable {0,1}
Probability mass function
Expectation
11/13/2018

9. Multinomial distribution
Model variables taking K possible states
1-of-K coding
Probability mass function
Expectation
11/13/2018

10. Beta Probability density function and are pesudo counts
Prior of the parameter in Bernoulli distribution
Probability density function
and are pesudo counts
Refer to note1.pdf
11/13/2018

11. Dirichlet distribution
Prior of parameters in multinomial distribution
Probability density function
are pesudo counts
11/13/2018

12. Univariate Gaussian distribution
Model continuous variables
Probability density function
Expectation and variance
11/13/2018

13. Multivariate Gaussian distribution
Defined on a continuous random vector
Probability density function
Expectation and covariance
11/13/2018

14. Outline Important Distributions Exponential Family Conjugate Prior
Biased and Unbiased Estimators
11/13/2018

15. Exponential Family Distribution
A class of distributions sharing a certain form
Natural parameters and sufficient statistics
Special cases: Bernoulli, Beta, multinomial, Dirichlet, Gaussian
Moment generating property
Refer to note2.pdf
11/13/2018

16. Outline Important Distributions Exponential Family Conjugate Prior
Biased and Unbiased Estimators
11/13/2018

17. Conjugate Prior From the same distribution both Beta both Dirichlet
both Gaussian
From the same distribution
Refer to note3.pdf
11/13/2018

18. Outline Important Distributions Exponential Family Conjugate Prior
Biased and Unbiased Estimators
11/13/2018

19. Estimator Bias Bias of an estimator
Unbiased estimator and biased estimator
Example: MLE for Gaussian mean and variance
Refer to note4.pdf
11/13/2018