Presentation is loading. Please wait.

Presentation is loading. Please wait.

Basics of Statistical Estimation. Learning Probabilities: Classical Approach Simplest case: Flipping a thumbtack tails heads True probability  is unknown.

Similar presentations


Presentation on theme: "Basics of Statistical Estimation. Learning Probabilities: Classical Approach Simplest case: Flipping a thumbtack tails heads True probability  is unknown."— Presentation transcript:

1 Basics of Statistical Estimation

2 Learning Probabilities: Classical Approach Simplest case: Flipping a thumbtack tails heads True probability  is unknown Given iid data, estimate  using an estimator with good properties: low bias, low variance, consistent (e.g., maximum likelihood estimate)

3 Maximum Likelihood Principle Choose the parameters that maximize the probability of the observed data

4 Maximum Likelihood Estimation (Number of heads is binomial distribution)

5 Computing the ML Estimate Use log-likelihood Differentiate with respect to parameter(s) Equate to zero and solve Solution:

6 Sufficient Statistics (#h,#t) are sufficient statistics

7 Bayesian Estimation tails heads True probability  is unknown Bayesian probability density for  p()p()  01

8 Use of Bayes’ Theorem prior likelihood posterior

9 Example: Application to Observation of Single “Heads" p(  |heads)  01 p()p()  01 p(heads|  )=   01 priorlikelihoodposterior

10 Probability of Heads on Next Toss

11 MAP Estimation Approximation: –Instead of averaging over all parameter values –Consider only the most probable value (i.e., value with highest posterior probability) Usually a very good approximation, and much simpler MAP value ≠ Expected value MAP → ML for infinite data (as long as prior ≠ 0 everywhere)

12 Prior Distributions for  Direct assessment Parametric distributions –Conjugate distributions (for convenience) –Mixtures of conjugate distributions

13 Conjugate Family of Distributions Beta distribution: Resulting posterior distribution:

14 Estimates Compared Prior prediction: Posterior prediction: MAP estimate: ML estimate:

15 Intuition The hyperparameters  h and  t can be thought of as imaginary counts from our prior experience, starting from "pure ignorance" Equivalent sample size =  h +  t The larger the equivalent sample size, the more confident we are about the true probability

16 Beta Distributions Beta(3, 2 )Beta(1, 1 )Beta(19, 39 )Beta(0.5, 0.5 )

17 Assessment of a Beta Distribution Method 1: Equivalent sample - assess  h and  t - assess  h +  t and  h /(  h +  t ) Method 2: Imagined future samples

18 Generalization to m Outcomes (Multinomial Distribution) Dirichlet distribution: Properties:

19 Other Distributions Likelihoods from the exponential family Binomial Multinomial Poisson Gamma Normal

20

21

22

23


Download ppt "Basics of Statistical Estimation. Learning Probabilities: Classical Approach Simplest case: Flipping a thumbtack tails heads True probability  is unknown."

Similar presentations


Ads by Google