G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 1 Statistical Data Analysis: Lecture 9 1Probability, Bayes’ theorem 2Random variables and.

Slides:



Advertisements
Similar presentations
Modeling of Data. Basic Bayes theorem Bayes theorem relates the conditional probabilities of two events A, and B: A might be a hypothesis and B might.
Advertisements

CHAPTER 8 More About Estimation. 8.1 Bayesian Estimation In this chapter we introduce the concepts related to estimation and begin this by considering.
1 12. Principles of Parameter Estimation The purpose of this lecture is to illustrate the usefulness of the various concepts introduced and studied in.
Chapter 7 Title and Outline 1 7 Sampling Distributions and Point Estimation of Parameters 7-1 Point Estimation 7-2 Sampling Distributions and the Central.
Chap 8: Estimation of parameters & Fitting of Probability Distributions Section 6.1: INTRODUCTION Unknown parameter(s) values must be estimated before.
Econ 140 Lecture 61 Inference about a Mean Lecture 6.
Statistical Data Analysis Stat 3: p-values, parameter estimation
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem, random variables, pdfs 2Functions.
G. Cowan Lectures on Statistical Data Analysis Lecture 2 page 1 Statistical Data Analysis: Lecture 2 1Probability, Bayes’ theorem 2Random variables and.
AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory We seek to determine from a set of data, a set of parameters such that their values would.
SUMS OF RANDOM VARIABLES Changfei Chen. Sums of Random Variables Let be a sequence of random variables, and let be their sum:
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
Bayesian Learning, Cont’d. Administrivia Various homework bugs: Due: Oct 12 (Tues) not 9 (Sat) Problem 3 should read: (duh) (some) info on naive Bayes.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
Statistical Background
G. Cowan 2011 CERN Summer Student Lectures on Statistics / Lecture 41 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 14 page 1 Statistical Data Analysis: Lecture 14 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 7 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
Lecture II-2: Probability Review
1 10. Joint Moments and Joint Characteristic Functions Following section 6, in this section we shall introduce various parameters to compactly represent.
Simulation Output Analysis
880.P20 Winter 2006 Richard Kass 1 Confidence Intervals and Upper Limits Confidence intervals (CI) are related to confidence limits (CL). To calculate.
G. Cowan 2009 CERN Summer Student Lectures on Statistics1 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability densities,
G. Cowan Lectures on Statistical Data Analysis Lecture 3 page 1 Lecture 3 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
Random Sampling, Point Estimation and Maximum Likelihood.
Estimating parameters in a statistical model Likelihood and Maximum likelihood estimation Bayesian point estimates Maximum a posteriori point.
Maximum Likelihood Estimator of Proportion Let {s 1,s 2,…,s n } be a set of independent outcomes from a Bernoulli experiment with unknown probability.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Deterministic vs. Random Maximum A Posteriori Maximum Likelihood Minimum.
G. Cowan Lectures on Statistical Data Analysis Lecture 1 page 1 Lectures on Statistical Data Analysis London Postgraduate Lectures on Particle Physics;
1 Lecture 16: Point Estimation Concepts and Methods Devore, Ch
ELEC 303 – Random Signals Lecture 18 – Classical Statistical Inference, Dr. Farinaz Koushanfar ECE Dept., Rice University Nov 4, 2010.
PROBABILITY AND STATISTICS FOR ENGINEERING Hossein Sameti Department of Computer Engineering Sharif University of Technology Principles of Parameter Estimation.
: Chapter 3: Maximum-Likelihood and Baysian Parameter Estimation 1 Montri Karnjanadecha ac.th/~montri.
Chapter 2 Statistical Background. 2.3 Random Variables and Probability Distributions A variable X is said to be a random variable (rv) if for every real.
Confidence Interval & Unbiased Estimator Review and Foreword.
Sampling and estimation Petter Mostad
G. Cowan Lectures on Statistical Data Analysis Lecture 4 page 1 Statistical Data Analysis: Lecture 4 1Probability, Bayes’ theorem 2Random variables and.
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 8 page 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem 2Random variables and.
1 Introduction to Statistics − Day 3 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Brief catalogue of probability densities.
G. Cowan TAE 2010 / Statistics for HEP / Lecture 11 Statistical Methods for HEP Lecture 1: Introduction Taller de Altas Energías 2010 Universitat de Barcelona.
G. Cowan Lectures on Statistical Data Analysis Lecture 4 page 1 Lecture 4 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
Joint Moments and Joint Characteristic Functions.
G. Cowan Computing and Statistical Data Analysis / Stat 9 1 Computing and Statistical Data Analysis Stat 9: Parameter Estimation, Limits London Postgraduate.
Statistics Sampling Distributions and Point Estimation of Parameters Contents, figures, and exercises come from the textbook: Applied Statistics and Probability.
1 Chapter 8: Model Inference and Averaging Presented by Hui Fang.
G. Cowan Lectures on Statistical Data Analysis Lecture 6 page 1 Statistical Data Analysis: Lecture 6 1Probability, Bayes’ theorem 2Random variables and.
Chapter 8 Estimation ©. Estimator and Estimate estimator estimate An estimator of a population parameter is a random variable that depends on the sample.
G. Cowan Lectures on Statistical Data Analysis Lecture 5 page 1 Statistical Data Analysis: Lecture 5 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
Parameter Estimation. Statistics Probability specified inferred Steam engine pump “prediction” “estimation”
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
Learning Theory Reza Shadmehr Distribution of the ML estimates of model parameters Signal dependent noise models.
G. Cowan Statistical methods for HEP / Freiburg June 2011 / Lecture 1 1 Statistical Methods for Discovery and Limits in HEP Experiments Day 1: Introduction,
Fundamentals of Data Analysis Lecture 11 Methods of parametric estimation.
Stat 223 Introduction to the Theory of Statistics
12. Principles of Parameter Estimation
Probability Theory and Parameter Estimation I
Ch3: Model Building through Regression
Lecture 4 1 Probability (90 min.)
Graduierten-Kolleg RWTH Aachen February 2014 Glen Cowan
Lecture 3 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
Stat 223 Introduction to the Theory of Statistics
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
Statistics for Particle Physics Lecture 3: Parameter Estimation
Computing and Statistical Data Analysis / Stat 7
Introduction to Statistics − Day 4
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
12. Principles of Parameter Estimation
Presentation transcript:

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 1 Statistical Data Analysis: Lecture 9 1Probability, Bayes’ theorem 2Random variables and probability densities 3Expectation values, error propagation 4Catalogue of pdfs 5The Monte Carlo method 6Statistical tests: general concepts 7Test statistics, multivariate methods 8Goodness-of-fit tests 9Parameter estimation, maximum likelihood 10More maximum likelihood 11Method of least squares 12Interval estimation, setting limits 13Nuisance parameters, systematic uncertainties 14Examples of Bayesian approach

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 2 Parameter estimation The parameters of a pdf are constants that characterize its shape, e.g. r.v. Suppose we have a sample of observed values: parameter We want to find some function of the data to estimate the parameter(s): ← estimator written with a hat Sometimes we say ‘estimator’ for the function of x 1,..., x n ; ‘estimate’ for the value of the estimator with a particular data set.

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 3 Properties of estimators If we were to repeat the entire measurement, the estimates from each would follow a pdf: biased large variance best We want small (or zero) bias (systematic error): → average of repeated measurements should tend to true value. And we want a small variance (statistical error): → small bias & variance are in general conflicting criteria

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 4 An estimator for the mean (expectation value) Parameter: Estimator: We find: (‘sample mean’)

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 5 An estimator for the variance Parameter: Estimator: (factor of n  1 makes this so) (‘sample variance’) We find: where

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 6 The likelihood function Suppose the entire result of an experiment (set of measurements) is a collection of numbers x, and suppose the joint pdf for the data x is a function that depends on a set of parameters  : Now evaluate this function with the data obtained and regard it as a function of the parameter(s). This is the likelihood function: (x constant)

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 7 The likelihood function for i.i.d.*. data Consider n independent observations of x: x 1,..., x n, where x follows f (x;  ). The joint pdf for the whole data sample is: In this case the likelihood function is (x i constant) * i.i.d. = independent and identically distributed

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 8 Maximum likelihood estimators If the hypothesized  is close to the true value, then we expect a high probability to get data like that which we actually found. So we define the maximum likelihood (ML) estimator(s) to be the parameter value(s) for which the likelihood is maximum. ML estimators not guaranteed to have any ‘optimal’ properties, (but in practice they’re very good).

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 9 ML example: parameter of exponential pdf Consider exponential pdf, and suppose we have i.i.d. data, The likelihood function is The value of  for which L(  ) is maximum also gives the maximum value of its logarithm (the log-likelihood function):

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 10 ML example: parameter of exponential pdf (2) Find its maximum by setting → Monte Carlo test: generate 50 values using  = 1: We find the ML estimate:

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 11 Functions of ML estimators Suppose we had written the exponential pdf as i.e., we use = 1/ . What is the ML estimator for ? For a function  (  ) of a parameter , it doesn’t matter whether we express L as a function of  or . The ML estimator of a function  (  ) is simply So for the decay constant we have Caveat: is biased, even thoughis unbiased. (bias →0 for n →∞) Can show

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 12 Example of ML: parameters of Gaussian pdf Consider independent x 1,..., x n, with x i ~ Gaussian ( ,  2 ) The log-likelihood function is

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 13 Example of ML: parameters of Gaussian pdf (2) Set derivatives with respect to ,  2 to zero and solve, We already know that the estimator for  is unbiased. But we find, however,so ML estimator for  2 has a bias, but b→0 for n→∞. Recall, however, that is an unbiased estimator for  2.

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 14 Variance of estimators: Monte Carlo method Having estimated our parameter we now need to report its ‘statistical error’, i.e., how widely distributed would estimates be if we were to repeat the entire measurement many times. One way to do this would be to simulate the entire experiment many times with a Monte Carlo program (use ML estimate for MC). For exponential example, from sample variance of estimates we find: Note distribution of estimates is roughly Gaussian − (almost) always true for ML in large sample limit.

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 15 Variance of estimators from information inequality The information inequality (RCF) sets a lower bound on the variance of any estimator (not only ML): Often the bias b is small, and equality either holds exactly or is a good approximation (e.g. large data sample limit). Then, Estimate this using the 2nd derivative of ln L at its maximum:

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 16 Variance of estimators: graphical method Expand ln L (  ) about its maximum: First term is ln L max, second term is zero, for third term use information inequality (assume equality): i.e., → to get, change  away from until ln L decreases by 1/2.

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 17 Example of variance by graphical method ML example with exponential: Not quite parabolic ln L since finite sample size (n = 50).

G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 18 Wrapping up lecture 9 We’ve seen some main ideas about parameter estimation: estimators, bias, variance, and introduced the likelihood function and ML estimators. Also we’ve seen some ways to determine the variance (statistical error) of estimators: Monte Carlo method Using the information inequality Graphical Method Next we will extend this to cover multiparameter problems, variable sample size, histogram-based data,...