G. Cowan RHUL Physics Comment on use of LR for limits page 1 Comment on definition of likelihood ratio for limits ATLAS Statistics Forum CERN, 2 September,

Slides:



Advertisements
Similar presentations
Using the Profile Likelihood in Searches for New Physics / PHYSTAT 2011 G. Cowan 1 Using the Profile Likelihood in Searches for New Physics arXiv:
Advertisements

Sampling: Final and Initial Sample Size Determination
G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 1 Statistical Methods for Particle Physics Lecture 2: Tests based on likelihood ratios.
EPIDEMIOLOGY AND BIOSTATISTICS DEPT Esimating Population Value with Hypothesis Testing.
G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 1 Use of profile likelihood to determine systematic uncertainties ATLAS Top.
8. Statistical tests 8.1 Hypotheses K. Desch – Statistical methods of data analysis SS10 Frequent problem: Decision making based on statistical information.
Hypothesis testing Some general concepts: Null hypothesisH 0 A statement we “wish” to refute Alternative hypotesisH 1 The whole or part of the complement.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem, random variables, pdfs 2Functions.
G. Cowan Statistics for HEP / NIKHEF, December 2011 / Lecture 3 1 Statistical Methods for Particle Physics Lecture 3: Limits for Poisson mean: Bayesian.
Evaluating Hypotheses
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
G. Cowan 2011 CERN Summer Student Lectures on Statistics / Lecture 41 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability.
G. Cowan RHUL Physics Higgs combination note status page 1 Status of Higgs Combination Note ATLAS Statistics/Higgs Meeting Phone, 7 April, 2008 Glen Cowan.
G. Cowan Statistics for HEP / NIKHEF, December 2011 / Lecture 2 1 Statistical Methods for Particle Physics Lecture 2: Tests based on likelihood ratios.
G. Cowan Discovery and limits / DESY, 4-7 October 2011 / Lecture 2 1 Statistical Methods for Discovery and Limits Lecture 2: Tests based on likelihood.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 7 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
G. Cowan Discovery and limits / DESY, 4-7 October 2011 / Lecture 3 1 Statistical Methods for Discovery and Limits Lecture 3: Limits for Poisson mean: Bayesian.
G. Cowan RHUL Physics Bayesian Higgs combination page 1 Bayesian Higgs combination using shapes ATLAS Statistics Meeting CERN, 19 December, 2007 Glen Cowan.
Aaker, Kumar, Day Ninth Edition Instructor’s Presentation Slides
1 Glen Cowan Statistics Forum News Glen Cowan Eilam Gross ATLAS Statistics Forum CERN, 3 December, 2008.
G. Cowan 2009 CERN Summer Student Lectures on Statistics1 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability densities,
G. Cowan Lectures on Statistical Data Analysis Lecture 3 page 1 Lecture 3 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
Results of combination Higgs toy combination, within and across experiments, with RooStats Grégory Schott Institute for Experimental Nuclear Physics of.
G. Cowan CLASHEP 2011 / Topics in Statistical Data Analysis / Lecture 21 Topics in Statistical Data Analysis for HEP Lecture 2: Statistical Tests CERN.
1 Proposal to change limits for exclusion N.Andari, L. Fayard, M.Kado, F.Polci LAL- Orsay Y.Fang, Y.Pan, H.Wang, Sau Lan Wu University of Wisconsin-Madison.
G. Cowan RHUL Physics Bayesian Higgs combination page 1 Bayesian Higgs combination based on event counts (follow-up from 11 May 07) ATLAS Statistics Forum.
1 Methods of Experimental Particle Physics Alexei Safonov Lecture #25.
G. Cowan Weizmann Statistics Workshop, 2015 / GDC Lecture 31 Statistical Methods for Particle Physics Lecture 3: asymptotics I; Asimov data set Statistical.
G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.
G. Cowan ATLAS Statistics Forum / Minimum Power for PCL 1 Minimum Power for PCL ATLAS Statistics Forum EVO, 10 June, 2011 Glen Cowan* Physics Department.
G. Cowan, RHUL Physics Discussion on significance page 1 Discussion on significance ATLAS Statistics Forum CERN/Phone, 2 December, 2009 Glen Cowan Physics.
G. Cowan RHUL Physics LR test to determine number of parameters page 1 Likelihood ratio test to determine best number of parameters ATLAS Statistics Forum.
1 Copyright © 2010, 2007, 2004 Pearson Education, Inc. All Rights Reserved. Example: In a recent poll, 70% of 1501 randomly selected adults said they believed.
G. Cowan RHUL Physics Input from Statistics Forum for Exotics page 1 Input from Statistics Forum for Exotics ATLAS Exotics Meeting CERN/phone, 22 January,
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
Easy Limit Statistics Andreas Hoecker CAT Physics, Mar 25, 2011.
G. Cowan Lectures on Statistical Data Analysis Lecture 8 page 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem 2Random variables and.
1 Introduction to Statistics − Day 3 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Brief catalogue of probability densities.
G. Cowan Lectures on Statistical Data Analysis Lecture 4 page 1 Lecture 4 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
G. Cowan Computing and Statistical Data Analysis / Stat 9 1 Computing and Statistical Data Analysis Stat 9: Parameter Estimation, Limits London Postgraduate.
G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 1 Report from the Statistics Forum ATLAS Analysis Strategies Workshop.
G. Cowan, RHUL Physics Statistics for early physics page 1 Statistics jump-start for early physics ATLAS Statistics Forum EVO/Phone, 4 May, 2010 Glen Cowan.
G. Cowan RHUL Physics Status of Higgs combination page 1 Status of Higgs Combination ATLAS Higgs Meeting CERN/phone, 7 November, 2008 Glen Cowan, RHUL.
Name Mean of Sampling Distribution Standard Deviation/Error of Sampling Distribution 1 sample z-Interval for Proportions 1 sample z-interval for Means.
In Bayesian theory, a test statistics can be defined by taking the ratio of the Bayes factors for the two hypotheses: The ratio measures the probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Statistical methods for HEP / Freiburg June 2011 / Lecture 2 1 Statistical Methods for Discovery and Limits in HEP Experiments Day 2: Discovery.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
1 Studies on Exclusion Limits N.Andari, L. Fayard, M.Kado, F.Polci LAL- Orsay LAL meeting 23/07/2009.
G. Cowan RHUL Physics Statistical Issues for Higgs Search page 1 Statistical Issues for Higgs Search ATLAS Statistics Forum CERN, 16 April, 2007 Glen Cowan.
G. Cowan CERN Academic Training 2010 / Statistics for the LHC / Lecture 21 Statistics for the LHC Lecture 2: Discovery Academic Training Lectures CERN,
G. Cowan SLAC Statistics Meeting / 4-6 June 2012 / Two Developments 1 Two developments in discovery tests: use of weighted Monte Carlo events and an improved.
G. Cowan CERN Academic Training 2012 / Statistics for HEP / Lecture 21 Statistics for HEP Lecture 2: Discovery and Limits Academic Training Lectures CERN,
Discussion on significance
Statistics for the LHC Lecture 3: Setting limits
Maximum Likelihood Estimation
Comment on Event Quality Variables for Multivariate Analyses
School on Data Science in (Astro)particle Physics
Computing and Statistical Data Analysis / Stat 8
TAE 2018 / Statistics Lecture 3
Statistical Methods for the LHC
Statistical Tests and Limits Lecture 2: Limits
Computing and Statistical Data Analysis / Stat 7
Statistical Methods for HEP Lecture 3: Discovery and Limits
Decomposition of Stat/Sys Errors
Parametric Methods Berlin Chen, 2005 References:
TAE 2018 Centro de ciencias Pedro Pascual Benasque, Spain
Presentation transcript:

G. Cowan RHUL Physics Comment on use of LR for limits page 1 Comment on definition of likelihood ratio for limits ATLAS Statistics Forum CERN, 2 September, 2009 Glen Cowan Physics Department Royal Holloway, University of London

G. Cowan RHUL Physics Comment on use of LR for limits page 2 Introduction At the Statistics Forum on N. Andari presented a study (Orsay & Wisconsin) showing that a modified definition for the likelihood ratio leads to a sampling distribution that accurately follows the half-chi-square distribution. This offered the possibility to increase greatly the ease and accuracy with which we can compute exclusion limits, even for small samples. At the time some (at least GC and EG) did not fully understand how this could work, so we thought through a simple example. Conclusions agree with approach of Andari et al. Purpose of present talk is to present this example; see also attached note.

G. Cowan RHUL Physics Comment on use of LR for limits page 3 The simple problem Suppose outcome of measurement is Gaussian distributed x with expectation value and variance s and b are contributions from signal and background, take  as a known constant;  is strength parameter. The likelihood function for the parameter of interest  is Suppose goal is to set an upper limit on  given a measurement x.

G. Cowan RHUL Physics Comment on use of LR for limits page 4 The likelihood ratio To test a value of , construct likelihood ratio: Suppose on physical grounds  should be positive, then the maximum of L(  ) from the allowed range of  is from Usually use logarithmic equivalent  2 ln (  ):

G. Cowan RHUL Physics Comment on use of LR for limits page 5 Likelihood ratio for upper limit For an upper limit on  one uses the test statistic which, putting together the ingredients, becomes

G. Cowan RHUL Physics Comment on use of LR for limits page 6 p-value for exclusion To quantify level of agreement between observed x and hypothesized , calculate p-value Note if x ~ Gaussian(  s+b,  ), then the quantity follows a chi-square pdf for 1 d.o.f. But the distribution of q  is more complicated (not chi-square), 95% CL upper limit on  is value for which p  = 0.05.

G. Cowan RHUL Physics Comment on use of LR for limits page 7 Likelihood ratio without constraint on  ^ Andari et al. propose to define an unphysical estimator which goes negative if x < b. Using this define then a corresponding test statistic for upper limits:

G. Cowan RHUL Physics Comment on use of LR for limits page 8 Exclusion significance from q  ′ From the definition of q  ′ one can see its pdf must be a half-chi-square distribution, i.e., a delta function at zero when x >  s + b, and a chi-square pdf for x ≤  s + b. Therefore (see CSC note), the significance from an observed value q  ′ is given by the simple relation p-value of 0.05 corresponds to Z = 1.64.

G. Cowan RHUL Physics Comment on use of LR for limits page 9 Comparison of test variables Both q  and q  ′ are shown here as a function of x for  =1, s=10, b=20,  2 =20. Note they are equal for b ≤ x ≤  s + b.

G. Cowan RHUL Physics Comment on use of LR for limits page 10 Equivalence of q  and q  ′ It is easy to see that the two test variables q  and q  ′ are monotonically related: and therefore they represent equivalent tests of .

G. Cowan RHUL Physics Comment on use of LR for limits page 11 Relation to study by Andari et al. (Eilam) Andari et al. presented the following table of the fraction of toy experiments with values of the test statistic below certain levels: Chi-square and “exact” (counting) formulae give same fractions for “median”, but not median +1 , +2 . This is because in this example it corresponds to having x < b, i.e., this where q  and q  ′ are different. But for example, median  1  or  2  would correspond to the region where q  and q  ′ are equal, and the two methods there would agree.

G. Cowan RHUL Physics Comment on use of LR for limits page 12 Conclusion on q  ′ Both test variables, q  and q  ′, give equivalent tests, because of their monotonic relation. If one were to work out (with difficulty) the exact sampling pdf of q , and to compute from it the p-value, and from it the significance Z, then it would be the same as from the simple formula Z = √ q  ′ using the same value of the observation x. We still regard only positive  as physical, but allow its estimator to go negative effectively as a mathematical trick to get the desired p-value. Also easy to show that this likelihood ratio gives a test equivalent to the ratio used for the LEP analyses (see attached note):

G. Cowan RHUL Physics Comment on use of LR for limits page 13 Extra slides

G. Cowan RHUL Physics Comment on use of LR for limits page 14 Example from CSC book E.g. H →  from the CSC combination chapter used the statistic q  (as did all other channels). MC studies show that distribution of q  departs significantly from half-chi-square form. If we had used q  ′, the agreement with half-chi-square would be much closer, even for low luminosity.