G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 1 Statistical Methods for Particle Physics Lecture 2: Tests based on likelihood ratios.

Slides:



Advertisements
Similar presentations
Using the Profile Likelihood in Searches for New Physics / PHYSTAT 2011 G. Cowan 1 Using the Profile Likelihood in Searches for New Physics arXiv:
Advertisements

G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 1 Use of profile likelihood to determine systematic uncertainties ATLAS Top.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem, random variables, pdfs 2Functions.
G. Cowan Statistics for HEP / NIKHEF, December 2011 / Lecture 3 1 Statistical Methods for Particle Physics Lecture 3: Limits for Poisson mean: Bayesian.
G. Cowan RHUL Physics Bayesian methods for HEP / DESY Terascale School page 1 Bayesian statistical methods for HEP Terascale Statistics School DESY, Hamburg.
1 Nuisance parameters and systematic uncertainties Glen Cowan Royal Holloway, University of London IoP Half.
G. Cowan Statistical methods for HEP / Freiburg June 2011 / Lecture 3 1 Statistical Methods for Discovery and Limits in HEP Experiments Day 3: Exclusion.
G. Cowan RHUL Physics Comment on use of LR for limits page 1 Comment on definition of likelihood ratio for limits ATLAS Statistics Forum CERN, 2 September,
G. Cowan Statistical methods for HEP / Birmingham 9 Nov Recent developments in statistical methods for particle physics Particle Physics Seminar.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
G. Cowan 2011 CERN Summer Student Lectures on Statistics / Lecture 41 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability.
G. Cowan RHUL Physics Statistical Methods for Particle Physics / 2007 CERN-FNAL HCP School page 1 Statistical Methods for Particle Physics (2) CERN-FNAL.
G. Cowan Statistics for HEP / NIKHEF, December 2011 / Lecture 2 1 Statistical Methods for Particle Physics Lecture 2: Tests based on likelihood ratios.
G. Cowan Statistical Data Analysis / Stat 4 1 Statistical Data Analysis Stat 4: confidence intervals, limits, discovery London Postgraduate Lectures on.
G. Cowan Shandong seminar / 1 September Some Developments in Statistical Methods for Particle Physics Particle Physics Seminar Shandong University.
G. Cowan Discovery and limits / DESY, 4-7 October 2011 / Lecture 2 1 Statistical Methods for Discovery and Limits Lecture 2: Tests based on likelihood.
G. Cowan Lectures on Statistical Data Analysis Lecture 13 page 1 Statistical Data Analysis: Lecture 13 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 7 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
G. Cowan Discovery and limits / DESY, 4-7 October 2011 / Lecture 3 1 Statistical Methods for Discovery and Limits Lecture 3: Limits for Poisson mean: Bayesian.
G. Cowan RHUL Physics Bayesian Higgs combination page 1 Bayesian Higgs combination using shapes ATLAS Statistics Meeting CERN, 19 December, 2007 Glen Cowan.
Graduierten-Kolleg RWTH Aachen February 2014 Glen Cowan
880.P20 Winter 2006 Richard Kass 1 Confidence Intervals and Upper Limits Confidence intervals (CI) are related to confidence limits (CL). To calculate.
G. Cowan SUSSP65, St Andrews, August 2009 / Statistical Methods 2 page 1 Statistical Methods in Particle Physics Lecture 2: Limits and Discovery.
G. Cowan 2009 CERN Summer Student Lectures on Statistics1 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability densities,
G. Cowan Lectures on Statistical Data Analysis Lecture 3 page 1 Lecture 3 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
G. Cowan Orsay 2014 / Discussion on Statistics1 Topics in Statistics for Particle Physics Discussion on Statistics LAL Orsay 16 June 2014 Glen Cowan Physics.
G. Cowan CLASHEP 2011 / Topics in Statistical Data Analysis / Lecture 21 Topics in Statistical Data Analysis for HEP Lecture 2: Statistical Tests CERN.
G. Cowan Statistical techniques for systematics page 1 Statistical techniques for incorporating systematic/theory uncertainties Theory/Experiment Interplay.
G. Cowan RHUL Physics Bayesian Higgs combination page 1 Bayesian Higgs combination based on event counts (follow-up from 11 May 07) ATLAS Statistics Forum.
G. Cowan Weizmann Statistics Workshop, 2015 / GDC Lecture 31 Statistical Methods for Particle Physics Lecture 3: asymptotics I; Asimov data set Statistical.
G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.
G. Cowan, RHUL Physics Discussion on significance page 1 Discussion on significance ATLAS Statistics Forum CERN/Phone, 2 December, 2009 Glen Cowan Physics.
G. Cowan CERN Academic Training 2010 / Statistics for the LHC / Lecture 41 Statistics for the LHC Lecture 4: Bayesian methods and further topics Academic.
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 8 page 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem 2Random variables and.
1 Introduction to Statistics − Day 3 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Brief catalogue of probability densities.
G. Cowan NExT Workshop, 2015 / GDC Lecture 11 Statistical Methods for Particle Physics Lecture 1: introduction & statistical tests Fifth NExT PhD Workshop:
G. Cowan Lectures on Statistical Data Analysis Lecture 4 page 1 Lecture 4 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
G. Cowan Statistical methods for particle physics / Cambridge Recent developments in statistical methods for particle physics HEP Phenomenology.
G. Cowan Computing and Statistical Data Analysis / Stat 9 1 Computing and Statistical Data Analysis Stat 9: Parameter Estimation, Limits London Postgraduate.
G. Cowan Statistical methods for particle physics / Wuppertal Recent developments in statistical methods for particle physics Particle Physics.
G. Cowan Lectures on Statistical Data Analysis Lecture 9 page 1 Statistical Data Analysis: Lecture 9 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Cargese 2012 / Statistics for HEP / Lecture 21 Statistics for HEP Lecture 2: Discovery and Limits International School Cargèse August 2012 Glen.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan Statistical methods for HEP / Freiburg June 2011 / Lecture 2 1 Statistical Methods for Discovery and Limits in HEP Experiments Day 2: Discovery.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan St. Andrews 2012 / Statistics for HEP / Lecture 21 Statistics for HEP Lecture 2: Discovery and Limits 69 th SUSSP LHC Physics St. Andrews
G. Cowan iSTEP 2014, Beijing / Statistics for Particle Physics / Lecture 31 Statistical Methods for Particle Physics Lecture 3: systematic uncertainties.
1 Nuisance parameters and systematic uncertainties Glen Cowan Royal Holloway, University of London IoP Half.
G. Cowan CERN Academic Training 2010 / Statistics for the LHC / Lecture 21 Statistics for the LHC Lecture 2: Discovery Academic Training Lectures CERN,
G. Cowan SOS 2010 / Statistical Tests and Limits -- lecture 31 Statistical Tests and Limits Lecture 3 IN2P3 School of Statistics Autrans, France 17—21.
G. Cowan SLAC Statistics Meeting / 4-6 June 2012 / Two Developments 1 Two developments in discovery tests: use of weighted Monte Carlo events and an improved.
G. Cowan CERN Academic Training 2012 / Statistics for HEP / Lecture 21 Statistics for HEP Lecture 2: Discovery and Limits Academic Training Lectures CERN,
Discussion on significance
Computing and Statistical Data Analysis / Stat 11
Statistics for the LHC Lecture 3: Setting limits
Lecture 4 1 Probability (90 min.)
TAE 2018 / Statistics Lecture 3
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
TAE 2018 Benasque, Spain 3-15 Sept 2018 Glen Cowan Physics Department
Statistical Tests and Limits Lecture 2: Limits
Computing and Statistical Data Analysis / Stat 7
TAE 2017 / Statistics Lecture 2
Statistical Methods for HEP Lecture 3: Discovery and Limits
Introduction to Statistics − Day 4
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
Computing and Statistical Data Analysis / Stat 10
Presentation transcript:

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 1 Statistical Methods for Particle Physics Lecture 2: Tests based on likelihood ratios Cours d’hiver 2012 du LAL Orsay, 3-5 January, 2012 Glen Cowan Physics Department Royal Holloway, University of London

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 2 Outline Lecture 1: Introduction and basic formalism Probability, statistical tests, confidence intervals. Lecture 2: Tests based on likelihood ratios Systematic uncertainties (nuisance parameters) Limits for Poisson mean Lecture 3: More on discovery and limits Upper vs. unified limits (F-C) Spurious exclusion, CLs, PCL Look-elsewhere effect Why 5σ for discovery?

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 3 A simple example For each event we measure two variables, x = (x 1, x 2 ). Suppose that for background events (hypothesis H 0 ), and for a certain signal model (hypothesis H 1 ) they follow where x 1, x 2 ≥ 0 and C is a normalization constant.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 4 Likelihood ratio as test statistic In a real-world problem we usually wouldn’t have the pdfs f(x|H 0 ) and f(x|H 1 ), so we wouldn’t be able to evaluate the likelihood ratio for a given observed x, hence the need for multivariate methods to approximate this with some other function. But in this example we can find contours of constant likelihood ratio such as:

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 5 Event selection using the LR Using Monte Carlo, we can find the distribution of the likelihood ratio or equivalently of signal (H 1 ) background (H 0 ) From the Neyman-Pearson lemma we know that by cutting on this variable we would select a signal sample with the highest signal efficiency (test power) for a given background efficiency.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 6 Search for the signal process But what if the signal process is not known to exist and we want to search for it. The relevant hypotheses are therefore H 0 : all events are of the background type H 1 : the events are a mixture of signal and background Rejecting H 0 with Z > 5 constitutes “discovering” new physics. Suppose that for a given integrated luminosity, the expected number of signal events is s, and for background b. The observed number of events n will follow a Poisson distribution:

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 7 Likelihoods for full experiment We observe n events, and thus measure n instances of x = (x 1, x 2 ). The likelihood function for the entire experiment assuming the background-only hypothesis (H 0 ) is and for the “signal plus background” hypothesis (H 1 ) it is where  s and  b are the (prior) probabilities for an event to be signal or background, respectively.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 8 Likelihood ratio for full experiment We can define a test statistic Q monotonic in the likelihood ratio as To compute p-values for the b and s+b hypotheses given an observed value of Q we need the distributions f(Q|b) and f(Q|s+b). Note that the term –s in front is a constant and can be dropped. The rest is a sum of contributions for each event, and each term in the sum has the same distribution. Can exploit this to relate distribution of Q to that of single event terms using (Fast) Fourier Transforms (Hu and Nielsen, physics/ ).

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 9 Distribution of Q Take e.g. b = 100, s = 20. f (Q|b) f (Q|s+b) p-value of b onlyp-value of s+b Suppose in real experiment Q is observed here.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 10 Systematic uncertainties Up to now we assumed all parameters were known exactly. In practice they have some (systematic) uncertainty. Suppose e.g. uncertainty in expected number of background events b is characterized by a (Bayesian) pdf  (b). Maybe take a Gaussian, i.e., where b 0 is the nominal (measured) value and  b is the estimated uncertainty. In fact for many systematics a Gaussian pdf is hard to defend – more on this later.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 11 Distribution of Q with systematics To get the desired p-values we need the pdf f (Q), but this depends on b, which we don’t know exactly. But we can obtain the prior predictive (marginal) model: With Monte Carlo, sample b from  (b), then use this to generate Q from f (Q|b), i.e., a new value of b is used to generate the data for every simulation of the experiment. This broadens the distributions of Q and thus increases the p-value (decreases significance Z) for a given Q obs.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 12 Distribution of Q with systematics (2) For s = 20, b 0 = 100,  b = 20 this gives f (Q|b) f (Q|s+b) p-value of b onlyp-value of s+b

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 13 Using the likelihood ratio L(s)/L(s) ˆ Instead of the likelihood ratio L s+b /L b, suppose we use as a test statistic Intuitively this is a measure of the level of agreement between the data and the hypothesized value of s. low : poor agreement high  : better agreement 0 ≤ ≤ 1 maximizes L(s)

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 14 L(s)/L(s) for counting experiment ˆ Consider an experiment where we only count n events with n ~ Poisson(s + b). Then. To establish discovery of signal we test the hypothesis s = 0 using whereas previously we had used which is monotonic in n and thus equivalent to using n as the test statistic.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 15 L(s)/L(s) for counting experiment (2) ˆ But if we only consider the possibility of signal being present when n > b, then in this range (0) is also monotonic in n, so both likelihood ratios lead to the same test. b

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 16 L(s)/L(s) for general experiment ˆ If we do not simply count events but also measure for each some set of numbers, then the two likelihood ratios do not necessarily give equivalent tests, but in practice should be very close. (s) has the important advantage that for a sufficiently large event sample, its distribution approaches a well defined form (Wilks’ Theorem). In practice the approach to the asymptotic form is rapid and one obtains a good approximation even for relatively small data samples (but need to check with MC). This remains true even when we have adjustable nuisance parameters in the problem, i.e., parameters that are needed for a correct description of the data but are otherwise not of interest (key to dealing with systematic uncertainties).

17 Large-sample approximations for prototype analysis using profile likelihood ratio Search for signal in a region of phase space; result is histogram of some variable x giving numbers: G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 signal where background strength parameter Assume the n i are Poisson distributed with expectation values Cowan, Cranmer, Gross, Vitells, arXiv: , EPJC 71 (2011) 1554

18 Prototype analysis (II) Often also have a subsidiary measurement that constrains some of the background and/or shape parameters: Assume the m i are Poisson distributed with expectation values G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 nuisance parameters (  s,  b,b tot ) Likelihood function is

19 The profile likelihood ratio Base significance test on the profile likelihood ratio: G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 maximizes L for Specified  maximize L The likelihood ratio of point hypotheses gives optimum test (Neyman-Pearson lemma). The profile LR hould be near-optimal in present analysis with variable  and nuisance parameters .

20 Test statistic for discovery Try to reject background-only (  = 0) hypothesis using G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 i.e. here only regard upward fluctuation of data as evidence against the background-only hypothesis. Note that even though here physically  ≥ 0, we allow to be negative. In large sample limit its distribution becomes Gaussian, and this will allow us to write down simple expressions for distributions of our test statistics.

21 p-value for discovery G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Large q 0 means increasing incompatibility between the data and hypothesis, therefore p-value for an observed q 0,obs is will get formula for this later From p-value get equivalent significance,

22 Expected (or median) significance / sensitivity When planning the experiment, we want to quantify how sensitive we are to a potential discovery, e.g., by given median significance assuming some nonzero strength parameter  ′. G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 So for p-value, need f(q 0 |0), for sensitivity, will need f(q 0 |  ′),

23 Test statistic for upper limits For purposes of setting an upper limit on  one may use G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Note for purposes of setting an upper limit, one does not regard an upwards fluctuation of the data as representing incompatibility with the hypothesized . From observed q  find p-value: 95% CL upper limit on  is highest value for which p-value is not less than where

24 Alternative test statistic for upper limits Assume physical signal model has  > 0, therefore if estimator for  comes out negative, the closest physical model has  = 0. Therefore could also measure level of discrepancy between data and hypothesized  with G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Performance not identical to but very close to q  (of previous slide). q  is simpler in important ways: asymptotic distribution is independent of nuisance parameters.

25 Wald approximation for profile likelihood ratio To find p-values, we need: For median significance under alternative, need: G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Use approximation due to Wald (1943) sample size

26 Noncentral chi-square for  2ln  (  ) G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 If we can neglect the O(1/√N) term,  2ln  (  ) follows a noncentral chi-square distribution for one degree of freedom with noncentrality parameter As a special case, if  ′ =  then Λ = 0 and  2ln  (  ) follows a chi-square distribution for one degree of freedom (Wilks).

27 The Asimov data set To estimate median value of  2ln  (  ), consider special data set where all statistical fluctuations suppressed and n i, m i are replaced by their expectation values (the “Asimov” data set): G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Asimov value of  2ln  (  ) gives non- centrality param. Λ  or equivalently, 

28 Relation between test statistics and G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2

29 Distribution of q 0 Assuming the Wald approximation, we can write down the full distribution of q  as G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 The special case  ′ = 0 is a “half chi-square” distribution:

30 Cumulative distribution of q 0, significance From the pdf, the cumulative distribution of q  is found to be G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 The special case  ′ = 0 is The p-value of the  = 0 hypothesis is Therefore the discovery significance Z is simply

31 Relation between test statistics and G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Assuming the Wald approximation for – 2ln  (  ), q  and q  both have monotonic relation with . ~ And therefore quantiles of q , q  can be obtained directly from those οf  (which is Gaussian). ˆ ̃ ~

32 Distribution of q  Similar results for q  G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2

33 Distribution of q  G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Similar results for q  ̃ ̃

34 Monte Carlo test of asymptotic formula G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Here take  = 1. Asymptotic formula is good approximation to 5  level (q 0 = 25) already for b ~ 20.

35 Monte Carlo test of asymptotic formulae G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 For very low b, asymptotic formula underestimates Z 0. Then slight overshoot before rapidly converging to MC value. Significance from asymptotic formula, here Z 0 = √q 0 = 4, compared to MC (true) value.

36 Monte Carlo test of asymptotic formulae G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Asymptotic f (q 0 |1) good already for fairly small samples. Median[q 0 |1] from Asimov data set; good agreement with MC.

37 Monte Carlo test of asymptotic formulae G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Consider again n ~ Poisson (  s + b), m ~ Poisson(  b) Use q  to find p-value of hypothesized  values. E.g. f (q 1 |1) for p-value of  =1. Typically interested in 95% CL, i.e., p-value threshold = 0.05, i.e., q 1 = 2.69 or Z 1 = √q 1 = Median[q 1 |0] gives “exclusion sensitivity”. Here asymptotic formulae good for s = 6, b = 9.

38 Monte Carlo test of asymptotic formulae G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Same message for test based on q . q  and q  give similar tests to the extent that asymptotic formulae are valid. ~ ~

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 39 Setting limits on Poisson parameter Consider again the case of finding n = n s + n b events where n b events from known processes (background) n s events from a new process (signal) are Poisson r.v.s with means s, b, and thus n = n s + n b is also Poisson with mean = s + b. Assume b is known. Suppose we are searching for evidence of the signal process, but the number of events found is roughly equal to the expected number of background events, e.g., b = 4.6 and we observe n obs = 5 events. → set upper limit on the parameter s. The evidence for the presence of signal events is not statistically significant,

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 40 Upper limit for Poisson parameter Find the hypothetical value of s such that there is a given small probability, say,  = 0.05, to find as few events as we did or less: Solve numerically for s = s up, this gives an upper limit on s at a confidence level of 1 . Example: suppose b = 0 and we find n obs = 0. For 1  = 0.95, →

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 41 Calculating Poisson parameter limits To solve for s lo, s up, can exploit relation to  2 distribution: Quantile of  2 distribution For low fluctuation of n the formula can give negative result for s up ; i.e. confidence interval is empty.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 42 Limits near a physical boundary Suppose e.g. b = 2.5 and we observe n = 0. If we choose CL = 0.9, we find from the formula for s up Physicist: We already knew s ≥ 0 before we started; can’t use negative upper limit to report result of expensive experiment! Statistician: The interval is designed to cover the true value only 90% of the time — this was clearly not one of those times. Not uncommon dilemma when limit of parameter is close to a physical boundary.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 43 Expected limit for s = 0 Physicist: I should have used CL = 0.95 — then s up = Even better: for CL = we get s up = 10  ! Reality check: with b = 2.5, typical Poisson fluctuation in n is at least √2.5 = 1.6. How can the limit be so low? Look at the mean limit for the no-signal hypothesis (s = 0) (sensitivity). Distribution of 95% CL limits with b = 2.5, s = 0. Mean upper limit = 4.44

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 44 The Bayesian approach to limits In Bayesian statistics need to start with ‘prior pdf’  (  ), this reflects degree of belief about  before doing the experiment. Bayes’ theorem tells how our beliefs should be updated in light of the data x: Integrate posterior pdf p(  | x) to give interval with any desired probability content. For e.g. n ~ Poisson(s+b), 95% CL upper limit on s from

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 45 Bayesian prior for Poisson parameter Include knowledge that s ≥0 by setting prior  (s) = 0 for s<0. Could try to reflect ‘prior ignorance’ with e.g. Not normalized but this is OK as long as L(s) dies off for large s. Not invariant under change of parameter — if we had used instead a flat prior for, say, the mass of the Higgs boson, this would imply a non-flat prior for the expected number of Higgs events. Doesn’t really reflect a reasonable degree of belief, but often used as a point of reference; or viewed as a recipe for producing an interval whose frequentist properties can be studied (coverage will depend on true s).

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 46 Bayesian interval with flat prior for s Solve numerically to find limit s up. For special case b = 0, Bayesian upper limit with flat prior numerically same as classical case (‘coincidence’). Otherwise Bayesian limit is everywhere greater than classical (‘conservative’). Never goes negative. Doesn’t depend on b if n = 0.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 47 Priors from formal rules Because of difficulties in encoding a vague degree of belief in a prior, one often attempts to derive the prior from formal rules, e.g., to satisfy certain invariance principles or to provide maximum information gain for a certain set of measurements. Often called “objective priors” Form basis of Objective Bayesian Statistics The priors do not reflect a degree of belief (but might represent possible extreme cases). In a Subjective Bayesian analysis, using objective priors can be an important part of the sensitivity analysis.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 48 Priors from formal rules (cont.) In Objective Bayesian analysis, can use the intervals in a frequentist way, i.e., regard Bayes’ theorem as a recipe to produce an interval with certain coverage properties. For a review see: Formal priors have not been widely used in HEP, but there is recent interest in this direction; see e.g. L. Demortier, S. Jain and H. Prosper, Reference priors for high energy physics, Phys. Rev. D 82 (2010) , arxiv: (Feb 2010)

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 49 Jeffreys’ prior According to Jeffreys’ rule, take prior according to where is the Fisher information matrix. One can show that this leads to inference that is invariant under a transformation of parameters. For a Gaussian mean, the Jeffreys’ prior is constant; for a Poisson mean  it is proportional to 1/√ .

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 50 Jeffreys’ prior for Poisson mean Suppose n ~ Poisson(  ). To find the Jeffreys’ prior for , So e.g. for  = s + b, this means the prior  (s) ~ 1/√(s + b), which depends on b. Note this is not designed as a degree of belief about s.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 51 Bayesian limits with uncertainty on b Uncertainty on b goes into the prior, e.g., Put this into Bayes’ theorem, Marginalize over the nuisance parameter b, Then use p(s|n) to find intervals for s with any desired probability content.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 52

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 53 MCMC basics: Metropolis-Hastings algorithm Goal: given an n-dimensional pdf generate a sequence of points 1) Start at some point 2) Generate Proposal density e.g. Gaussian centred about 3) Form Hastings test ratio 4) Generate 5) If else move to proposed point old point repeated 6) Iterate

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 54 Metropolis-Hastings (continued) This rule produces a correlated sequence of points (note how each new point depends on the previous one). For our purposes this correlation is not fatal, but statistical errors larger than naive The proposal density can be (almost) anything, but choose so as to minimize autocorrelation. Often take proposal density symmetric: Test ratio is (Metropolis-Hastings): I.e. if the proposed step is to a point of higher, take it; if not, only take the step with probability If proposed step rejected, hop in place.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 55 More on priors Suppose we measure n ~ Poisson(s+b), goal is to make inference about s. Suppose b is not known exactly but we have an estimate b with uncertainty  b. For Bayesian analysis, first reflex may be to write down a Gaussian prior for b, But a Gaussian could be problematic because e.g. b ≥ 0, so need to truncate and renormalize; tails fall off very quickly, may not reflect true uncertainty. ˆ

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 56 Gamma prior for b What is in fact our prior information about b? It may be that we estimated b using a separate measurement (e.g., background control sample) with m ~ Poisson(  b) (  = scale factor, here assume known) Having made the control measurement we can use Bayes’ theorem to get the probability for b given m, If we take the “original” prior  0 (b) to be to be constant for b ≥ 0, then the posterior  (b|m), which becomes the subsequent prior when we measure n and infer s, is a Gamma distribution with: mean = (m + 1) /  standard dev. = √(m + 1) / 

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 57 Gamma distribution

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 58 Frequentist approach to same problem In the frequentist approach we would regard both variables n ~ Poisson(s+b) m ~ Poisson(  b) as constituting the data, and thus the full likelihood function is Use this to construct test of s with e.g. profile likelihood ratio Note here that the likelihood refers to both n and m, whereas the likelihood used in the Bayesian calculation only modeled n.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 59 Extra Slides

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 60 Example: ALEPH Higgs search p-value (1 – CL b ) of background only hypothesis versus tested Higgs mass measured by ALEPH Experiment Possible signal? Phys.Lett.B565:61-75,2003. hep-ex/

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 61 Example: LEP Higgs search Not seen by the other LEP experiments. Combined analysis gives p-value of background-only hypothesis of 0.09 for m H = 115 GeV. Phys.Lett.B565:61-75,2003. hep-ex/

62 Example 2: Shape analysis G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Look for a Gaussian bump sitting on top of:

63 Monte Carlo test of asymptotic formulae G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Distributions of q  here for  that gave p  = 0.05.

64 Using f(q  |0) to get error bands G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 We are not only interested in the median [q μ |0]; we want to know how much statistical variation to expect from a real data set. But we have full f(q  |0); we can get any desired quantiles.

65 Distribution of upper limit on  G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 ±1  (green) and ±2  (yellow) bands from MC; Vertical lines from asymptotic formulae

66 Limit on  versus peak position (mass) G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 ±1  (green) and ±2  (yellow) bands from asymptotic formulae; Points are from a single arbitrary data set.

67 Using likelihood ratio L s+b /L b G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Many searches at the Tevatron have used the statistic likelihood of  = 1 model (s+b) likelihood of  = 0 model (bkg only) This can be written

68 Wald approximation for L s+b /L b G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Assuming the Wald approximation, q can be written as i.e. q is Gaussian distributed with mean and variance of To get  2 use 2 nd derivatives of lnL with Asimov data set.

69 Example with L s+b /L b G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 Consider again n ~ Poisson (  s + b), m ~ Poisson(  b) b = 20, s = 10,  = 1. So even for smallish data sample, Wald approximation can be useful; no MC needed.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 70 Discovery significance for n ~ Poisson(s + b) Consider again the case where we observe n events, model as following Poisson distribution with mean s + b (assume b is known). 1) For an observed n, what is the significance Z 0 with which we would reject the s = 0 hypothesis? 2) What is the expected (or more precisely, median ) Z 0 if the true value of the signal rate is s?

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 71 Gaussian approximation for Poisson significance For large s + b, n → x ~ Gaussian( ,  ),  = s + b,  = √(s + b). For observed value x obs, p-value of s = 0 is Prob(x > x obs | s = 0),: Significance for rejecting s = 0 is therefore Expected (median) significance assuming signal rate s is

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 72 Better approximation for Poisson significance Likelihood function for parameter s is or equivalently the log-likelihood is Find the maximum by setting gives the estimator for s:

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 73 Approximate Poisson significance (continued) The likelihood ratio statistic for testing s = 0 is For sufficiently large s + b, (use Wilks’ theorem), To find median[Z 0 |s+b], let n → s + b (i.e., the Asimov data set): This reduces to s/√b for s << b.

G. Cowan Statistics for HEP / LAL Orsay, 3-5 January 2012 / Lecture 2 74 n ~ Poisson(  s+b), median significance, assuming  = 1, of the hypothesis  = 0 “Exact” values from MC, jumps due to discrete data. Asimov √q 0,A good approx. for broad range of s, b. s/√b only good for s « b. CCGV, arXiv: