G. Cowan RHUL Physics Comment on use of LR for limits page 1 Comment on definition of likelihood ratio for limits ATLAS Statistics Forum CERN, 2 September, 2009 Glen Cowan Physics Department Royal Holloway, University of London

G. Cowan RHUL Physics Comment on use of LR for limits page 2 Introduction At the Statistics Forum on N. Andari presented a study (Orsay & Wisconsin) showing that a modified definition for the likelihood ratio leads to a sampling distribution that accurately follows the half-chi-square distribution. This offered the possibility to increase greatly the ease and accuracy with which we can compute exclusion limits, even for small samples. At the time some (at least GC and EG) did not fully understand how this could work, so we thought through a simple example. Conclusions agree with approach of Andari et al. Purpose of present talk is to present this example; see also attached note.

G. Cowan RHUL Physics Comment on use of LR for limits page 3 The simple problem Suppose outcome of measurement is Gaussian distributed x with expectation value and variance s and b are contributions from signal and background, take  as a known constant;  is strength parameter. The likelihood function for the parameter of interest  is Suppose goal is to set an upper limit on  given a measurement x.

G. Cowan RHUL Physics Comment on use of LR for limits page 4 The likelihood ratio To test a value of , construct likelihood ratio: Suppose on physical grounds  should be positive, then the maximum of L(  ) from the allowed range of  is from Usually use logarithmic equivalent  2 ln (  ):

G. Cowan RHUL Physics Comment on use of LR for limits page 5 Likelihood ratio for upper limit For an upper limit on  one uses the test statistic which, putting together the ingredients, becomes

G. Cowan RHUL Physics Comment on use of LR for limits page 6 p-value for exclusion To quantify level of agreement between observed x and hypothesized , calculate p-value Note if x ~ Gaussian(  s+b,  ), then the quantity follows a chi-square pdf for 1 d.o.f. But the distribution of q  is more complicated (not chi-square), 95% CL upper limit on  is value for which p  = 0.05.

G. Cowan RHUL Physics Comment on use of LR for limits page 7 Likelihood ratio without constraint on  ^ Andari et al. propose to define an unphysical estimator which goes negative if x < b. Using this define then a corresponding test statistic for upper limits:

G. Cowan RHUL Physics Comment on use of LR for limits page 8 Exclusion significance from q  ′ From the definition of q  ′ one can see its pdf must be a half-chi-square distribution, i.e., a delta function at zero when x >  s + b, and a chi-square pdf for x ≤  s + b. Therefore (see CSC note), the significance from an observed value q  ′ is given by the simple relation p-value of 0.05 corresponds to Z = 1.64.

G. Cowan RHUL Physics Comment on use of LR for limits page 9 Comparison of test variables Both q  and q  ′ are shown here as a function of x for  =1, s=10, b=20,  2 =20. Note they are equal for b ≤ x ≤  s + b.

G. Cowan RHUL Physics Comment on use of LR for limits page 10 Equivalence of q  and q  ′ It is easy to see that the two test variables q  and q  ′ are monotonically related: and therefore they represent equivalent tests of .

G. Cowan RHUL Physics Comment on use of LR for limits page 11 Relation to study by Andari et al. (Eilam) Andari et al. presented the following table of the fraction of toy experiments with values of the test statistic below certain levels: Chi-square and “exact” (counting) formulae give same fractions for “median”, but not median +1 , +2 . This is because in this example it corresponds to having x < b, i.e., this where q  and q  ′ are different. But for example, median  1  or  2  would correspond to the region where q  and q  ′ are equal, and the two methods there would agree.

G. Cowan RHUL Physics Comment on use of LR for limits page 12 Conclusion on q  ′ Both test variables, q  and q  ′, give equivalent tests, because of their monotonic relation. If one were to work out (with difficulty) the exact sampling pdf of q , and to compute from it the p-value, and from it the significance Z, then it would be the same as from the simple formula Z = √ q  ′ using the same value of the observation x. We still regard only positive  as physical, but allow its estimator to go negative effectively as a mathematical trick to get the desired p-value. Also easy to show that this likelihood ratio gives a test equivalent to the ratio used for the LEP analyses (see attached note):

G. Cowan RHUL Physics Comment on use of LR for limits page 13 Extra slides

G. Cowan RHUL Physics Comment on use of LR for limits page 14 Example from CSC book E.g. H →  from the CSC combination chapter used the statistic q  (as did all other channels). MC studies show that distribution of q  departs significantly from half-chi-square form. If we had used q  ′, the agreement with half-chi-square would be much closer, even for low luminosity.