1. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 1 Use of profile likelihood to determine systematic uncertainties ATLAS Top Properties Meeting CERN/phone, 16 December, 2008 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan

2. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 2 Eilam Gross, 4.12.08

3. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 3

4. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 4

5. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 5 The basic idea Suppose one needs to know the shape of a distribution. MC model is available, but known to be imperfect. Q: How can one incorporate the systematic error arising from use of the incorrect model? A: Improve the model. That is, introduce more adjustable parameters into the model so that for some point in the enlarged parameter space it is very close to the truth. Then use profile the likelihood with respect to the additional (nuisance) parameters. The correlations with the nuisance parameters will inflate the errors in the parameters of interest. Difficulty is deciding how to introduce the additional parameters.

6. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 6 A simple example The naive model (a) could have been e.g. from MC (here statistical errors suppressed; point is to illustrate how to incorporate systematics.) 0th order model True model (Nature) Data

7. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 7 Comparing model vs. data In the example shown, the model and data clearly don't agree well. To compare, use e.g. Model number of entries n i in ith bin as ~Poisson( i ) Will follow chi-square distribution for N dof for sufficiently large n i.

8. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 8 Model-data comparison with likelihood ratio This is very similar to a comparison based on the likelihood ratio where L( ) = P(n; ) is the likelihood and the hat indicates the ML estimator (value that maximizes the likelihood). Here easy to show that Equivalently use logarithm variable If model correct, q ~ chi-square for N degrees of freedom.

9. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 9 p-values Using either  2 P or q, state level of data-model agreement by giving the p-value: the probability, under assumption of the model, of obtaining an equal or greater incompatibility with the data relative to that found with the actual data: where (in both cases) the integrand is the chi-square distribution for N degrees of freedom,

10. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 10 Comparison with the 0th order model The 0th order model gives q = 258.8, p  

11. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 11 Enlarging the model Here try to enlarge the model by multiplying the 0th order distribution by a function s: where s(x) is a linear superposition of Bernstein basis polynomials of order m:

12. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 12 The enlarged model Using increasingly high order for the basis polynomials gives an increasingly flexible function. For any order, when all  k =1, then s(x) = 1. The models are nested, since one can write a Bernstein basis polynomial of order m  1 in terms of basis polynomials of order m:

13. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 13 How far to enlarge the model So the 0th order model corresponds to m=0,  0 = 1. Test this against the one-parameter alternative where  0 is adjustable by constructing the likelihood ratio: As before use the logarithmic variable: If the more restrictive (numerator) hypothesis is correct, then this will follow a chi-square distribution for 1 dof. So the p-value is

14. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 14 How far to enlarge the model (2) Compare the p-value to some threshold, e.g., 0.1 or 0.2, to decide whether to include the additional parameter. Now simply iterate this procedure, and stop when the data do not require addition of further parameters based on the likelihood ratio test.

15. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 15 Fits using increasing numbers of parameters

16. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 16 Sampling distribution of q Usually no need to include 4th parameter

17. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 17 Using the enlarged model Once the enlarged model has been found, simply include it in any further statistical procedures, and the statistical errors from the additional parameters will account for the systematic uncertainty in the original model, e.g. for search: For an individual search channel, n i ~ Poisson(  s+b), m i ~ Poisson(u i ). The likelihood is: Parameter of interest Here  represents all nuisance parameters

18. G. Cowan RHUL Physics Profile likelihood for systematic uncertainties page 18