G. Cowan RHUL Physics LR test to determine number of parameters page 1 Likelihood ratio test to determine best number of parameters ATLAS Statistics Forum CERN, 18 February, 2009 Glen Cowan Physics Department Royal Holloway, University of London

G. Cowan RHUL Physics LR test to determine number of parameters page 2 Introduction Present study motivated by discussions with Eilam, Stephan Horner, Sascha Caron, et al., regarding Stephan's presentation on SUSYFit at 3 December 2008 Statistics Forum. Discussions also in Top Properties meeting (16 Dec 08) and Exotics meeting (22 Jan 08). Basic idea is to develop general method for increasing number of parameters in a model; stop when fit is OK. Systematics in the original model are then included in the statistical errors of the extended model. A draft note is attached on the agenda page; also at

G. Cowan RHUL Physics LR test to determine number of parameters page 3 Determining distributions: systematics E.g. M ll distribution from Z'→dilepton search (CSC Book p 1709), uses 4-parameter function for signal. Sidebands provide estimate of background. So nothing in real analysis from MC, but... Still should consider some systematic due to fact that assumed parametric functions not perfect.

G. Cowan RHUL Physics LR test to determine number of parameters page 4 A general strategy (see attached note) Suppose one needs to know the shape of a distribution. Initial model (e.g. MC) is available, but known to be imperfect. Q: How can one incorporate the systematic error arising from use of the incorrect model? A: Improve the model. That is, introduce more adjustable parameters into the model so that for some point in the enlarged parameter space it is very close to the truth. Then use profile the likelihood with respect to the additional (nuisance) parameters. The correlations with the nuisance parameters will inflate the errors in the parameters of interest. Difficulty is deciding how to introduce the additional parameters.

G. Cowan RHUL Physics LR test to determine number of parameters page 5 Comparing model vs. data In the example shown, the model and data clearly don't agree well. To compare, use e.g. Model number of entries n i in ith bin as ~Poisson( i ) Will follow chi-square distribution for N dof for sufficiently large n i.

G. Cowan RHUL Physics LR test to determine number of parameters page 6 Model-data comparison with likelihood ratio This is very similar to a comparison based on the likelihood ratio where L( ) = P(n; ) is the likelihood and the hat indicates the ML estimator (value that maximizes the likelihood). Here easy to show that Equivalently use logarithmic variable If model correct, q ~ chi-square for N degrees of freedom.

G. Cowan RHUL Physics LR test to determine number of parameters page 7 p-values Using either  2 P or q, state level of data-model agreement by giving the p-value: the probability, under assumption of the model, of obtaining an equal or greater incompatibility with the data relative to that found with the actual data: where (in both cases) the integrand is the chi-square distribution for N degrees of freedom,

G. Cowan RHUL Physics LR test to determine number of parameters page 8 A simple example The naive model (a) could have been e.g. from MC (here statistical errors suppressed; point is to illustrate how to incorporate systematics.) 0th order model True model (Nature) Data

G. Cowan RHUL Physics LR test to determine number of parameters page 9 Comparison with the 0th order model The 0th order model gives q = 258.8, p  ×  

G. Cowan RHUL Physics LR test to determine number of parameters page 10 Enlarging the model Here try to enlarge the model by multiplying the 0th order distribution by a function s: where s(x) is a linear superposition of Bernstein basis polynomials of order m:

G. Cowan RHUL Physics LR test to determine number of parameters page 11 Bernstein basis polynomials

G. Cowan RHUL Physics LR test to determine number of parameters page 12 Enlarging the parameter space Using increasingly high order for the basis polynomials gives an increasingly flexible function. At each stage compare the p-value to some threshold, e.g., 0.1 or 0.2, to decide whether to include the additional parameter. Now iterate this procedure, and stop when the data do not require addition of further parameters based on the likelihood ratio test. Once the enlarged model has been found, simply include it in any further statistical procedures, and the statistical errors from the additional parameters will account for the systematic uncertainty in the original model.

G. Cowan RHUL Physics LR test to determine number of parameters page 13 Fits using increasing numbers of parameters Stop here

G. Cowan RHUL Physics LR test to determine number of parameters page 14 Goodness-of-fit for the extended models q gives overall goodness-of-fit q compares model with n par parameters to that with n par +1 p-values

G. Cowan RHUL Physics LR test to determine number of parameters page 15 Summary Example shown here uses a very general idea; similar philosophy applied in many analyses (cf. choosing order of a polynomial for LS fit). Example here assumes distribution can be corrected by a scale factor; need somewhat different strategy for the tail of a distribution, where MC bin contents go to zero. What to do if e.g. overall goodness-of-fit not great, but additional parameters do not help? (Tom LeCompte: F-test using ratio of chi-squares?) How to proceed if the additional parameters add too much flex- ibility, e.g., what if normalization is well known, but not, say, slope? Stephan Horner et al. have done similar things with SUSYFit (next talk).