G. Cowan, RHUL Physics Statistics for early physics page 1 Statistics jump-start for early physics ATLAS Statistics Forum EVO/Phone, 4 May, 2010 Glen Cowan (RHUL) Eilam Gross (Weizmann Institute)

G. Cowan, RHUL Physics Statistics for early physics page 2 Issues For early physics we need to finalize methodologies for setting limits (and quoting discovery significance). For today focus mainly on limits Need to agree with CMS what procedures we will use (a) for comparison (also with e.g. Tevatron) (b) for combination Review of methods in use Profile likelihood CLs Bayesian Status of software tools (RooStats, …) Look-elsewhere-effect (an issue for discovery, not limits).

G. Cowan, RHUL Physics Statistics for early physics page 3 Frequentist discovery and limits For frequentist methods, focus on p-value or equivalent significance Z =   (1 – p) “Discovery” if p-value of background-only 5) Can compute e.g. median discovery significance assuming different signal models. (In some cases may not have well-defined signal model.) Exclusion at CL = 1 –  if p <  (e.g.  = 0.05) Can compute e.g. median limit assuming background-only. For exclusion, parameter (model) being tested is the null; compute power relative to background-only alternative: Power = P(reject model(parameter) | background only)

G. Cowan, RHUL Physics Statistics for early physics page 4 Bayesian discovery For Bayesian discovery, compute Bayes factor to compare e.g. model i (Higgs) to j (no Higgs): Gives posterior odds if prior odds were Avoids dependence on unlikely data outcomes that were never seen (cf. tail probabilities for p-values.) Work needed here, mainly on computational issues.

G. Cowan, RHUL Physics Statistics for early physics page 5 Bayesian limits Just integrate the posterior probability obtained from Bayes thm, Not widely used so far in ATLAS (?) but will need anyway for comparison with Tevatron. Despite recent discussion on reference priors (a la Bernardo, Jeffereys…), not aware of realistic applications in HEP. Need survey of code and of who is using this in ATLAS. Await outcome of CMS discussion on priors.

G. Cowan, RHUL Physics Statistics for early physics page 6 Systematics Connect systematic to nuisance parameters. Then form e.g. Profile likelihood: Marginal likelihood: and use these to construct e.g. likelihood ratios for tests. Coverage not guaranteed for all values of the nuisance params. Literature contains some variants that we do not recommend, e.g., forming a likelihood ratio and integrating it with a prior. Need to decide what to do when systematic cannot be dealt with using nuisance parameters (e.g. corrections using PYTHIA vs. HERWIG).

G. Cowan, RHUL Physics Statistics for early physics page 7 Z Bi, Z , Z N, etc. Several authors have studied various ways to obtain the significance in the simple counting experiment with systematic uncertainty on the background, e.g., Cranmer (PHYSTAT 03,05) and Cousins, Tucker, Linnemann (physics/ ). Our usual profile likelihood method with a subsidiary measurement for the background gives Z Bi, also preferred by CMS. Z N assumes a Gaussian prior for the background (truncated at zero) and in many cases is not a realistic model. Should only be used if the analyst actually believes that an average with respect to the truncated Gaussian prior is the appropriate model (not likely). CMS has flagged this as an area where we should reach agreement; do not anticipate trouble here (does ATLAS use Z N ?)

G. Cowan, RHUL Physics Statistics for early physics page 8 ATLAS practice We need to complete our survey of methods used in ATLAS. In StatForum we have made important progress in using profile likelihood ratio tests. Can get significance, limits without any toy MC (valid for large samples, in practice can even be smallish). For 95% CL limits for very low lumi, can always turn to toy MC to calibrate method.

G. Cowan, RHUL Physics Statistics for early physics page 9 CLs If we test parameter values to which we have no sensitivity (e.g. very large Higgs mass), then there is a probability of 1 – CL (e.g. 5%) that we will reject. In the CLs method the p-value is reduced according to the recipe Statistics community does not smile upon ratio of p-values; would prefer to regard parameter m as excluded if: (a) p-value of m < 0.05 (b) power of test of m with respect to background-only > some threshold (0.5?) Needs study. In any case should produce CLs result for purposes of comparison with CMS/Tevatron.

G. Cowan, RHUL Physics Statistics for early physics page 10 Choice of likelihood ratio statistic Ongoing discussion as to whether best to use LEP-style likleihood ratio or and in both cases how to deal with the nuisance parameters. In simple cases one obtains the same test from both statistics.

G. Cowan, RHUL Physics Statistics for early physics page 11 Questions to discuss Need to decide about CLs (CMS have not yet decided). Decide on methods for incorporating systematics (Zbi, ZN, …?) Decide on ATLAS recommended methods to be used in presentation of results Check the status of Roostats as a tool for hypothesis testing and for combinations.