G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 1 Report from the Statistics Forum ATLAS Analysis Strategies Workshop CERN, 6 September, 2007 Glen Cowan Physics Department Royal Holloway, University of London

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 2 The ATLAS Statistics Forum Initiated at Statistics Workshop in January 07 Statistics Forum started in March (Karl Jakobs) Since then, meetings ~monthly (plus ~3 joint meetings with CMS)

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 3 Current Membership Conveners:Eilam Gross, Glen Cowan “Statistics Experts”: GDC, Kyle Cranmer, Tom Junk, Bill Murray, Alex Read Tool developers:Bill Quayle, Krzysztof Ciba Representatives from ATLAS groups: Higgs: Eilam Gross & Guillaume Unal SUSY: Dan Tovey SM: Maarten Boonekamp Exotics: Samir Ferrag B-Physics: Eduard Burelo Top:Yoram Rozen Heavy Ions,... Ex officio: Ian Hinchliffe & Karl Jakobs

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 4 Mandate Define ATLAS Statistics Standards in contact with Physics Coordination and ATLAS groups. Develop, validate and approve standard statistical tools in close contact with the software group. Represent ATLAS in discussions on statistical issues with CMS. Discuss and make recommendations on statistical issues such as blind analyses, model independent searches, etc.

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 5 Web page

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 6 Hypernews

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 7 cf. BaBar Statistics Group Hypernews Detailed discussions Heavy traffic

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 8 Some ongoing activities ATLAS Statistics Book Higgs combination (different channels & ATLAS + CMS) Software development (RooStat) Sequential analysis

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 9 Twiki, i.e., a Living Document (thanks, Krzysztof) Core authors assigned, detailed outline discussed + worked examples, particularly related to systematics + FAQ, recommendations Goal: first draft by October (need to work hard...)

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 10 Higgs Combination Higgs suggested as a first major exercise for the group, but we would like to undertake similar efforts with other Physics Groups. Many people involved; see next talk by Eilam. In particular, frequentist combination based on profile likelihood very advanced (Kyle, Eilam, Ofer,...) But a brief comment on the Bayesian methods: Bayesian limits already widely used e.g. in CDF (→Tom J.). We are also investigating use of Bayes factors as a means of quantifying significance of a discovery: Higgs no Higgs Bayes factor B 10 prior odds posterior odds

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 11 Higgs combination based on event counts N independent channels, count n i events in search regions: Expected number of signal events: (  is global parameter,  = 1 for SM). Consider a fixed Higgs mass and assume SM branching ratios B i. Suggested method: constrain  with limit  up ; consider m H excluded if upper limit  up < 1.0. For discovery, compute Bayes factor for H 0 :  = 0 vs. H 1 :  = 1 Constrain expected background b i with sideband measurements:

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 12 Bayes factor ( = ratio of integrated likelihoods) Here B 10 and profile likelihood ratio close; comparisons ongoing. B 10 B 10 at  = 1 gives change in odds of SM Higgs vs. no Higgs in light of data, i.e., measures of weight of evidence provided by data in support of one hypothesis over another.

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 13 Statistical software Bill Quayle, Kyle Cranmer (from Statistics Forum) + developers of TMVA (Andreas Hoecker et al.), RooFit (Wouter Verkerke et al.), MadTools/StatTools (Wisconsin group),... (see e.g. Bill Quayle’s 15 March 07 talk) Some issues: ATLAS specific (private) or HEP-wide (public)? Use FrameworkXXX or, integrate ComponentXXX into an existing framework? Distribution ( phystat.org, bundle with ROOT?)

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 14 RooFit/RooStats Wouter Verkerke, Kyle Cranmer Developments reported in ATLAS/CMS Forum, i.e., HEP-wide, not ATLAS-specific software. E.g. Combination Toy Model (W. Verkerke July 2007 ATLAS/CMS meeting) ‘Atlas’ Combined Profile likelihood ‘CMS’ → talk by Wouter

15 Looking at data as they are recorded and compute a new p-value or likelihood ratio test every 10 pb -1. What is probability that p-value (or LR test) < 0.01 versus number of data looks ? looking 1 time at data : p(p-value < 0.01|H 0 ) = 0.01 looking sequentially 100 times at data : p(at least 1 p-value < 0.01|H 0 ) ~ 0.08 ! p-value Likelihood ratio test b = 200 fb p-Value result versus L when H 0 is true P(at least 1 p-Value < 0.01) versus L when H 0 is true Sequential analysis (Renaud Bruneliere, 7 June and 3 July 07) → ‘Look-elsewhere effect in time’. Need to understand how to best quantify / deal with this!

G. Cowan RHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 16 Conclusions, thoughts Statistics Forum working since March, several projects ongoing: Statistics Book Higgs combination Software tools Sequential analysis Want to increase interaction with Physics Groups, If your group has a statistics issue/problem, is developing a statistical tool, etc. come tell us. (Also we will come to you...)