Multichannel number counting experiments

Slides:



Advertisements
Similar presentations
Using the Profile Likelihood in Searches for New Physics / PHYSTAT 2011 G. Cowan 1 Using the Profile Likelihood in Searches for New Physics arXiv:
Advertisements

27 th March CERN Higgs searches: CL s W. J. Murray RAL.
1 LIMITS Why limits? Methods for upper limits Desirable properties Dealing with systematics Feldman-Cousins Recommendations.
Setting Limits in the Presence of Nuisance Parameters Wolfgang A Rolke Angel M López Jan Conrad, CERN.
Statistics In HEP 2 Helge VossHadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP 1 How do we understand/interpret our measurements.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem, random variables, pdfs 2Functions.
G. Cowan Statistics for HEP / NIKHEF, December 2011 / Lecture 3 1 Statistical Methods for Particle Physics Lecture 3: Limits for Poisson mean: Bayesian.
G. Cowan RHUL Physics Bayesian methods for HEP / DESY Terascale School page 1 Bayesian statistical methods for HEP Terascale Statistics School DESY, Hamburg.
Search for B     with SemiExclusive reconstruction C.Cartaro, G. De Nardo, F. Fabozzi, L. Lista Università & INFN - Sezione di Napoli.
G. Cowan RHUL Physics Comment on use of LR for limits page 1 Comment on definition of likelihood ratio for limits ATLAS Statistics Forum CERN, 2 September,
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan 2011 CERN Summer Student Lectures on Statistics / Lecture 41 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 7 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
G. Cowan Discovery and limits / DESY, 4-7 October 2011 / Lecture 3 1 Statistical Methods for Discovery and Limits Lecture 3: Limits for Poisson mean: Bayesian.
G. Cowan RHUL Physics Bayesian Higgs combination page 1 Bayesian Higgs combination using shapes ATLAS Statistics Meeting CERN, 19 December, 2007 Glen Cowan.
Statistical aspects of Higgs analyses W. Verkerke (NIKHEF)
Discovery Experience: CMS Giovanni Petrucciani (UCSD)
1 Glen Cowan Statistics Forum News Glen Cowan Eilam Gross ATLAS Statistics Forum CERN, 3 December, 2008.
G. Cowan 2009 CERN Summer Student Lectures on Statistics1 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability densities,
Statistical methods in LHC data analysis part II.2 Luca Lista INFN Napoli.
Results of combination Higgs toy combination, within and across experiments, with RooStats Grégory Schott Institute for Experimental Nuclear Physics of.
1 Methods of Experimental Particle Physics Alexei Safonov Lecture #25.
G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.
G. Cowan, RHUL Physics Discussion on significance page 1 Discussion on significance ATLAS Statistics Forum CERN/Phone, 2 December, 2009 Glen Cowan Physics.
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
1 Methods of Experimental Particle Physics Alexei Safonov Lecture #24.
G. Cowan Lectures on Statistical Data Analysis Lecture 4 page 1 Lecture 4 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
G. Cowan Computing and Statistical Data Analysis / Stat 9 1 Computing and Statistical Data Analysis Stat 9: Parameter Estimation, Limits London Postgraduate.
2005 Unbinned Point Source Analysis Update Jim Braun IceCube Fall 2006 Collaboration Meeting.
G. Cowan, RHUL Physics Statistics for early physics page 1 Statistics jump-start for early physics ATLAS Statistics Forum EVO/Phone, 4 May, 2010 Glen Cowan.
G. Cowan RHUL Physics Status of Higgs combination page 1 Status of Higgs Combination ATLAS Higgs Meeting CERN/phone, 7 November, 2008 Glen Cowan, RHUL.
LIGO- G Z August 17, 2005August 2005 LSC Meeting 1 Bayesian Statistics for Burst Search Results LIGO-T Keith Thorne and Sam Finn Penn State.
In Bayesian theory, a test statistics can be defined by taking the ratio of the Bayes factors for the two hypotheses: The ratio measures the probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan RHUL Physics Statistical Issues for Higgs Search page 1 Statistical Issues for Higgs Search ATLAS Statistics Forum CERN, 16 April, 2007 Glen Cowan.
S. Ferrag, G. Steele University of Glasgow. RooStats and MClimit comparison Exercise to use RooStats by an MClimit-formatted person: – Use two programs.
Investigation on CDF Top Physics Group Ye Li Graduate Student UW - Madison.
G. Cowan SOS 2010 / Statistical Tests and Limits -- lecture 31 Statistical Tests and Limits Lecture 3 IN2P3 School of Statistics Autrans, France 17—21.
G. Cowan SLAC Statistics Meeting / 4-6 June 2012 / Two Developments 1 Two developments in discovery tests: use of weighted Monte Carlo events and an improved.
23 Jan 2012 Background shape estimates using sidebands Paul Dauncey G. Davies, D. Futyan, J. Hays, M. Jarvis, M. Kenzie, C. Seez, J. Virdee, N. Wardle.
G. Cowan CERN Academic Training 2012 / Statistics for HEP / Lecture 21 Statistics for HEP Lecture 2: Discovery and Limits Academic Training Lectures CERN,
Discussion on significance
Hands-on Session RooStats and TMVA Exercises
(Day 3).
Status of the Higgs to tau tau
arXiv:physics/ v3 [physics.data-an]
The expected confident intervals for triple gauge coupling parameter
Statistical Methods used for Higgs Boson Searches
Confidence Intervals and Limits
Two Interpretations of What it Means to Normalize the Low Energy Monte Carlo Events to the Low Energy Data Atms MC Atms MC Data Data Signal Signal Apply.
CMS RooStats Higgs Combination Package
Statistics for the LHC Lecture 3: Setting limits
Grégory Schott Institute for Experimental Nuclear Physics
Strategy 6th December 2010 Bill Murray LHC-HCG
Lecture 4 1 Probability (90 min.)
Incorporating systematic uncertainties into upper limits
B  at B-factories Guglielmo De Nardo Universita’ and INFN Napoli
TAE 2018 / Statistics Lecture 3
Likelihood intervals vs. coverage intervals
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
Statistical Methods for the LHC
Statistical Tests and Limits Lecture 2: Limits
TAE 2017 / Statistics Lecture 2
Feldman-Cousins Method
Application of CLs Method to ATLAS Higgs Searches
Introduction to Statistics − Day 4
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
Statistics for HEP Roger Barlow Manchester University
Computing and Statistical Data Analysis / Stat 10
Presentation transcript:

Multichannel number counting experiments V.Zhukov M.Bonsch KIT, Karlsruhe

Multichannel searches at LHC Beyond SM(eg. SUSY) can manifest in different signal topologies. LHC detectors can identify different objects (lepton, jets, MET, etc) → consider different exclusive topologies 1) independently, and then 2) combine them. Eg. mSUGRA 95%CL for different topologies Combination has clear benefits: Extend statistics by combining many (hundredths) searches with relatively low signal efficiency each, increases sensitivity. Consistent treatment of systematics Consistency check of different measurements can be used to constrain H1 model parameters … and challenges: How to combine/split topologies in presence of correlated systematics? How to optimize selection(S/B) for each channel for optimum combination? Which method to use for confidence intervals, hypothesis testing? Effect of the initial and boundary conditions (zero observation, unphysical systematics, truncation, range for common signal, flip-flop for combinations) When it make sense to use auxiliary measurements and how to combine them?....

Framework Use RooFit/Stats framework in root_v27.6 *) Likelihood formalism, common workspace... Trust the RooStat implementations. Experimentalist approach, no attempt to improve the existing codes. Bugs(features) are not excluded... Simple number counting test model to check basic behavior of different methods. Run thousands jobs on cluster. Show some part of results... *) Thanks to RooStat developers and CMS Stat committee for consultations

Statistical model (as in RooFit) Single channel (simple model for demonstration): Nobs observation seff efficiency of signal s=[0, 3*(Nobs+Nbkg)] Nbkg background expectation sigmb relative background uncertainties. Here consider truncated Gaussian b' =[0., 5*sigmb]. Other shapes(Lognormal, Gamma) have similar qualitative behavior. Poisson::signal(Nobs, s*seff+b') Gaussian::sysb(b', Nbkg, sigmb) (“nuis”, “ b' ”) L(n|x)= PROD::model1(signal,sysb) Combined model: Systematics: use either individual nuisance or common systematics L(n|x)= PROD::combmodel(model1,model2,....) (“nuis”,”b1', b2',...”) (“nuis”, “ b' ”) //common systematics Auxiliary measurement: 'Data-driven' background estimation. Constrain background in the signal region by auxiliary measurements. Introduce extra Poisson term and systematics on tau=b/c relating signal(b) and sideband control regions (c) Poisson::signal(Nobs, s*effs+c'*tau') Poisson::aux(Naux, c') Gaussian::systau(tau ', tau, sigmtau) (“nuis”, “ c' , tau' ”) L(n|x)= PROD::model(signal, aux, systau)

Statistical methods Profile Liklehood (PLC): Based on Lhood ratio and Wilk's theorem Minuit for nuisance Unified, Feldman-Cousins(FC): Neyman construction with ordering Minuit for nuisances ProfileLikelihoodCalculator plc(*data, *smodel); plc.SetConfidenceLevel(cls); LikelihoodInterval* plInt = plc.GetInterval(); pl_L= plInt->LowerLimit( *w->var("s") ); pl_U= plInt->UpperLimit( *w->var("s") ); HypoTestResult* plh=plc.GetHypoTest(); pl_sig=plh->Significance(); FeldmanCousins fc(*data, *smodel); fc.SetConfidenceLevel(cls); fc.FluctuateNumDataEntries(false); fc.UseAdaptiveSampling(true); fc.SetNBins(100); PointSetInterval*fcInt=(PointSetInterval*) fc.GetInterval(); fc_L= fcInt->LowerLimit( *w->var("s") ); fc_U= fcInt->UpperLimit( *w->var("s") ); RooStat presentations: Bayes credible intervals (Bayes): use flat prior on signal here Numerical integration Hybrid (Hyb) modified Cousins-Highland. MC toys for nuisance integration, use CLs BayesianCalculator bc(*data, *smodel); bc.SetTestSize(1.-cls); bc.SetLeftSideTailFraction(0.5); //0 for central SimpleInterval* bInt = bc.GetInterval(); bayes_L= bInt->LowerLimit( ); bayes_U= bInt->UpperLimit( ); HybridCalculatorOriginal hyb(*data,*smodel,*bmodel); hyb.PatchSetExtended(false); hyb.SetTestStatistic(1); hyb.UseNuisance(true); hyb.SetNuisancePdf(*w->pdf("prior_nuis")); hyb.SetNuisanceParameters(*w->set("nuis")); HypoTestInverter myInv(hyb,s); myInv.UseCLs(true); myInv.SetTestSize(1.0-cls); hyb.SetNumberOfToys(5000); myInv.RunAutoScan(lr1,lr2,myInv.Size()/2.,0.01,1); HypoTestInverterResult* results = myInv.GetInterval(); hyb_U = results->UpperLimit(); hyb_L = results->LowerLimit(); HybridResult* hcResult = hyb.GetHypoTest(); hyb_significance = hcResult->Significance(); Binominal significance Z_Bi Correspondance of on/off and sigmb problem Analytical for single channel(arx.0702156) double tau=_Nbkg/(sigmb*Nbkg*sigmb*Nbkg); double noff=tau*_Nbkg; double pBi=TMath:: BetaIncomplete(1/(1.+tau),_Nobs,noff+1.); double Z_Bi=sqrt(2.)*TMath::ErfInverse(1-2.*pBi);

Confidence intervals Calculate central 95%CL upper and lower limits and one sided upper limit versus Nobs and Nbkg with PLC*), FC and Bayes. Test frequentist coverage (Neyman construction) for PLC, FC (should cover) and Bayes credible (not really motivated) Use different models: 1) Single channel without and with Gaussian rel. systematics sb=0 - 0.5 Compare different methods (with different treatment of systematics) 2) Combined Nch=5 identical channels with Nobs/Nch, Nbkg/Nch, Seff/Nch without systematics and the same individual systematics, or correlated. This 'split' combined model should be equivalent to the single channel with the Nobs and Nbkg. 3) Combined Nch channels, but with observations only in one channel Nobs(1)=Nobs, others are Nobs(2,3,4,5..)=0. Check treatment of 'zero' observation. *) upper limits for PLC are calculated with the offset of CL to 90%.

Confidence intervals for single channel 1. CL vs Nbkg, Nobs, sb=0 (no systematics) Central intervals One sided Upper limit Some differences at large Nbkg>20 small Nobs<20 2. Nbkg, Nobs, sb=0.5 Systematics boundaries? flip-flop in Bayes?

Intervals for the combined model 1. Nbkg, Nobs, sb=0 (no systematics) Central intervals 5 identical channels Without systematics: similar to single channel (as expected) For large Nbkg: Methods differ For Nobs=0: PLC doesn't work(no Wilks!) Bayes is not sensitive to bkg FC improves with large Nbkg 2. Nbkg, Nobs, sb=0.5 (non correlated) With systematics: Effect of systematics is changed or moved to higher Nbkg. Not equivalent to single channel.

Intervals for the single channel and combined More comparison (with systematics, without systematics there is no difference) Bayes credible Profile Lhood Feldman-Cousins single channel(sb=0.5) 5 identical channels b/nch, nobs/nch 5 identical channels with only one with observation Good agreement for single channel and split model(5 identical channels) at small Nbkg but significant difference with large Nbkg for all methods, especially FC. Tighter limits for the only one channel with observations.

Confidence intervals with correlated systematics Combined model: 5 channels Correlated systematics can have significant effect in Bayesian and FC intervals at large Nbkg, Nobs and lesser for PLC .

Coverage of central intervals Single channel Five channels 200toys per point Nbkg=2 sigmb=0.5 Signal s=0-20 (s=2 → Z~5) Reasonable coverage(some overcoverage) in single channel and combined model for all methods in presence of background Nbkg=0 sigmb=0.5 Less stable without background

Hypothesis testing Consider channels with different S/B and different systematics(Gauss, Gamma), calculate significance with PLC, Hybrid, Z_Bi Eg. mSUGRA search 1) Saturation behavior. Increase of statistics(lumi) Important for combination of channels with different S/B and systematics. Defines selection optimization strategy for each combination. Related: 'coverage' of methods used for significance 2) Systematics correlations. Correlations can decrease or increase significance of combined model (similar to auxiliary measurements). What are the conditions? 3) Auxiliary measurements (data driven background estimation) Can constrain some model parameters(bkg) with extra measurements Provided there is some benefit comparing with MC truth uncertainties

Significance vs luminosity S/B=1 sb~0 S/B=1 sb~0.4 S/B=10 sb~0.4 Without systematics: excellent agreement among methods With systematics : Some differences and different asymptotic behavior Saturation for models with different S/B and systematics. Good scaling for PLC, S/B is the right parameters for optimization of selection. Z_Bi is always below (needs modification in on/off-sigmb formulas)

Correlated systematics Without correlations the combined significance Z~S z/n, with systematics depends... Eg. Consider 2 channels with the same signal s1=s2=50 and bkg b1=b2=100 and systematics sigmb=0.1-0.7 correlated (Zcorr) and uncorrelated (Zuncorr): DZ/Z=(Zcorr-Zuncorr)/Zuncorr (use PLC) The effect of correlations on significance can change with luminosity When channels are in disbalance the correlation can be beneficial (red regions) otherwise ~20% decrease(green) Combined significance of 2 ch. S/B=2 and S/B=1.4 sigmb=0.5 with b1=1 and b2=5.

Auxiliary measurements Main idea: reduces systematics in absolute normalization, but systematics in shape remains via tau factor t=bkg-in-signal-region /bkg-in-sideband Data driven(dd) bkg constrain can be used if the main uncertainties in MC are expected from this absolute normalization. It can be beneficial when sb(MC)2 > naux+st (MC)2 (naux measurement in side band) And: Have to compare aux measurements itself with MC predictions anyway. 1. If there is a large difference can't use dd bkg because the st (MC) is undefined 2. If there are no difference why we need it? Better (and more correct) to use all measurements in G.O.F test to tune the MC. Eg. Consider 3 models (S/B=1) 1. sb(MC)=0.2 2. dd bkg. with st=0.2 3. dd bkg with st=0.15 Evolution of significance (PLC) with luminosity.

Coverage test of significance PLC : sb=0 -0.5 gaussian Z_Bi: DZ=Ztrue- Zclaim Huge overcoverage for high systematics (>0.3), especially for Z_Bi getting even worse for high significance (Z>3) and sensitive to the systematics shape. Is that what you call conservative?

Summary 1. RooFit/Stats is an excellent tool for statistical modeling. used in real physics analysis (yet for rather simple models) 2. Different methods: ProfileLikelihood, Feldman-Cousins, Bayes calculator gives rather similar results for confidence intervals without systematics (Poisson). Similarly the Hybrid, LikelihoodRatio and Hybrid methods are in good agreement in hypothesis testing without systematics. With the combined multichannel model and some realistic systematics results start to diverge substantially. Confidence intervals: ProfileLhood doesn't work with zero observation and gives least conservative limits often close to undercoverage Feldman-Cousins deliver conservative limits al low bakcground with good coverage though sensitive to the systematics boundaries. In case of zero observation it gives more optimistic upper limits for high background expectation. Bayesian intervals(flat prior) gives conservative limits but the coverage is not guarantied. Doesn't depend on bkg for zero observation. The result are sensitive to the prior and the range of expected signal.

Summary(cont.) Hypothesis testing: Likelihood ratio is working well for single channel and combined models even at low observation and bkg expectations and high systematics(but sym.) Hybrid method is consistent with Likelihood but somewhat more conservative The Z_Bi gives most conservative limits, huge overcoverage for high systematics 3. When combining different (exclusive) channels one has to consider: split search topologies to have least correlations in bkg. Systematics split topologies to have complimentary sensitivity in model parameters space optimize selection (S/B) to have similar evolution with statistics, i.e balance of stat and systematic uncertainties use data driven background estimation when most of uncertainties are in the absolute normalization(rarely the case) otherwise use as a control measurement in g.o.f. If still using, evaluate systematics in tau factor. check consistency among all channels before combining them (eg. CLb for excl.) present results with different methods and, ideally, perform coverage test.