S. Ferrag, G. Steele University of Glasgow

RooStats and MClimit comparison Exercise to use RooStats by an MClimit-formatted person: – Use two programs with the same test statistic and the same format of toy-MC material: signal, background and data plots, s/b, nuisance parameters… – Vary s, b and nuisance parameters and compare the output results on test statistic distribution, sensitivities and Confidence Levels – Not all the Mclimit outputs provided by RooStats  manipulate the input data to get RooStats to produce the same outpu Two approaches in RooStats: – Build WorkSpace, ModelConfig and PDFs à la RooFit – Use of HistFactory

MClimitRooStats with WorkSpace à la RooFit Test Statistic Simple Log Likelihood Ratio (SLLR) With HybridCalculator SLLR Profile Likelihood Ratio Ratio of Maximum Likelihood Other tests Input: HistogramsNormalised to required Luminosity: Signal: variation of Xsec factor Background Data: PDF Normalised to unity using RootHistPdf Signal: variation of Xsec Background: same as s+b with Xsec=0 Data: RooDataSet object Nuisance parameters: Normalisation, Can be asymmetric Varition of above shapes Gaussian constraints Normalisation, can be asymmetric Variation of above shapes Gaussian, Gamma, LogNormal constraints

MC LimitRooStats Number of toy PseudoExperiments (NPE) Same NPE for b only and s+b hypotheses Different NPE between b only and s+b hypotheses Output: Histogram of SLLR test Observed and expected Confidence Levels, CLs, CLb, CLsb Observed test statistic Obs + exp: CLs/  SM CLs/  SM +1,+2,-1,-2 sigma Luminosity Scale factor for 95% exclusion Histogram of SLLR test Observed Confidence Levels, CLs, CLb, CLsb Observed test statistic

3 channels for signal, background and data (1bin, 2bins and 3bins with an empty bin) Format: plots shown Cross sections factors – Signal rate=0.1, or 0.4 – Background rate= 1 S and B rates varied between 0-100% of above values – Expected median and ±1  CLs – Expected median and ±1  test statistics distributions for B and S+B Varying nuisance parameter from 0-50% in MClimit Signal Background 1 bin 2 bins 3 bins

RooStats provides only: Observed Confidence Levels, CLs, CLb, CLsb but not the expected median and ±1  test statistic values nor the corresponding Confidence levels Needed 3 plots: plot for S, plot for B, plot for data If data plot taken as: – B plot  median in test statistics distribution for Null Hypothesis and median expected confidence levels – S+B plot  median in test statistics distribution for Test hypothesis. and median expected confidence levels – Above + renormalized by 1±1  for ±1  values in test statistic distributions and ±1  expected confidence levels –  Useful to draw expected bands with RooStats Data=B Data=S+B Data=(S+B)*(1+1 

Median (black) and ±1  (red, blue) expected CLs RooStats in dashed and Mclimit in plain plots Good agreement of RooStats with Mclimit when Mclimit poisson flag switched off. Mclimit without poisson fluctuation on input S and B plots Mclimit with poisson fluctuation on input S and B plots

Mclimit without poisson fluctuation on input S and B plots Mclimit with poisson fluctuation on input S and B plots Median (black) and ±1  (red, blue) expected CLs RooStats in dashed and Mclimit in plain plots Good agreement of RooStats with Mclimit when Mclimit poisson flag switched off.

Mclimit with poisson fluctuation on input S and B plots Mclimit without poisson fluctuation on input S and B plots Buggy results They were agreeing this morning !

3 bin plots with 1 empty (non empty for signal) bin for background plot RooStats and Poisson flag off MClimit crashed for low S rate values Poisson flag on MClimit worked but non usual

Nuisance Parameter Value (%)

We didn’t see equivalence of poisson flag in RooStats 1 st satisfactory agreement between RooStats and Mclimits without Poisson fluctions on input plots Detailed study with nuisance parameters