1. Results of combination Higgs toy combination, within and across experiments, with RooStats
Grégory Schott
Institute for Experimental Nuclear Physics
of the Karlsruhe Institute of Technology
on behalf of the ATLAS-CMS Higgs groups and statistics forum
ATLAS-CMS statistics meeting, 5th July 2010

2. Introduction
I will present the results of ATLAS, CMS and the combination
summary of the results obtained by Kyle Cranmer and myself
thanks to Kyle for the FC, PL and some Hybrid results
in case of PL we both get results that of course agree very well
This is on-going work; while a large part of the results are available, not everything has been done yet and not everything has been validated yet
Toy study (made with mockup numbers), H→WW at 160 GeV with 1fb-1
don't draw physics conclusion of the results
Significance and upper limit computed with RooStats (for a given data)
different approaches tested as recommanded by the statistics forums
Validation of RooStats (ROOT , June 2009)
Conclusion and discussions
Grégory Schott - ATLAS-CMS statistics meeting

3. The combined model Combination of H→WW→ee, mm, em analyses
Parameter of interest: ratio of cross-sections, r = m = s / sSM
CMS analysis:
3 observables, 37 nuisance parameters
Lognormal distribution of nuisance parameters
Uncertainty on control region measurement in the systematics
ATLAS analysis:
9 observables, 12 nuisance parameters
Truncated gaussian distribution of nuisance parameters
Control regions measurements included in the analysis
Combined analysis:
common parameter interest (thus, 100% correlated)
no other correlation (yet) across the experiments
Concretely, all the information for the combination is shared via 2 ROOT files (for ATLAS & CMS) containing RooFit/RooStats workspaces
graphical representation of the objects in the combined model
Grégory Schott - ATLAS-CMS statistics meeting

4. Toy-data Data used for significance:
generated from the model in the hypothesis that signal is present
Data used for exclusion:
generated from the model in the background-only hypothesis
Check the backups if you're interested by the number of observed events assumed (or the likelihood functions)
Grégory Schott - ATLAS-CMS statistics meeting

5. Profiled likelihood results
significance:
reasonably good estimate if large enough
95% CL UL taken for -log l(r) = 1.921
assumption of Wilks's asymptotia
that's not always valid!
with systematics r
signif.
ATLAS
1.13
2.70
CMS
0.98
4.89
COMBI
1.02
5.58 ^
with systematics r UL
ATLAS
0.79
CMS
0.28
COMBI
0.25 ^
Grégory Schott - ATLAS-CMS statistics meeting

6. Hybrid Frequentist-Bayesian
B-toys
CMS, 477k toys
SB-toys
QLEP
QTEV
l(m)
data
test statistics
significance (no syst.)
significance (with syst.)
ATLAS
QLEP
QTEV
l(m)
3.78 -
3.07 ± 0.01
2.8 ± 0.1
CMS
6.22 ± 0.02
4.77 ± 0.02
> 4.6
4.3 ± 0.1
COMBI
> 3.5
run for the 3 test statistics mentionned in the previous talk
need to study the advantage and inconvenient of the 3 approaches
Grégory Schott - ATLAS-CMS statistics meeting

7. Hybrid test statistics distributions
computing the p-value for significance in this approach is challenging:
speed improvements would be useful
or use importance sampling techniques
CMS distribution (and results previous slide) made with a RooFit-independent tool
QLEP
QLEP
CMS model with 'LandS'
Grégory Schott - ATLAS-CMS statistics meeting

8. Summary of upper limits
with BAT [ Caldwell,
Kollar, Kroeninger,
Comp.Phys.Com.
180, 2197 (2009) ]
CMS
ATLAS
95% CL UL
posterior (r | data) r
95% CL upper limits: results with systematics (except if indicated otherwise)
technique
test stat
rule
sampling
UL ATLAS
UL CMS
UL COMBI
Feldman-Cousins (no syst.)
l(m)
CLS+B
toys
0.69 ± 0.05 -
Profile LR (Wilks)
asymptotic
0.79
0.28
0.25
Feldman-Cousins++
0.78 ± 0.05
0.26 ± 0.02
0.23 ± 0.02
Hybrid
QLEP
CLS
~ 0.68
0.29 ± 0.03 (LandS)
~ 0.61
Bayesian
n/a, flat prior on r
MCMC*
0.72
0.31
Grégory Schott - ATLAS-CMS statistics meeting

9. Validations
Some validation of the analysis performed with independent software (LandS, M. Chen, CMS)
Since 'LandS' was developped specifically for this analysis, it is faster than RooStats which is a general tool showing that RooStats still need to improve
table of compared results: we still need to work on the missing parts
PL and toy-MC results using an earlier version of the ATLAS model were validated with an independent code (Hao Liu, et. al.)
further on-going work on validating Bayesian limits: S. Schmitz (CMS)
currently using BAT, RooStats providing another MCMCCalculator
Grégory Schott - ATLAS-CMS statistics meeting

10. Conclusion and outlook
A good start, work is on-going on
Lessons learned:
we achieved in a short time scale to combine the Atlas and CMS analyses -> proof of principle that we can do that
proof of principle we can compare different statistical methods to one another
still some weaknesses identified that RooStats need to improve
we also need to continue the validations of RooStats
still need to understand those results
discussion has started on statistical methods we want to use in common
proper treatment of correlations across experiments will require planning and thoughtful parametrization of systematics
activity will continue (other masses, other channels, more thorough correlation studies, ...)
Grégory Schott - ATLAS-CMS statistics meeting

11. End of talk − Backup slides
Grégory Schott - ATLAS-CMS statistics meeting

12. Toy-data Data used for significance:
generated from the model in the hypothesis that signal is present
Data used for exclusion:
generated from the model in the background-only hypothesis
Observables:
1) nobs_bin1 = 15
2) nobs_bin2 = 7
3) nobs_bin3 = 13
4) obs_s_em_0j = 36
5) obs_ww_em_0j = 52
6) obs_tt_em_0j = 3650
7) obs_wj_em_0j = 143
8) obs_s_ee_0j = 9
9) obs_ww_ee_0j = 5
10) obs_wj_ee_0j = 49
11) obs_s_mm_0j = 18
12) obs_ww_mm_0j = 28
Observables:
1) nobs_bin1 = 4
2) nobs_bin2 = 2
3) nobs_bin3 = 3
4) obs_s_em_0j = 19
5) obs_ww_em_0j = 63
6) obs_tt_em_0j = 3814
7) obs_wj_em_0j = 123
8) obs_s_ee_0j = 6
9) obs_ww_ee_0j = 14
10) obs_wj_ee_0j = 53
11) obs_s_mm_0j = 16
12) obs_ww_mm_0j = 26
Grégory Schott - ATLAS-CMS statistics meeting

13. ATLAS likelihood model
slide from K. Cranmer,
Grégory Schott - ATLAS-CMS statistics meeting

14. CMS likelihood model slide from A. Korytov, 24.06.2010
Grégory Schott - ATLAS-CMS statistics meeting

15. Profile likelihood (zoom)
Grégory Schott - ATLAS-CMS statistics meeting

16. Bayesian upper limits posterior (r | data) ATLAS posterior (r | data)
COMBI r r
Grégory Schott - ATLAS-CMS statistics meeting