1. Difficulties in Limit setting and the Strong Confidence approach Giovanni Punzi SNS and INFN - Pisa Advanced Statistical Techniques in Particle Physics Durham, 18-22 March 2002

2. Durham 2002 G. Punzi - Strong CL 2 Outline Motivations for a Strong CL Summary of properties of Strong CL Some examples Limits in presence of systematic uncertainties.

3. Durham 2002 G. Punzi - Strong CL 3 Motivation The set of Neyman’s bands is large, and contains all sorts of inferences like: “I bought a lottery ticket. If I win, I will conclude then donkeys can fly @99.9999% CL” I want to get rid of those, but keep being frequentist.

4. Durham 2002 G. Punzi - Strong CL 4 Why should you care ? Wrong reason: to make the CL look more like p(hypothesis | data). Right reason: You don’t want to have to quote a conclusion you know is bad. If you think harder, you can do better: –You are drawing conclusions based on irrelevant facts (like a bad fit). –As a consequence, you are not exploiting at best the information you have –Your results are counter-intuitive and convey little information. You must make sure your conclusions do not depend on irrelevant information

5. Durham 2002 G. Punzi - Strong CL 5 SOLUTION: Impose a form of Likelihood Principle Take any two experiments whose pdf are equal for some subset  of observable values of x, apart for a multiplicative constant. Any valid Confidence Limits you can derive in one experiment from observing x in  must also be valid for the other experiment. If you ask the Limits to be univocally determined, there is no solution.

6. Durham 2002 G. Punzi - Strong CL 6 RESULT Neyman’s CL bands Strong bands Non-coverage land Surprise: a solution exists, and gives for any experiment a well-defined, unique subset of Confidence Bands

7. Durham 2002 G. Punzi - Strong CL 7 Construction of CL bandsRegular Strong Strong

8. Durham 2002 G. Punzi - Strong CL 8 Strong CL vs. standard CL A new parameter emerges: sCL. Every valid band @xx% sCL is also a valid band @xx% CL. You can check sCL for a band built in any other way. sCL requirement effectively amounts to re-applying the usual Neyman’s condition locally on every subsample of possible results.This ensures uniform treatment of all experimental results, but in a frequentist way. Strong Band definition is not an ordering algorithm and answer is still not unique. You may need to add an ordering to obtain a unique solution.

9. Durham 2002 G. Punzi - Strong CL 9 Strong CL It is similar to conditioning, a standard practice in modern frequentist statistics. “There is a long history of attempts to modify frequentist theory by utilizing some form of conditioning. Earlier works are summarized in Kiefer(1977), Berger and Wolpert(1988) […] Kiefer(1977) formally established the conditional confidence approach” “The first point to stress is the unreasonable nature of the unconditional test […] the unconditional test is arguably the worst possible frequentist test […] it is in some sense true that, the more one can condition, the better” “It is sometimes argued that conditioning on non-ancillary statistics will ‘lose information’, but nothing loses as much information as use of unconditional testing” (J. Berger) Neyman: (CR(x) is the accepted region for µ given the observation of x.  is an arbitrary subset of x space)

10. Durham 2002 G. Punzi - Strong CL 10 Summary of sCL properties 100% frequentist, completely general. The only frequentist method complying with Likelihood Principle Invariant for any change of variables No empty regions, in full generality No “unlucky results”, no need for quoting additional information on sensitivity. No pathologies. Robust for small changes of pdf More information gives tighter limits Easier incorporation of systematics Price tag: –Overcoverage –Heavy computation (see CLW proceedings and hep-ex/9912048)

11. Durham 2002 G. Punzi - Strong CL 11 Invariance for change of the observable All classical bands are invariant for change of variable in the parameter (unlike Bayesian limits) The CL definition is invariant for change of variable in the observable, too. But, most rules for constructing bands break this invariance ! Strong-CL is also invariant for any change of variable. Likelihood Ratio is also invariant (non-advertised property?), so it is a natural choice of ordering to select a unique Strong Band.

12. Durham 2002 G. Punzi - Strong CL 12 Effect of changing variables Neyman’s CL bands Strong bands Non-coverage land LR-ordered bands

13. Durham 2002 G. Punzi - Strong CL 13 Poisson+background The upper limit on µ decreases with expected background in all unconditioned approaches. Often criticized on the basis that for n=0 the value of b should be irrelevant. LR-ordering upper limit @90%CL for n=0 background sCL = 90%, or R.-W.

14. Durham 2002 G. Punzi - Strong CL 14 Behavior when new observables are added Do you expect limits to improve when you add extra information ? A simple example shows that neither PO or LRO have this property (conjecture: no ordering algorithm has it !) Example: comparing a signal level with gaussian noise with some fixed thresholds Problem: the limit loosens dramatically when adding an extra threshold measurement.

15. Durham 2002 G. Punzi - Strong CL 15 Example Unknown electrical level µ plus gaussian noise (  =1). Limited to |µ|< 0.5. Compare with a fixed threshold (2.5  ), get a (0,1) response. Observe > threshold: –PO: empty region @90%CL –LR: 0.49 < µ < 0.50 @90%CL –sCL: -0.34 < µ < 0.50 @90%sCL N.B. you MUST overcover unless you want an empty region. L(µ)LR(µ)

16. Durham 2002 G. Punzi - Strong CL 16 Add another threshold Now, add a second independent threshold measurement at 0: limit become much looser ! sCL limit is unaffected Conjecture: no ordering algorithm can provide a sensible answer in all cases. L(µ) LR(µ) 0.27< µ < 0.5

17. Durham 2002 G. Punzi - Strong CL 17 Observations It may be impossible to get sensible results without accepting some overcoverage. Why blame sCL for overcoverage ? Ordering algorithms alone seem unable to prevent very strange results: the inclusion of additional (irrelevant) information may produce a dramatic worsening of limits.

18. Durham 2002 G. Punzi - Strong CL 18 Adding systematics to CL limits Problem: –My pdf p(x|µ) is actually a p(x|µ, ), where is an unknown parameter I don’t care about, but it influences my measurement (nuisance) –I may have some info of coming from another measurement y: q(y| ) –My problem is: p(x,y|µ, ) = p(x|µ, )*q(y| ) Many attempts to get rid of : three main routes: –Integration/smearing (a la Bayes) –Maximization (“profile Likelihood”) –Projection (strictly classical)

19. Durham 2002 G. Punzi - Strong CL 19 Hybrid method: Bayesian Smearing 1) define a new (smeared) pdf: p’(x|µ) =  p(x|µ, )π( ) d where π( ) is obtained through Bayes: –π( ) = q(y| )p( )/q(y) –Need to assume some prior p( ) 2) Use p’ to obtain Conf. Limits as usual GOOD: –Simple and fast –Used in many places –Intuitively appealing BAD: –Intuitively appealing –Interpretation: mix Bayes and Neyman. Output results have neither coverage nor correct Bayesian probability => waste effort of calculating a rigorous CL –May undercover –May exhibit paradoxical tightening of limits

20. Durham 2002 G. Punzi - Strong CL 20 A simple example + Bayes systematics Introduce a systematic uncertainty on the actual position of the 0 threshold. Assume a flat prior in [-1,1]. Do smearing => get tighter limits ! No reason to expect a good behavior µ > 0.272µ > 0.294 LR(µ)

21. Durham 2002 G. Punzi - Strong CL 21 Approximate classical method: Profile Likelihood 1) define a new (profile) pdf: p prof (x|µ) = p(x,y 0 |µ, best (µ)) where best (µ) maximizes the value of a) p(x 0,y 0 |µ, best ) b) p(x,y 0 |µ, best ) ( best = best (µ,x) !)  This means maximizing the likelihood wrt the nuisance parameters, for each µ 2) Use p prof to obtain Conf. Limits as usual GOOD: –Reasonably simple and fast –Approximation of an actual frequentist method BAD: –Flip-flop in case a), non-normalized in case b) !! –Only approximate for low-statistics, which is when you need limits after all. –You don’t know how far off it is unless you explicitly calculate correct limits. –Systematically undercovers

22. Durham 2002 G. Punzi - Strong CL 22 Exact Classical Treatment of Systematics in Limits 1) Use p(x,y|µ, ) = p(x|µ, )*q(y| ), and consider it as p( (x,y) | (µ, ) ) 2) Evaluate CR in (µ, ) from the measurement (x 0,y 0 ) 3) Project on µ space to get rid of uninteresting information on It is clean and conceptually simple. It is well-behaved. No issues like Bayesian integrals Why is it used so rarely ? 1) It produces overcoverage 2) The idea is simple, but computation is heavy. Have to deal with large dimensions 3) Results may strongly depend on ordering algorithm, even more than usual.

23. Durham 2002 G. Punzi - Strong CL 23 “profile method”

24. Durham 2002 G. Punzi - Strong CL 24 “Overcoverage” Projecting on µ effectively widens the CR  overcoverage. BUT: –You chose to ignore information on - cannot ask Neyman to give it all back to you as information on µ - the two things are just not interchangeable.  overcoverage is a natural consequence, not a weakness Q: can you find a smaller µ interval that does not undercover ? (same situation with discretization)

25. Durham 2002 G. Punzi - Strong CL 25 Optimization issue You want to stretch out the CR along direction as far as possible. BUT: –The choice of band is constrained by the need to avoid paradoxes (empty regions, and the like) ! –No method on the marked allows you to treat µ and in a different fashion Strong CL allows you to specify µ as the parameters of interest, and to obtain the narrowest µ interval The solution does not require constructing a multidimensional region

26. Durham 2002 G. Punzi - Strong CL 26 Strong CL Band with systematics The solution does not require explicit construction of a multidimensional region The narrowest µ interval compatible with Strong CL is readily found.

27. Durham 2002 G. Punzi - Strong CL 27 Conclusions Strong Confidence bands have all good properties you may ask for. Systematics can be included naturally and rigorously They can even be actually evaluated

28. Durham 2002 G. Punzi - Strong CL 28 Poisson Example: n=5, b=2, A=0.02±0.006(gaussian) Strong CL Upper limit ~30% higher than from Bayesian calculations shown by Luc Demortier (arbitrary units)