Distinguishability of Hypotheses S.Bityukov (IHEP,Protvino; INR RAS, Moscow) N.Krasnikov (INR RAS, Moscow ) December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov Institute for High Energy Physics, Protvino, Russia Institute for Nuclear Research RAS, Moscow, Russia

Planned experiment will give information about : (a) existence of new phenomenon (yes or no), (b) magnitude and accuracy of measured value. We consider the case (a) from the frequentist point of view, namely, we will calculate the total amount of possible cases and, after that, the amount of cases in favor of one of the statements. This approach allows to estimate the probability of making a correct decision and, correspondingly, the quality of planned experiment by using the hypotheses testing. Also we can introduce the conception of the distinguishability of hypotheses. Concept Concept December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov Distinguishability of hypotheses: What is it ? Let us consider the planned experiment for searching for new phenomenon.

Hypotheses testing Hypotheses testing Hypothesis Hypothesis H0: new physics is present in Nature : new physics is absentHypothesis H1  = P(reject H0 | H0 is true) -- Type I error  = P(accept H0 | H0 is false) -- Type II error  is a significance of the test 1-  is a power of the test Note that  and  can be considered as random variable. If H0 is true then  takes place and  is absent. If H1 is true then  is absent and  takes place. Many researchers prefer to exchange the places of these hypotheses December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

What to test ? What to test ? Suppose the probability of the realization of n events in the experiment is described by function f(n;  ) with parameter  Expected number of signal events in experiment is  s Expected number of background events is  b Hypothesis H0 corresponds to  =  s +  b, i.e. f(n;  s +  b ) H1 corresponds to  =  b, i.e. f(n;  b ) The Type I error  and the Type II error  allows to estimate the probability of making a correct decision when testing H0 versus H1 with an equal-tailed test and to estimate the distinguishability of these hypotheses H0 and H1 with an equal probability test. December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

The probability of making a correct decision in hypotheses testing The probability of making a correct decision in hypotheses testing where the estimator Let us consider the random variable  =  +  the case of applying the equal-tailed test December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov is a constant term and is a stochastic term. are the estimators of Type I (  ) and Type II (  ) errors. In the stochastic term is equal to 0 independently of whether H0 or H1 is true. Hence the estimator can be named the probability of making incorrect choice in favor of one of the hypotheses. Correspondingly,is the probability to make a correct decision in hypotheses testing. Here

: advantages and disadvantages : advantages and disadvantages Advantages +1° This probability is independent of whether H0 or H1 is true +2° In the case of discrete distributions the error  of this estimator can be taken into account +3° This is an estimator of quality of planned experiment … However, the probability of making a correct decision has disadvantages to be the measure of distinguishability of hypotheses. Disadvantages -1° Non minimal estimation of possible error in hypotheses testing -2° The region of determination is [0, 0.5] (desirable area [0, 1]) -3° Difficulties in applying of equal-tailed test for complex distributions … Goal: disadvantages ---> advantages December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Distinguishability of hypotheses Distinguishability of hypotheses -1° --> + 4° The applying of the equal probability test gives the minimal half-sum of estimators of Type I error  and Type II  by the definition of this test. The critical value n 0 is chosen by the condition f(n 0 ;  b ) = f(n 0 ;  s +  b ). December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

The relative number of incorrect decisions under equal probability test The relative number of incorrect decisions under equal probability test December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov It is a relative number of incorrect decisions. The region of determination is [0, 1]. -3° --> + 6° The applicability of equal probability test to the complex distributions is obviously. -2° --> + 5° The transformation makes a good candidate to be a measure of distinguishability

The equal probability test : the critical value for Poisson distribution The equal probability test : the critical value for Poisson distribution The applying of the equal probability test for Poisson distribution gives the critical value December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov where square brackets mean the the integer part of a number. Note that the critical value conserves the liner dependence on the time of measurements

The estimation of the hypotheses distinguishability The estimation of the hypotheses distinguishability The relative number of correct decisions under equal probability test is a measure of distinguishability of hypotheses at given  s and  b. The magnitude of this value can be found from equations December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Dependence of relative number correct decisions on the measurement time December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Gamma- and Poisson distributions Gamma- and Poisson distributions Let us consider the Gamma-distributions with probability density December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov If to redefine the parameters and variable 1/ ,  and x via a, n+1, , correspondingly, the probability density will be If a = 1 then the probability density looks like Poisson probabilities

Conditional probability density Conditional probability density The probability density g(  ;N) of the true value of the parameter of Poisson distribution to be  in the case of the single observation N has Gamma distribution, where It follows from identity We checked the statement about g(  ;N) by the Monte Carlo and these calculations give the confirmation of given supposition. As a result we can take into account the statistical uncertainties by the simple way (see, NIM A502(2003)795; JHEP09(2002)060 ). December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov

Statistical uncertainties in determination of  s and  b Statistical uncertainties in determination of  s and  b December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov Suppose that the expected number of signal events  s and expected background  b are determined with statistical errors. If we can reconstruct the distribution g(  ; N ) of the true value of the parameter  of the function f(n;  ) in the case of the observed N events then we can determine the relative number of correct decisions:

Conclusions Conclusions The probability language in terms of  and  is more acceptable for estimation of the quality of planned experiments than the language of standard deviations. The proposed approach allows to use the relative number of correct decisions in the equal probability test as a measure of distinguishability of hypotheses. This approach gives the easy way for including of the systematics and statistical uncertainties into the estimation of distinguishability of hypotheses. December 1, 2003 ACAT’2003 KEK, Japan S.Bityukov