Tests Jean-Yves Le Boudec

Contents 1.The Neyman Pearson framework 2.Likelihood Ratio Tests 3.ANOVA 4.Asymptotic Results 5.Other Tests 1

Tests Tests are used to give a binary answer to hypotheses of a statistical nature Ex: is A better than B? Ex: does this data come from a normal distribution ? Ex: does factor n influence the result ? 2

Example: Non Paired Data Is red better than blue ? For data set (a) answer is clear (by inspection of confidence interval) no test required 3

Is this data normal ? 4

5.1 The Neyman-Pearson Framework 5

Example: Non Paired Data 6  Is red better than blue ?

Critical Region, Size and Power 7

Example : Paired Data 8

Power 9

10 Grey Zone

11

p-value of a test 12

13

Tests are just tests 14

Test versus Confidence Intervals If you can have a confidence interval, use it instead of a test 15

2. Likelihood Ratio Test A special case of Neyman-Pearson A Systematic Method to define tests, of general applicability 16

17

18

A Classical Test: Student Test The model : The hypotheses : 19

20

21

Here it is the same as a Conf. Interval 22

The “Simple Goodness of Fit” Test Model Hypotheses 23

1. compute likelihood ratio statistic 24

2. compute p-value 25

Mendel’s Peas P= 0.92 ± 0.05 => Accept H 0 26

3 ANOVA Often used as “Magic Tool” Important to understand the underlying assumptions Model Data comes from iid normal sample with unknown means and same variance Hypotheses 27

28

29

The ANOVA Theorem We build a likelihood ratio statistic test The assumption that data is normal and variance is the same allows an explicit computation it becomes a least square problem = a geometrical problem we need to compute orthogonal projections on M and M 0 30

The ANOVA Theorem 31

Geometrical Interpretation Accept H 0 if SS2 is small The theorem tells us what “small” means in a statistical sense 32

33

ANOVA Output: Network Monitoring 34

The Fisher-F distribution 35

36

Compare Test to Confidence Intervals For non paired data, we cannot simply compute the difference However CI is sufficient for parameter set 1 Tests disambiguate parameter sets 2 and 3 37

Test the assumptions of the test… Need to test the assumptions Normal In each group: qqplot… Same variance 38

39

4 Asymptotic Results 40 2 x Likelihood ratio statistic

41

The chi-square distribution 42

Asymptotic Result Applicable when central limit theorem holds If applicable, radically simple Compute likelihood ratio statistic Inspect and find the order p (nb of dimensions that H1 adds to H0) This is equivalent to 2 optimization subproblems lrs = = max likelihood under H1 - max likelihood under H0 The p-value is 43

Composite Goodness of Fit Test We want to test the hypothesis that an iid sample has a distribution that comes from a given parametric family 44

Apply the Generic Method Compute likelihood ratio statistic Compute p-value Either use MC or the large n asymptotic 45

46

Is it normal ? 47

48

49

Mendel’s Peas P= 0.92 ± 0.05 => Accept H 0 50

Test of Independence Model Hypotheses 51

Apply the generic method 52

53

5 Other Tests Simple Goodness of Fit Model: iid data Hypotheses: H 0 common distrib has cdf F() H 1 common distrib is anything Kolmogorov-Smirnov: under H 0, the distribution of is independent of F() 54

55

Anderson-Darling An alternative to K-S, less sensitive to “outliers” 56

57

58

Jarque Bera test of normality (Chapter 4) Based on Kurtosis and Skewness Should be 0 for normal distribution 59

60

Robust Tests Median Test Model : iid sample Hypotheses 61

Median Test 62

Wilcoxon Signed Rank Test 63

Wilcoxon Rank Sum Test Model: X i and Y j independent samples, each is iid Hypotheses: H 0 both have same distribution H 1 the distributions differ by a location shift 64

Wilcoxon Rank Sum Test 65

Turning Point 66

Questions What is the critical region of a test ? What is a type 1 error ? Type 2 ? The size of a test ? What is the p-value of a test ? 67

Questions What are the hypotheses for ANOVA ? How do you compute a p-value by Monte Carlo simulation ? A Monte Carlo simulation returns p = 0; what can we conclude ? What is a likelihood ratio statistic test ? What can we say about its p-value ? 68

We have data X_1,…,X_m and Y_1, …,Y_m. Explain how we can compute the p-value of a test that compares the variance of the two samples ? We have a collection of random variables X[i,j] that corresponds to the result of the ith simulation when the machine uses configuration j. How can you test whether the configuration plays a role or not ? 69