1. 1 Dec 2011COMP80131-SEEDSM81 Scientific Methods 1 Barry & Goran ‘Scientific evaluation, experimental design & statistical methods’ COMP80131 Lecture 8: Statistical Methods-Significance tests & confidence limits www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131

2. 1 Dec 2011COMP80131-SEEDSM82 Introduction Statistical significance testing has so far been applied on the assumption of a (1) discrete population with binomial distribution (2) continuous population with known normal pdf & known std. Before proceeding further, take a quick look at a few more prob distributions & pdfs. Significance testing can be adapted to any of these.

3. 1 Dec 2011COMP80131-SEEDSM83 Exponential pdf Lifetimes e.g. of light bulbs follow an exponential distribution: mean = 2; x = 0:0.1:10; y = exppdf(x,mean); plot(x,y); Mean =  Std =  also

4. 1 Dec 2011COMP80131-SEEDSM84 Poisson Distribution λ, is both mean & variance of the distribution. Poisson & exponential distributions are related. If number of counts follows a Poisson distribution, then interval between individual counts follows exponential distribution. As λ gets larger, Poisson pdf  normal with µ = λ, σ 2 = λ. For applications that involve counting number of times a random event occurs in a given amount of time, e.g. number of people walking into a store in an hour.

5. 1 Dec 2011COMP80131-SEEDSM85 Poisson distributions in MATLAB x=0:16 y = poisspdf(x,5); stem(x,y); x=0:60 y = poisspdf(x,20); stem(x,y);

6. 1 Dec 2011COMP80131-SEEDSM86 Chi-squared distribution Given V indep normally distrib random variables, X 1, X 2, …, X V all with mean = 0 & std =1, let  2(V) = X 1 2 + X 2 2 + … + X V 2 Then the pdf of samples x of  2 is: ‘Gamma function’  (x) is a generalisation of x! to non-integers. This pdf will tell us how about variance of a population. If s=std of samples of V observations of normally distributed pop with std σ: Vs 2 /  2   2 (V)

7. 1 Dec 2011COMP80131-SEEDSM87 Plot chi2 pdf with V = 4 x = 0:0.2:15; y = chi2pdf(x,4); plot(x,y)

8. 1 Dec 2011COMP80131-SEEDSM88 Student’s t-distribution pdf Depends on a single parameter V (degrees of freedom). As V , t-pdf approaches standard normal distribution If x is a random sample of size n from a normal distribution with mean μ, then the t-statistic has Student's t-pdf with V = n – 1 degrees of freedom.

9. 1 Dec 2011COMP80131-SEEDSM89 Compare t-pdf(V=5) with normal x = -5:0.1:5; y = tpdf(x,5); z = normpdf(x,0,1); plot(x,y,'b',x,z,'r');

10. 1 Dec 2011COMP80131-SEEDSM810 MATLAB functions for t-dist pdf for t-distribution with V degrees of freedom: y = tpdf ( t,V); (With samples with n values, V = n-1). Cumulative df with V degrees of freedom p = tcdf ( t, V) Prob of rand var being  t Complementary df (area under ‘tail’ from t to  ) p = 1 – tcdf ( t, V) Prob of rand var being > t

11. 1 Dec 2011COMP80131-SEEDSM811 Inverse-cdf in MATLAB Inverse of cumulative distrib function: If p=tcdf(t,V) then t = tinv(p,V) Value of t such that prob of rand var being  t is p If p = normcdf(z,m,  ) then z = norminv(p,m,  ) Value of z such that prob of rand var being  z is p Complementary version: t = tinv(1-p,V) Value of t such that prob of rand var being > t is p. Similarly for complementary version of norminv

12. 1 Dec 2011COMP80131-SEEDSM812 Significance testing: z-test Assume Normal population with known stddev = . Null hypothesis: pop-mean =  0 Alternative hyp: pop-mean <  0 Take one sample of n values & calculate the z-statistic: If pop-mean =  0, dist of z will be standard Normal (mean=0, std=1) -2 0 1 2 4 0 0.1 0.2 0.3 0.4 Std Normal pdf z If mean of z is 0, how likely is a value  z as just calculated? p-value = prob (x  z) = 1-normcdf(z,0,1) If p-value < significance level alpha (  ) reject null hyp.

13. 1 Dec 2011COMP80131-SEEDSM813 Alternative formulation Assuming we need 95% confidence,  = 0.05 Let z(  ) = norminv(1- ,0,1) = 1.65 Prob of getting rand var  1.65 is less than 0.05 If z  1.65, it is outside our 95% ‘confidence limit’ that the null hyp may be true. So reject null hyp. Confidence limit is for z is -  to 1.65 Neglect possibility that z may be negative.(1-tailed test) Confidence limit for sample-mean is -  to 1.65  /  n +  0

14. 1 Dec 2011COMP80131-SEEDSM814 2-tailed test Assuming we need 95% confidence,  = 0.05 Allowing possibility that z z(  /2)) and for z < -z(  /2)). prob(z  z(  /2)) + prob (z  -z((  /2) ) = 2 prob(z  z(  /2)) =  Now, z(  /2) = norminv(1-  /2,0,1) = 1.96 Prob of getting rand var  1.96 or  -1.96 is 0.05 If z > 1.96 or z < - 1.96, it is outside our 95% ‘confidence limit’ that the null hyp may be true. So reject null hyp. Confidence limit is for z is -1.96 to 1.96 Confidence limits for sample-mean is  0 - 1.96  /  n to  0 + 1.96  /  n

15. 1 Dec 2011COMP80131-SEEDSM815 Significance testing: t-test Assume Normal population with unknown stddev. Null hypothesis: pop-mean =  0 Alternative hyp: pop-mean <  0 Take one sample of n values & calculate the t-statistic: If pop-mean =  0, dist of t will be standard t-pdf (blue) with V=n-1. How likely is calculated value of t? ‘1-tailed’ p-value = prob (x  t) = 1 - tcdf(t, n-1) If p-value < significance level alpha (  ) reject null hyp. t -5-4-3-2012345 0 0.1 0.2 0.3 0.4 T-pdf(blue) Norm-pdf(red)

16. 1 Dec 2011COMP80131-SEEDSM816 Alternative formulation (2-tailed) Assuming we need 95% confidence,  = 0.05 Confidence limits for  0 is: Null Hyp is that pop-mean is  0 If value of  0 is outside these limits, reject the null hyp that population mean is  0 Can say with 95% confidence that pop-mean >  0 or <  0 If  0 is within these confidence limits, cannot reject null-hyp.

17. 1 Dec 2011COMP80131-SEEDSM817 Difference betw z-test & t-test(2-tailed) With z-test pop-std (  ) is known; with t-test  is unknown. For z-test, p-value = prob (  x   z) = 1- normcdf(z,0,1) For t-test, p-value = prob(  x   t) = 1 – tcdf(t,n-1) Same Null-hyp: pop-mean =  0 : reject if  0 outside conf limits Confidence limits for z-test: Confidence limits for t-test:

18. 1 Dec 2011COMP80131-SEEDSM818 Non-Gaussian populations If samples of size n are ‘randomly’ chosen from a pop with mean  & std , the pdf of their mean, m 1 say, approaches a Normal (Gaussian) pdf with mean  & std  /  n as n is made larger & larger. Regardless of whether the population is Gaussian or not! This is Central Limit Theorem Tests can be made to work for non-Gaussian populations provided n is ‘large enough’.

19. 1 Dec 2011COMP80131-SEEDSM819 Barry’s Assignment Deadline 20 Dec 2011 Email to barry@man.ac.uk with ‘SEEDSM’ in titlebarry@man.ac.uk or Hand in paper copy to SSO Exam statistics are in examdata.dat and examdata.xls in www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131 (or navigate from www.cs.man.ac.uk/~barry)

20. 1 Dec 2011COMP80131-SEEDSM820 Question 1 What are the essential differences between Baysian and ‘frequentist’ statistics?

21. 1 Dec 2011COMP80131-SEEDSM821 Question 2: fair coin test Suppose we obtain heads 15 times out of 20 flips of a coin. By establishing confidence limits, state whether it is it likely to be a fair coin?

22. 1 Dec 2011COMP80131-SEEDSM822 Question 3: Exam statistics Analyse the ficticious exam results & comment on features. Compute means, stds & vars for each subject & histograms for the distributions. Make observations about performance in each subject & overall Do marks support the hypothesis that people good at Music are also good at Maths? Do they support the hypothesis that people good at English are also good at French? Do they support the hypothesis that people good at Art are also good at Maths? If you have access to only 50 rows of this data, investigate the same hypotheses What conclusions could you draw, and with what degree of certainty?

23. 1 Dec 2011COMP80131-SEEDSM823 Question 4: Bayes Theorem (a) A patent goes to a doctor with a bad cough & a fever. The doctor needs to decide whether he has ‘swine flu’. Let statement S = ‘has bad cough and fever’ & statement F = ‘has swine flu’. The doctor consults his medical books and finds that about 40% of patients with swine-flu have these same symptoms. Assuming that, currently, about 1% of the population is suffering from swine-flu and that currently about 5% have bad cough and fever (due to many possible causes including swine-flu), we can apply Bayes theorem to estimate the probability of this particular patient having swine-flu. (b)A doctor in another country knows form his text-books that for 40% of patients with swine-flu, the statement S, ‘has bad cough and fever’ is true. He sees many patients and comes to believe that the probability that a patient with ‘bad cough and fever’ actually has swine-flu is about 0.1 or 10%. If there were reason to believe that, currently, about 1% of the population have a bad cough and fever, what percentage of the population is likely to be suffering from swine-flu?