1. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 1 How To Conduct Good Experiments? Ernesto Costa DEI/CISUC ernesto@dei.uc.pt http://www.dei.uc.pt/~ernesto

2. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 2 Summary What is the goal of this talk? Background Probabilities Random Variables and Probability distributions Inferential Statistics Applying the Theory Conclusions

3. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 3 What is the goal of this talk? I don’t know! I have been asked to give a talk on that subject… I do know! EC is (much) an experimental discipline Most of our work is to compare things Algorithms Parameters settings What is a fair comparison?

4. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 4 What is the goal of this talk? Looking for EC papers One problem One run Several runs 10, 20, 30? Use average values Use average of the bests Use the mean Use the mean and the standard deviation Use Confidence Levels / Intervals

5. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 5 What is a good experiment? Identify independent and dependent variables Mutation rate  fitness Different crossover operators  fitness Evolution and Learning  # of survivors Identify the conditions of the experiment Initial conditions Number of runs Parameters Settings Identify the kind of Statistics you will need Descriptive Inferential Non parametric What is the goal of this talk?

6. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 6 Background Probabilities Experiment: procedure whose variable result cannot be predicted ahead of time. Tossing a coin, rolling a dice Sample Space: set of possible outcomes of an experiment. {Heads, Tails} {1,2,3,4,5,6} Event: subset of the sample space {Heads} {1,3,5}

7. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 7 Background Probability of an Event Measure the likelihod that the event will occur Tossing a (fair) coin: probability(outcome=heads) =1/2 Axioms P(E)  0 P(S)=1 For mutually exclusive events Probabilities

8. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 8 1/6 Experiment Prob. Assign.Sample SpaceEvent Example What is the probability of when rolling two dice the sum of the two outcomes equal 7? Working Methodology Background Probabilities

9. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 9 Definition: Let E and F be two events, with p(F)>0. The conditional probability of E given F, p(E|F), is defined as: Probabilities Example: A family has two children. Knowing that one is a boy what is the probability that they have two boys? 1/3

10. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 10 Probabilities Theorem of Bayes: Example: A building has two lifts. One is used by 45% of the residents And the other by 55%. The first one, 5% of the time have problems, while The second 8% of the time can let you in trouble. Knowing that one lift had a problem, what is the probability of being lift number 1? 33,8%

11. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 11 Random Variables Definition: A random variable, X, is a function from the sample space of an experiment to the set of real numbers. X(s) s S X 0123 SXSX A RV is a function … and is not random!!! Random Variables and Probability Distributions

12. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 12 Experiment Prob. Assign.Sample SpaceEvent Random VariableProb. Distribution Working Methodology Random Variables and Probability Distributions Toss coin (3x)f(x i )=p(X=x i )8 possibilities# Heads X(HHT)=2X  f(x i ) Experiment Sample Space Random Variable Event Prob. Distrib. Example Prob. Assign.

13. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 13 Example: Suppose you toss a coin three times. Let X(t) denote the number of heads that appear when t is the result. Então X(t): X(HHH) = 3 X(HHT) = X(HTH) = X(THH) = 2 X(TTH) = X(THT) = X(HTT) = 1 X(TTT) = 0 Random Variables and Probability Distributions Probabilty Distribution

14. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 14 Random Variables and Probability Distributions Discrete Probability Mass Function Continuous Probability Density Function (pdf) Types of Random Variables x f(x) 0 x1x1 x2x2

15. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 15 Location Mean Dispersion Variance Standard Deviation Random Variables and Probability Distributions Measures of Random Variables

16. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 16 Two random Variables, X and Y, over the same sample space S, are said to be independent iff: Theorem of the Product Theorem of Sum Random Variables and Probability Distributions Independence of Random Variables

17. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 17 Random Variables and Probability Distributions Binomial Distribution Domain {0,1,2,…n} Probability mass function Mean np  Variance npq  Discrete Probability Distributions P=0.3P=0.5

18. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 18 Poisson Distribution Approach the Binomial Distribution Domain {0,1,2,3,...} Probability mass function Mean: Variance: Random Variables and Probability Distributions Discrete Probability Distributions =np =6 =8,4

19. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 19 Normal (Gaussian) Distribution Standard Normal Distribution Random Variables and Probability Distributions Continuous Probability Distributions

20. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 20 Converting a normal distribution to a standard normal distribution X a random Variable with Mean  Standard Deviation σ Using a translation Defining a new Random variable Random Variables and Probability Distributions Continuous Probability Distributions

21. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 21 N(0,1)  =1  =5  =10 Student’s t-Distribution Approximates the standard normal distribution N(0,1) Degrees of freedom (df),  Mean 0,  Variance  Random Variables and Probability Distributions Continuous Probability Distributions

22. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 22 Goal: to apply probability theory to data analysis How? Model the data (population) by mean of a probability distribution Use a sample of the data instead of the all population Estimate the population parameters ( , σ, p) using correspondent sample statistics (x, s, ) Statistics Background population sample  σ p x s parameters statistics

23. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 23 Unbiased estimator A statistics with mean value equal to the population parameter being estimated Point Estimators Interval Estimators Background Statistics

24. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 24 Background Consider a population with mean  and standard deviation σ. Let denote the mean of the observations in random samples of size n. Then: When the population distribution is normal, the sampling distribution of is also normal for any sample size n (Central Limit Theorem) When n is sufficient large (n>30) the sampling distribution is well aproximated by a normal curve, even if the population distribution is not itself normal Sample distribution of the sample mean and the Central Limit Theorem

25. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 25 Background Unbiased estimators Mean Standard Deviation Sample distribution of the sample mean (n-1) are the degrees of freedom (df)

26. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 26 Background Consequence For a large sample or population whose distribution is normal: has (approximately) a standard normal (Z) distribution. Sample distribution of the sample mean and the Central Limit Theorem

27. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 27 Background Estimate the mean  The population standard deviation, σ, is known; The sample mean from a random sample, is known, The sample size is large (>30) The one sample Z confidence interval is Example: for an 95% confidence interval Z=1.96. Confidence Intervals – one sample

28. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 28 Background Example: we want a confidence level of 90% Look into a N(0,1) For a CL of 90%, we have to isolate the area of 5% to the left and to the right of the bell shaped normal distribution. The confidence interval will be given by Looking in a table for the value of Z we obtain Z=1.65 Confidence Intervals – one sample

29. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 29 Background What does it means having a confidence interval of 95%? That there is a probability of 95% that the true mean (population) is in the interval? NO!! Mean that 95% of all possible samples result in an interval that includes the true mean! Confidence Intervals – one sample

30. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 30 Background Estimate the mean  The population standard deviation, is NOT known; The sample mean from a random sample, is known, The sample size is large (>30) OR the population distribution is normal The one sample t confidence interval is where the t critical value is based on (n-1) degrees of freedom (df). Example: for an 95% confidence interval and 19 df t=2.09. The Student T Distribution can be used for small samples assuming that the population distribution is approximately normal Confidence Intervals – one sample

31. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 31 Background A hypothesis is a claim about the value of one or more population characteristics. A test procedure is a method for using sample data to decide between to competing claims about population characteristics. (  = 100 or   100) Method by contradiction: we assume a particular hypothesis. Using the sample data we try to find out if there is convincing evidence to reject this hypothesis in favor of a competing one Hypothesis Testing – one sample

32. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 32 Background The null hypothesis, H 0, is a claim about a population characteristic that is initially assumed to be true. H a is the alternative hypothesis or competing claim. Testing H 0 versus H a can lead to the conclusion the H 0 must be rejected or we fail to reject H 0. I that last case we cannot say that H 0 is accepted! Hypothesis Testing – one sample

33. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 33 Background Errors Type I error Rejecting H 0 when H 0 is true The probability of a type I error, , is called Level of Significance of the test. Type II error Failing to reject H 0 when H 0 is false The probability of a Type II error is denoted by . There is a tradeoff between  and  : making type I error very small increase the probability of type II error. Hypothesis Testing – one sample

34. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 34 Background Test Statistic (Z,t): function of the sample data on which a decision about reject or fail to reject H 0 is based; p-value (observed significance level): is the probability, assuming that H 0 is true, of obtaining a test statistics at least as inconsistent with H 0 as what actually resulted. Decision about H 0 : comparing the p-value with the chosen . Reject H 0 if p-value   Hypothesis Testing – one sample

35. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 35 Background Hypothesis Testing – principles What is the population parameter (mean,…) State the H 0 and H a Define the significance level  The assumptions for the test are reasonable (big sample,…) Calculate the test statistic (Z,…) Calculate the associated p-value State the conclusion (reject if p-value  ,…) Hypothesis Testing – one sample

36. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 36 Background Example Population parameter the mean,  H 0 :  =100, H a :   100 Significance level  =0.01 n=40 is large From the sample: =105,3, σ=8.4 From the z-curve we know that the p-value  0 Therefore the null hypothesis, H 0, is rejected with a significance level of 0.01. Hypothesis Testing – one sample

37. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 37 Background Use the sample distribution of the difference of the sample means: Properties The mean of the difference is equal to the difference of the means The variance of the difference is equal to the sum of the individuals variances. Thus, the standard deviation: The sampling distribution of the difference of the sample means, can be considered approximately normal (each n large, each sample mean come from a population (approximately) normal Comparing Two Populations based on independent samples

38. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 38 Background Assumptions The two samples are independently random samples Sample sizes are both large (n >30) OR the population distributions are (approximately) normal. Formulas Confidence interval for the mean of

39. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 39 Background Same procedure, only the formulas are different! Z Test Large samples OR Population distributions are (at least approximately) normal Hypothesis Test

40. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 40 Background t test Large samples OR Population distributions normal AND the random samples are independent Hypothesis Test

41. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 41 Applying the Theory Two algorithms A standard GA A standard GA + local learning (Baldwin Effect) Goal: good quality machines Who is better? Comparing the means! H 0 :  1 =  2 (no improvement!!!), H a :  1 ≠  2 Confidence level,  =0.01 Assuming that the population distributions are normal Number of (independent) runs = 30 for each case Use t test The Busy Beaver Problem

42. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 42 From the samples (# good machines)  sga =0.1  be =0.23 S ga 2 =0.093 S be 2 =0.185 From the formulas df=53 t=1.35 p-value  2*0.1=0.2 Conclusion With  =0.01and p-value =0.2, the null hypothesis H 0 cannot be rejected The Busy Beaver Problem Applying the Theory

43. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 43 Applying the Theory Two different GAs applied to function optimization A standard GA using a 2 point CXover A modified GA using transformation Goal: find the minimum Function Optimization The Schwefel Function Minimum = 0

44. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 44 Who is better? Two point Crossover or Transformation? Comparing the means of the best fit! H 0 :  1 =  2 (no improvement!!!), H a :  1 ≠  2 Confidence level,  =0.05 Assuming the population distributions are normal Number of (independent) runs = 30 for each case Use t test Applying the Theory Function Optimization

45. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 45 From the samples (fitness of the best individuals)  sga =5.4838  tr =0.0768 S ga 2 =149.788 S tr 2 =0.02958 From the formulas df=29 t=2.42 p-value  2*0.012=0.024 Conclusion With  =0.05 and p-value =0.024, the null hypothesis H 0 is rejected. Applying the Theory Function Optimization

46. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 46 Conclusions This is a very simple presentation Assuming Normal distributions There are many others In many situations we cannot assume a normal distribution Many things left unmentioned More than two populations Analysis of Variance (ANOVA) Regression and Correlation Non parametric methods

47. DEI/CISUC Evonet Summer School - Parma © 2003 Ernesto Costa 47 Want to know more? Paul Cohen, Empirical Methods for Artificial Intelligence. MIT Press, Boston, 1995 James Kennedy and Russell Eberhart, Swarm Intelligence (Appendix A),Morgan Kaufman, 2001. Roxy Peck, Chris Olsen and Jay Devore, Introduction to Statistics and Data Analysis,Duxbury, 2001. Mark Wineberg and Steffen Christensen, Using Appropriate Statistics, GECCO’2003 Tutorial.