1. Experimental Design and Analysis Instructor: Mark Hancock February 8, 2008Slides by Mark Hancock

2. What is an experiment? February 8, 2008Slides by Mark Hancock

3. You will be able to describe some of the key elements of an experiment that can be analyzed with a statistical test. February 8, 2008Slides by Mark Hancock

4. Elements of an Experiment People Data – Measurement Hypothesis (There are more… we’ll learn about them later) February 8, 2008Slides by Mark Hancock

5. People Sample (participants) – People in your study Population – E.g., Canadians, computer scientists, artists – People we want to generalize to February 8, 2008Slides by Mark Hancock

6. Data Variable – E.g., technique, task time, number of errors Statistic – Mean, median, mode, standard deviation, etc. – Taken from the sample Parameter – Taken from the population February 8, 2008Slides by Mark Hancock

7. Hypothesis (examples) The average temperature in Calgary is less than -20˚C. A pair of dice will result in a roll of 7 more than it will result in a roll of 10. Canadians prefer Hockey to Baseball. February 8, 2008Slides by Mark Hancock

8. Hypotheses Carman is a great foosball player. Carman is better at foosball than Mark. Carman wins more foosball games than Mark. Mark scores more points in foosball than Carman. February 8, 2008Slides by Mark Hancock

9. Hypotheses TreeMaps are easy to use. TreeMaps are better than Phylotrees. People find leaf nodes faster with TreeMaps than with Phylotrees. People find sibling nodes faster with Phylotrees than with TreeMaps. February 8, 2008Slides by Mark Hancock

10. Null Hypothesis “… the null hypothesis is a pinpoint statement as to the unknown quantitative value of the parameter in the population[s] of interest.” Huck, S.W. Reading Statistics and Research February 8, 2008Slides by Mark Hancock

11. Null Hypothesis Calgary Temperature: μ Calgary = -20˚C Dice Rolling: μ 7 = μ 10 μ 7 - μ 10 = 0 Foosball: μ Carman = μ Mark Tree Vis: μ TreeMap = μ Phylotrees February 8, 2008Slides by Mark Hancock

12. What is the null hypothesis? Hypothesis: The average temperature in Vancouver is higher than the average temperature in Calgary H 0 : μ Calgary = μ Vancouver February 8, 2008Slides by Mark Hancock

13. Elements of an Experiment People – Sample/Participants – Population Data – Variables – Statistics – Parameters Hypotheses – Null Hypothesis February 8, 2008Slides by Mark Hancock

14. Label this graph: Independent Variable? Dependent Variable? February 8, 2008Slides by Mark Hancock

15. Variables Independent Variables (Factors) – What you (the experimenter) are changing during the experiment Dependent Variables (Measures) – What is being measured Constants – What is the same for all participants February 8, 2008Slides by Mark Hancock

16. Variables (Exercise) Questionnaire: ask computer science students to rate their favourite teacher. What independent variables would you use? What dependent variables would you use? What would you keep constant? February 8, 2008Slides by Mark Hancock

17. Problem: hypotheses are about population, but we only have access to data from a sample. February 8, 2008Slides by Mark Hancock

18. You will be able to describe why the Law of Large Numbers and the Central Limit Theorem allow us to make general statements about a population based on information about a sample. February 8, 2008Slides by Mark Hancock

19. Dice Rolling Roll one die – Predictions? Roll one die 5 times and take the average? – Predictions? Roll one die 100 times and take the average? – Predictions? February 8, 2008Slides by Mark Hancock

20. Dice Rolling Roll one die n times: – Possible outcomes: 1, 2, 3, 4, 5, 6 – Probability of rolling X: P(X) = 1/6 = 16.7% – Expected Value: E(X) = 1(1/6) + 2(1/6) + … + 6(1/6) = 3.5 February 8, 2008Slides by Mark Hancock

21. Law of Large Numbers “Given a sample of independent and identically distributed random variables with a finite expected value, the average of these observations will eventually approach and stay close to the expected value.” "Law of large numbers." Wikipedia February 8, 2008Slides by Mark Hancock

22. Experiment “Law of Large Numbers”, Wikipedia February 8, 2008Slides by Mark Hancock

23. Central Limit Theorem “…if the sum of independent identically distributed random variables has a finite variance, then it will be approximately normally distributed.” “Central Limit Theorem." Wikipedia February 8, 2008Slides by Mark Hancock

24. Gaussian/Normal Distribution Standard Deviation Mean February 8, 2008Slides by Mark Hancock

25. LLN vs. CLT Law of Large Numbers “Given a sample of independent and identically distributed random variables with a finite expected value, the average of these observations will eventually approach and stay close to the expected value.” Central Limit Theorem “…if the sum of independent identically distributed random variables has a finite variance, then it will be approximately normally distributed.” February 8, 2008Slides by Mark Hancock

26. Generalize from μ sample to μ population What information do we have? – Sample mean – Sample variance What information do we seek? – Population mean – Population variance February 8, 2008Slides by Mark Hancock

27. Generalize from μ sample to μ population Assumptions: – population has expected value of μ population – population has finite variance σ population Conclude: – provided we have enough people (N is large): μ sample  μ population (by LLN) σ sample  σ population (by CLT) February 8, 2008Slides by Mark Hancock

28. Summary Dependent/independent variables Constants Law of Large Numbers: – eventually data tends to the expected value Central Limit Theorem: – most data tends toward a normal distribution February 8, 2008Slides by Mark Hancock

29. Break: 15 Minutes February 8, 2008Slides by Mark Hancock

30. Significance and Power February 8, 2008Slides by Mark Hancock

31. You will be able to identify two types of errors and be able to avoid these errors when running a study. February 8, 2008Slides by Mark Hancock

32. Types of mistakes 1.Find a difference when there isn’t one 2.Find no difference when there is one February 8, 2008Slides by Mark Hancock

33. Rejecting the Null Hypothesis (H 0 ) Reality H 0 trueH 0 false Decision H 0 t rue Type II H 0 false Type I February 8, 2008Slides by Mark Hancock

34. Rejecting the Null Hypothesis (H 0 ) Reality H 0 trueH 0 false Decision H 0 t rue β H 0 false α or p February 8, 2008Slides by Mark Hancock

35. Significance (α) – calculated after the experiment Power (1 - β) – calculated before (a priori) or after (post hoc) – depends on effect size and sample size February 8, 2008Slides by Mark Hancock

36. How do we avoid these errors? 1.Decide before the analysis how acceptable this would be (e.g., p <.05). 2.The smaller the effect size you expect, the larger sample size you need. February 8, 2008Slides by Mark Hancock

37. (Student’s) T-Test February 8, 2008Slides by Mark Hancock

38. Who is attributed with the discovery of the Student’s T-Test? February 8, 2008Slides by Mark Hancock

39. A student! – William Sealy Gosset Guinness Brewery employee Monitored beer quality February 8, 2008Slides by Mark Hancock

40. You will be able to formulate the appropriate null hypothesis and calculate the t-value for data from a sample. February 8, 2008Slides by Mark Hancock

41. Null Hypotheses μ = μ 0 (constant value) μ A = μ B February 8, 2008Slides by Mark Hancock

42. Assumptions Data is distributed normally Equal variance: σ A = σ B (for second H 0 ) February 8, 2008Slides by Mark Hancock

43. Example Independent Variable: – TreeMap vs. Phylotrees Dependent Variable: – Time to find a leaf node Data: – 30 people used TreeMap, 30 used Phylotrees – Found one leaf node each February 8, 2008Slides by Mark Hancock

44. Check the Null Hypothesis Null Hypothesis (for population): μ T = μ P or μ T – μ P = 0 Test A (for sample): check value of μ T – μ P February 8, 2008Slides by Mark Hancock

45. How is the data distributed? Sample Distribution February 8, 2008Slides by Mark Hancock

46. How do you account for the differences in variance? February 8, 2008Slides by Mark Hancock

47. February 8, 2008Slides by Mark Hancock

48. February 8, 2008Slides by Mark Hancock

49. a.k.a. Standard Error of difference between means February 8, 2008Slides by Mark Hancock

50. February 8, 2008Slides by Mark Hancock

51. Interpreting the T Ratio What makes the ratio large? 1.Larger difference 2.Smaller variance Large t => more likely to be a real difference February 8, 2008Slides by Mark Hancock

52. How do we find significance? Look up in a table (the math is too hard for humans to do) Pick a level of significance (e.g., α =.05) and find the row corresponding to your sample size (df = n – 1). If t > (value in that cell), then p < α February 8, 2008Slides by Mark Hancock

53. df0.20.10.050.0250.020.010.005 11.3763.0786.31412.70615.89431.82163.656 21.0611.8862.9204.3034.8496.9659.925 30.9781.6382.3533.1823.4824.5415.841 40.9411.5332.1322.7762.9993.7474.604 50.9201.4762.0152.5712.7573.3654.032 … 950.8451.2911.6611.9852.0822.3662.629 960.8451.2901.6611.9852.0822.3662.628 970.8451.2901.6611.9852.0822.3652.627 980.8451.2901.6611.9842.0812.3652.627 990.8451.2901.6601.9842.0812.3652.626 1000.8451.2901.6601.9842.0812.3642.626 … February 8, 2008Slides by Mark Hancock

54. Break: 20 Minutes February 8, 2008Slides by Mark Hancock

55. Analysis of Variance (ANOVA) February 8, 2008Slides by Mark Hancock

56. You will be able to formulate the appropriate null hypothesis and calculate the F-score for data from a sample. February 8, 2008Slides by Mark Hancock

57. Null Hypotheses μ A = μ B = μ c = … Remember: “the null hypothesis is a pinpoint statement “ σ μ = 0 February 8, 2008Slides by Mark Hancock

58. Assumptions Data is distributed normally Homogeneity of variance: σ A = σ B = σ C = … A, B, C, … are independent from one another February 8, 2008Slides by Mark Hancock

59. Example Independent Variable: – TreeMap vs. Phylotrees vs. ArcTrees Dependent Variable: – Time to find a leaf node Data: – 30 people used TreeMap, 30 used Phylotrees, 30 used ArcTrees – Found one leaf node each February 8, 2008Slides by Mark Hancock

60. Check the Null Hypothesis Null Hypothesis (for population): μ T = μ P = μ A Test A (for sample): check H 0 for sample we know this is not enough! February 8, 2008Slides by Mark Hancock

61. February 8, 2008Slides by Mark Hancock

62. February 8, 2008Slides by Mark Hancock

63. Degrees of Freedom (df) How many more pieces of data you need e.g., if you have μ, you need n-1 pieces of data to find the missing piece of data February 8, 2008Slides by Mark Hancock

64. Sum of Squares (SS) Measure of variance February 8, 2008Slides by Mark Hancock

65. Mean Square “Mean” of the sum of squares February 8, 2008Slides by Mark Hancock

66. F-Score (Fisher’s Test) February 8, 2008Slides by Mark Hancock

67. F-Score (Fisher’s Test) February 8, 2008Slides by Mark Hancock

68. Example Degrees of Freedom Sum of Squares Mean Square F Between Groups3988.19329.404.5 Within Groups14610679.7273.15 Total14911667.91 How many groups (i.e., how many means are we comparing)? How many total participants? Report as: F(3,146) = 4.5 February 8, 2008Slides by Mark Hancock

69. Example “…The results of a one-way ANOVA indicated that UFOV [useful field of view] reduction increased with dementia severity, F(2,52) = 15.36, MSe = 5371.5, p <.0001. How many groups of participants were there? How many total participants were there? Fill in the table from before… February 8, 2008Slides by Mark Hancock

70. Example Degrees of Freedom Sum of Squares Mean Square F Between Groups2165,012.4882,506.2415.36 Within Groups52279,3185,371.5 Total54444,330.48 February 8, 2008Slides by Mark Hancock

71. Example Independent Variable: – TreeMap vs. Phylotrees vs. ArcTrees Data: – 30 people used TreeMap, 30 used Phylotrees, 30 used ArcTrees – Found one leaf node each Fill in the degrees of freedom column February 8, 2008Slides by Mark Hancock

72. Example Degrees of Freedom Sum of Squares Mean Square F Between Groups2 Within Groups87 Total89 February 8, 2008Slides by Mark Hancock

73. What does large F mean? Remember: Consider the null hypothesis What does each value estimate? February 8, 2008Slides by Mark Hancock

74. F-table (α =.05) df between 12345678910 df within 310.139.559.289.129.018.948.898.858.818.79 47.716.946.596.396.266.166.096.046.005.96 56.615.795.415.195.054.954.884.824.774.74 65.995.144.764.534.394.284.214.154.104.06 75.594.744.354.123.973.873.793.733.683.64 85.324.464.073.843.693.583.503.443.393.35 95.124.263.863.633.483.373.293.233.183.14 104.964.103.713.483.333.223.143.073.022.98 February 8, 2008Slides by Mark Hancock

75. Summary Analysis of Variance (ANOVA) is used to compare 2 or more means The F-score and df indicate the probability of a Type I error in rejecting the null hypothesis February 8, 2008Slides by Mark Hancock

76. Summary of First Day Elements of an experiment Null Hypothesis Variables (independent/dependent) Law of Large Numbers/Central Limit Theorem Significance and Power T-Test One-way ANOVA February 8, 2008Slides by Mark Hancock

77. Next Week Two-way & three-way ANOVA Non-parametric tests February 8, 2008Slides by Mark Hancock