If you think you made a lot of mistakes in the survey project…. Think of how much you accomplished and the mistakes you did not make…

Went from not knowing much about surveys to having designed, deployed, and completed one in 1 ½ monthsWent from not knowing much about surveys to having designed, deployed, and completed one in 1 ½ months Actually got people to respond!Actually got people to respond! Did not end up with 100 open ended responses which you had to content analyze!Did not end up with 100 open ended responses which you had to content analyze!

One Tailed and Two Tailed tests One tailed tests: Based on a uni-directional hypothesis Example: Effect of training on problems using PowerPoint Population figures for usability of PP are known Hypothesis: Training will decrease number of problems with PP Two tailed tests: Based on a bi-directional hypothesis Hypothesis: Training will change the number of problems with PP

If we know the population mean Mean Usability Index Sampling Distribution Population for usability of Powerpoint Frequency Std. Dev =.45 Mean = 5.65 N = Unidirectional hypothesis:.05 level Bidirectional hypothesis:.05 level Identify region

What does it mean if our significance level is.05?What does it mean if our significance level is.05? XFor a uni-directional hypothesis XFor a bi-directional hypothesis PowerPoint example: UnidirectionalUnidirectional XIf we set significance level at.05 level, 5% of the time we will higher mean by chance5% of the time we will higher mean by chance 95% of the time the higher mean mean will be real95% of the time the higher mean mean will be real BidirectionalBidirectional XIf we set significance level at.05 level 2.5 % of the time we will find higher mean by chance2.5 % of the time we will find higher mean by chance 2.5% of the time we will find lower mean by chance2.5% of the time we will find lower mean by chance 95% of time difference will be real95% of time difference will be real

Changing significance levels What happens if we decrease our significance level from.01 to.05What happens if we decrease our significance level from.01 to.05 XProbability of finding differences that don’t exist goes up (criteria becomes more lenient) What happens if we increase our significance from.01 to.001What happens if we increase our significance from.01 to.001 XProbability of not finding differences that exist goes up (criteria becomes more conservative)

PowerPoint example:PowerPoint example: XIf we set significance level at.05 level, 5% of the time we will find a difference by chance5% of the time we will find a difference by chance 95% of the time the difference will be real95% of the time the difference will be real XIf we set significance level at.01 level 1% of the time we will find a difference by chance1% of the time we will find a difference by chance 99% of time difference will be real99% of time difference will be real For usability, if you are set out to find problems: setting lenient criteria might work better (you will identify more problems)For usability, if you are set out to find problems: setting lenient criteria might work better (you will identify more problems)

Effect of decreasing significance level from.01 to.05Effect of decreasing significance level from.01 to.05 XProbability of finding differences that don’t exist goes up (criteria becomes more lenient) XAlso called Type I error (Alpha) Effect of increasing significance from.01 to.001Effect of increasing significance from.01 to.001 XProbability of not finding differences that exist goes up (criteria becomes more conservative) XAlso called Type II error (Beta)

Degree of Freedom The number of independent pieces of information remaining after estimating one or more parametersThe number of independent pieces of information remaining after estimating one or more parameters Example: List= 1, 2, 3, 4 Average= 2.5Example: List= 1, 2, 3, 4 Average= 2.5 For average to remain the same three of the numbers can be anything you want, fourth is fixedFor average to remain the same three of the numbers can be anything you want, fourth is fixed New List = 1, 5, 2.5, __ Average = 2.5New List = 1, 5, 2.5, __ Average = 2.5

Major Points T tests: are differences significant?T tests: are differences significant? One sample t tests, comparing one mean to populationOne sample t tests, comparing one mean to population Within subjects test: Comparing mean in condition 1 to mean in condition 2Within subjects test: Comparing mean in condition 1 to mean in condition 2 Between Subjects test: Comparing mean in condition 1 to mean in condition 2Between Subjects test: Comparing mean in condition 1 to mean in condition 2

Effect of training on Powerpoint use Does training lead to lesser problems with PP?Does training lead to lesser problems with PP? 9 subjects were trained on the use of PP.9 subjects were trained on the use of PP. Then designed a presentation with PP.Then designed a presentation with PP. XNo of problems they had was DV

Powerpoint study data Mean = 23.89Mean = SD = 4.20SD = 4.20

Results of Powerpoint study. ResultsResults XMean number of problems = Assume we know that without training the mean would be 30, but not the standard deviationAssume we know that without training the mean would be 30, but not the standard deviation Population mean = 30 Is enough smaller than 30 to conclude that training affected results?Is enough smaller than 30 to conclude that training affected results?

One sample t test cont. Assume mean of population known, but standard deviation (SD) not knownAssume mean of population known, but standard deviation (SD) not known Substitute sample SD for population SD (standard error)Substitute sample SD for population SD (standard error) Gives you the t statisticsGives you the t statistics Compare t to tabled values which show critical values of tCompare t to tabled values which show critical values of t

t Test for One Mean Get mean difference between sample and population meanGet mean difference between sample and population mean Use sample SD as variance metric = 4.40Use sample SD as variance metric = 4.40

Degrees of Freedom Skewness of sampling distribution of variance decreases as n increasesSkewness of sampling distribution of variance decreases as n increases t will differ from z less as sample size increasest will differ from z less as sample size increases Therefore need to adjust t accordinglyTherefore need to adjust t accordingly df = n - 1df = n - 1 t based on dft based on df

Looking up critical t (Table E.6)

Conclusions Critical t= n = 9, t.05 = 2.62 (two tail significance)Critical t= n = 9, t.05 = 2.62 (two tail significance) If t > 2.62, reject H 0If t > 2.62, reject H 0 Conclude that training leads to less problemsConclude that training leads to less problems

Factors Affecting t Difference between sample and population meansDifference between sample and population means Magnitude of sample varianceMagnitude of sample variance Sample sizeSample size

Factors Affecting Decision Significance level Significance level  One-tailed versus two-tailed testOne-tailed versus two-tailed test

Sampling Distribution of the Mean We need to know what kinds of sample means to expect if training has no effect.We need to know what kinds of sample means to expect if training has no effect. Xi. e. What kinds of sample means if population mean = XRecall the sampling distribution of the mean.

Sampling Distribution of the Mean--cont. The sampling distribution of the mean depends onThe sampling distribution of the mean depends on XMean of sampled population XSt. dev. of sampled population XSize of sample

Cont.

Sampling Distribution of the mean--cont. Shape of the sampled populationShape of the sampled population XApproaches normal XRate of approach depends on sample size XAlso depends on the shape of the population distribution

Implications of the Central Limit Theorem Given a population with mean =  and standard deviation = , the sampling distribution of the mean (the distribution of sample means) has a mean = , and a standard deviation =  /  n.Given a population with mean =  and standard deviation = , the sampling distribution of the mean (the distribution of sample means) has a mean = , and a standard deviation =  /  n. The distribution approaches normal as n, the sample size, increases.The distribution approaches normal as n, the sample size, increases.

Demonstration Let population be very skewedLet population be very skewed Draw samples of 3 and calculate meansDraw samples of 3 and calculate means Draw samples of 10 and calculate meansDraw samples of 10 and calculate means Plot meansPlot means Note changes in means, standard deviations, and shapesNote changes in means, standard deviations, and shapes Cont.

Parent Population Cont.

Sampling Distribution n = 3 Cont.

Sampling Distribution n = 10 Cont.

Demonstration--cont. Means have stayed at 3.00 throughout-- except for minor sampling errorMeans have stayed at 3.00 throughout-- except for minor sampling error Standard deviations have decreased appropriatelyStandard deviations have decreased appropriately Shapes have become more normal--see superimposed normal distribution for referenceShapes have become more normal--see superimposed normal distribution for reference

Within subjects t tests Related samplesRelated samples Difference scoresDifference scores t tests on difference scorest tests on difference scores Advantages and disadvantagesAdvantages and disadvantages

Related Samples The same participants give us data on two measuresThe same participants give us data on two measures Xe. g. Before and After treatment XUsability problems before training on PP and after training With related samples, someone high on one measure probably high on other(individual variability).With related samples, someone high on one measure probably high on other(individual variability). Cont.

Related Samples--cont. Correlation between before and after scoresCorrelation between before and after scores XCauses a change in the statistic we can use Sometimes called matched samples or repeated measuresSometimes called matched samples or repeated measures

Difference Scores Calculate difference between first and second scoreCalculate difference between first and second score Xe. g. Difference = Before - After Base subsequent analysis on difference scoresBase subsequent analysis on difference scores XIgnoring Before and After data

Effect of training

Results The training decreased the number of problems with PowerpointThe training decreased the number of problems with Powerpoint Was this enough of a change to be significant?Was this enough of a change to be significant? Before and After scores are not independent.Before and After scores are not independent. XSee raw data Xr =.64 Cont.

Results--cont. If no change, mean of differences should be zeroIf no change, mean of differences should be zero  So, test the obtained mean of difference scores against  = 0. XUse same test as in one sample test

t test D and s D = mean and standard deviation of differences. df = n - 1 = = 8 Cont.

t test--cont. With 8 df, t.025 = (Table E.6)With 8 df, t.025 = (Table E.6) We calculated t = 6.85We calculated t = 6.85 Since 6.85 > 2.306, reject H 0Since 6.85 > 2.306, reject H 0 Conclude that the mean number of problems after training was less than mean number before trainingConclude that the mean number of problems after training was less than mean number before training

Advantages of Related Samples Eliminate subject-to-subject variabilityEliminate subject-to-subject variability Control for extraneous variablesControl for extraneous variables Need fewer subjectsNeed fewer subjects

Disadvantages of Related Samples Order effectsOrder effects Carry-over effectsCarry-over effects Subjects no longer naïveSubjects no longer naïve Change may just be a function of timeChange may just be a function of time Sometimes not logically possibleSometimes not logically possible

Between subjects t test Distribution of differences between meansDistribution of differences between means Heterogeneity of VarianceHeterogeneity of Variance NonnormalityNonnormality

Powerpoint training again Effect of training on problems using PowerpointEffect of training on problems using Powerpoint XSame study as before --almost Now we have two independent groupsNow we have two independent groups XTrained versus untrained users XWe want to compare mean number of problems between groups

Effect of training

Differences from within subjects test Cannot compute pairwise differences, since we cannot compare two random people We want to test differences between the two sample means (not between a sample and population)

Analysis How are sample means distributed if H 0 is true?How are sample means distributed if H 0 is true? Need sampling distribution of differences between meansNeed sampling distribution of differences between means XSame idea as before, except statistic is (X 1 - X 2 ) (mean 1 – mean2)

Sampling Distribution of Mean Differences Mean of sampling distribution =  1 -  2Mean of sampling distribution =  1 -  2 Standard deviation of sampling distribution (standard error of mean differences) =Standard deviation of sampling distribution (standard error of mean differences) = Cont.

Sampling Distribution--cont. Distribution approaches normal as n increases.Distribution approaches normal as n increases. Later we will modify this to “pool” variances.Later we will modify this to “pool” variances.

Analysis--cont. Same basic formula as before, but with accommodation to 2 groups.Same basic formula as before, but with accommodation to 2 groups. Note parallels with earlier tNote parallels with earlier t

Degrees of Freedom Each group has 6 subjects.Each group has 6 subjects. Each group has n - 1 = = 8 dfEach group has n - 1 = = 8 df Total df = n n = n 1 + n = 16 dfTotal df = n n = n 1 + n = 16 df t.025 (16) = (approx.)t.025 (16) = (approx.)

Conclusions T = 4.13T = 4.13 Critical t = 2.12Critical t = 2.12 Since 4.13 > 2.12, reject H 0.Since 4.13 > 2.12, reject H 0. Conclude that those who get training have less problems than those without trainingConclude that those who get training have less problems than those without training

Assumptions Two major assumptionsTwo major assumptions XBoth groups are sampled from populations with the same variance “homogeneity of variance”“homogeneity of variance” XBoth groups are sampled from normal populations Assumption of normalityAssumption of normality XFrequently violated with little harm.

Heterogeneous Variances Refers to case of unequal population variances.Refers to case of unequal population variances. We don’t pool the sample variances.We don’t pool the sample variances. We adjust df and look t up in tables for adjusted df.We adjust df and look t up in tables for adjusted df. Minimum df = smaller n - 1.Minimum df = smaller n - 1. XMost software calculates optimal df.