F, t, and p Basic Statistics for Computer Scientists (aka knowing enough to be critical of user studies) April 4, 2002 Benjamin Lok

User Studies Trying to identify phenomena or trends –Hypothesis Blood pressure increases with age and weight Smoking increases risk of cancer Real objects in VEs improve performance –How might we investigate this?

Variables and Conditions Hypothesis: Real objects in VEs improve performance Independent Variable – the variable that is being manipulated by the experimenter (VE type) Dependent Variable – the variable that is caused by the independent variable. (performance) Experimental conditions – The level of independent variable in which the situation of interest was created.

Descriptive Statistics Hypothesis: Real objects in VEs improve performance null hypothesis - assume real objects in VEs are the SAME as virtual objects in VEs Innocent until proven guilty Your job: Prove otherwise! alternate hypothesis – interacting with real objects is better than interacting with virtual objects

Raw Data What does the mean tell us? Is that enough?

Variances standard deviation – measure of dispersion (square root of the sum of squares divided by N) Small Pattern (seconds)Large Pattern (seconds) MeanS.D.MinMaxMeanS.D.MinMax Real Space (n=41) Purely Virtual (n=13) Hybrid (n=13) Vis Faith Hybrid (n=14)

Hypothesis We assumed the means are “equal” But are they? Or is the difference due to chance? Small Pattern (seconds)Large Pattern (seconds) MeanS.D.MinMaxMeanS.D.MinMax Real Space (n=41) Purely Virtual (n=13) Hybrid (n=13) Vis Faith Hybrid (n=14)

T - test T – test – statistical test used to determine whether two observed means are statistically different

T – test (rule of thumb) Good values of t > 1.96 Look at what contributes to t tm

F statistic, p values F statistic – assesses the extent to which the means of the experimental conditions differ more than would be expected by chance t is related to F statistic Look up a table, get the p value. Compare to α α value – probability of making a Type I error (rejecting null hypothesis when really true) p value – statistical likelihood of an observed pattern of data, calculated on the basis of the sampling distribution of the statistic. (% chance it was due to chance)

Let’s look at data Small PatternLarge Pattern t – test with unequal variance p – value t – test with unequal variance p - value PVE – RSE vs. VFHE – RSE ** *** PVE – RSE vs. HE – RSE ** * VFHE – RSE vs. HE – RSE

Total Sense of Presence Score Scale from 0..6 Mea n S.D Mi n Ma x Purely VE Hybrid VE Visually Faithful Hybrid VE Between Groups Total Sense of Presence t – test with unequal variance p – value PVE – VFHE PVE – HE VFHE – HE

Significance What does it mean to be significant? You have some confidence it was not due to chance. But difference between statistical significance and meaningful significance Always know: –samples (n) –p value –variance/standard deviation –means