CS 321: Human-Computer Interaction Design Lesson Sixteen Sample Size Iterative Procedure Problem Discovery Large Samples Chapters 6-7

Determining Sample Size could be solved for n, the needed sample size. When conducting a usability test, how large should you make the sample size? Essentially, if you can estimate the critical difference from the test (i.e., d = the smallest difference between the obtained and true value that you need to detect), the sample’s standard deviation (which might be estimated from previous similar experiments), and the critical t-value (for the desired level of statistical confidence), then the formula for t: 𝒕= 𝒅 ( 𝒔 𝒏 ) could be solved for n, the needed sample size. Unlike the z-value, however, which uses a normal distribution, estimating the t-value complicates matters by also being dependent on the degrees of freedom (for a one-sample t-test, df = n - 1). To overcome this problem, an iterative procedure is suggested… CS 321 Lesson Sixteen Sample Size Page 207

Determining Sample Size: Iterative Procedure d = critical difference between obtained and true value 𝒕= 𝒅 ( 𝒔 𝒏 ) s = sample standard deviation estimate (from previous experiments) Use the z-score with the desired level of confidence (from a unit normal table) as an initial estimate of the t-value. Solve the above equation for n. Use a t-distribution table to find the t-score for that value of n (with df = n-1). Recalculate n by using this new t-value in the equation above. Revise the t-score from the t-distribution table. Continue this iteration until two consecutive cycles yield the same n value. CS 321 Lesson Sixteen Sample Size Page 208

Solving the basic t formula for n yields: Sample Size Example Assume that you have been using a 100-point item as a post-task measure of ease-of-use in past usability tests. One of the tasks that you routinely conduct is software installation. For the most recent usability study of the current version of the software package, the variability of this measurement on the 100-point scale is 25 (i.e., s = 5). You’re planning your first usability study with a new version of the software, and you want to get an estimate of this measure with 90% confidence and to be within 2.5 points of the true value. Let’s calculate how many participants you need to run in the study. Solving the basic t formula for n yields: n = 𝒕 𝟐 𝒔 𝟐 𝒅 𝟐 The question indicates that s = 5 and d = 2.5, so an appropriate t-value needs to be determined. CS 321 Lesson Sixteen Sample Size Page 209

Sample Size Example (Continued) For two-sided testing with a 90% confidence interval (i.e., 5% in each tail), a unit normal table indicates that a z-value of 1.645 would make a good first estimate for the t-value. Using the above formula, this yields an n-value of 10.8241, which rounds up to 11. Switching to a t-distribution table, n = 11 (i.e., df = 10) gives us a t-value of 1.812 for a 2-tailed 90% confidence interval, which produces an n-value of 13.133376 in the formula, rounding up to 14. Using n = 14 (df = 13) yields a t-value of 1.771, yielding an n-value of 12.545764, rounding up to 13. Using n = 13 (df = 12) yields a t-value of 1.782, yielding an n-value of 12.702096, again rounding up to 13. Therefore, the final sample estimate size for this study is 13 participants. CS 321 Lesson Sixteen Sample Size Page 210

Problem Discovery Sample Size Recall the binomial probability function, which determines the probability that a problem with probability p will occur k times during a study with n subjects: 𝒏! 𝒌! 𝒏−𝒌 ! 𝒑 𝒌 (𝟏−𝒑) 𝒏−𝒌 By setting k to zero, we get the probability that the problem won’t occur at all with n subjects: 𝒏! 𝟎! 𝒏−𝟎 ! 𝒑 𝟎 (𝟏−𝒑) 𝒏−𝟎 = (𝟏−𝒑) 𝒏 So, for n test subjects, the probability that the problem will occur at least once is: 𝒑 𝒙≥𝟏 =𝟏− (𝟏−𝒑) 𝒏 Solving this for the sample size, we get: 𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) CS 321 Lesson Sixteen Sample Size Page 211

Problem Discovery Sample Size (Continued) So, for example, if we want to be 85% certain of observing a problem that affects 31% of all users, then our usability study will need a sample size of: 𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) = ln (𝟏−𝟎.𝟖𝟓) ln (𝟏−𝟎.𝟑𝟏) = ln (𝟎.𝟏𝟓) ln (𝟎.𝟔𝟗) ≈ −𝟏.𝟖𝟗𝟕 −𝟎.𝟑𝟕𝟏 ≈𝟓 If we want to be 85% certain of observing a problem that affects 10% of all users: 𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) = ln (𝟏−𝟎.𝟖𝟓) ln (𝟏−𝟎.𝟏) = ln (𝟎.𝟏𝟓) ln (𝟎.𝟗) ≈ −𝟏.𝟖𝟗𝟕 −𝟎.𝟏𝟎𝟓 ≈𝟏𝟖 And to be 85% certain of observing a problem that affects only 1% of all users: 𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) = ln (𝟏−𝟎.𝟖𝟓) ln (𝟏−𝟎.𝟏) = ln (𝟎.𝟏𝟓) ln (𝟎.𝟗𝟗) ≈ −𝟏.𝟖𝟗𝟕 −𝟎.𝟎𝟏 ≈𝟏𝟖𝟗 CS 321 Lesson Sixteen Sample Size Page 212

Weak Arguments For Large Samples “If the population is large, then the sample needs to be large.” The variance in statistical sampling is determined by the sample size, not the size of the overall population. The evaluation of a design element’s quality is independent of how many people are going to use it. “The more features in the interface, the larger the sample size.” When the interface is loaded with features, more tests are needed, not more users in each test. Test subjects will be overwhelmed if asked to evaluate too many features. “The interface is being designed to accommodate many target audiences.” This only requires larger sample sizes if the different target audiences will use the interface in very different ways (e.g., buyers vs. sellers, teachers vs. students, doctors vs. patients). CS 321 Lesson Sixteen Sample Size Page 213