206. CS 321: Human-Computer Interaction Design
Lesson Sixteen
Sample Size
Iterative Procedure
Problem Discovery
Large Samples
Chapters 6-7

207. Determining Sample Size could be solved for n, the needed sample size.
When conducting a usability test, how large should you make the sample size?
Essentially, if you can estimate the critical difference from the test (i.e., d = the smallest difference between the obtained and true value that you need to detect), the sample’s standard deviation (which might be estimated from previous similar experiments), and the critical t-value (for the desired level of statistical confidence), then the formula for t:
𝒕= 𝒅 ( 𝒔 𝒏 )
could be solved for n, the needed sample size.
Unlike the z-value, however, which uses a normal distribution, estimating the t-value complicates matters by also being dependent on the degrees of freedom (for a one-sample t-test, df = n - 1). To overcome this problem, an iterative procedure is suggested…
CS 321
Lesson Sixteen
Sample Size
Page 207

208. Determining Sample Size: Iterative Procedure
d = critical difference between obtained and true value
𝒕= 𝒅 ( 𝒔 𝒏 )
s = sample standard deviation estimate (from previous experiments)
Use the z-score with the desired level of confidence (from a unit normal table) as an initial estimate of the t-value.
Solve the above equation for n.
Use a t-distribution table to find the t-score for that value of n (with df = n-1).
Recalculate n by using this new t-value in the equation above.
Revise the t-score from the t-distribution table.
Continue this iteration until two consecutive cycles yield the same n value.
CS 321
Lesson Sixteen
Sample Size
Page 208

209. Solving the basic t formula for n yields:
Sample Size Example
Assume that you have been using a 100-point item as a post-task measure of ease-of-use in past usability tests. One of the tasks that you routinely conduct is software installation. For the most recent usability study of the current version of the software package, the variability of this measurement on the 100-point scale is 25 (i.e., s = 5).
You’re planning your first usability study with a new version of the software, and you want to get an estimate of this measure with 90% confidence and to be within 2.5 points of the true value. Let’s calculate how many participants you need to run in the study.
Solving the basic t formula for n yields:
n = 𝒕 𝟐 𝒔 𝟐 𝒅 𝟐
The question indicates that s = 5 and d = 2.5, so an appropriate t-value needs to be determined.
CS 321
Lesson Sixteen
Sample Size
Page 209

210. Sample Size Example (Continued)
For two-sided testing with a 90% confidence interval (i.e., 5% in each tail), a unit normal table indicates that a z-value of would make a good first estimate for the t-value. Using the above formula, this yields an n-value of , which rounds up to 11.
Switching to a t-distribution table, n = 11 (i.e., df = 10) gives us a t-value of for a 2-tailed 90% confidence interval, which produces an n-value of in the formula, rounding up to 14.
Using n = 14 (df = 13) yields a t-value of 1.771, yielding an n-value of , rounding up to 13. Using n = 13 (df = 12) yields a t-value of 1.782, yielding an n-value of , again rounding up to 13. Therefore, the final sample estimate size for this study is 13 participants.
CS 321
Lesson Sixteen
Sample Size
Page 210

211. Problem Discovery Sample Size
Recall the binomial probability function, which determines the probability that a problem with probability p will occur k times during a study with n subjects:
𝒏! 𝒌! 𝒏−𝒌 ! 𝒑 𝒌 (𝟏−𝒑) 𝒏−𝒌
By setting k to zero, we get the probability that the problem won’t occur at all with n subjects:
𝒏! 𝟎! 𝒏−𝟎 ! 𝒑 𝟎 (𝟏−𝒑) 𝒏−𝟎 = (𝟏−𝒑) 𝒏
So, for n test subjects, the probability that the problem will occur at least once is:
𝒑 𝒙≥𝟏 =𝟏− (𝟏−𝒑) 𝒏
Solving this for the sample size, we get:
𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑)
CS 321
Lesson Sixteen
Sample Size
Page 211

212. Problem Discovery Sample Size (Continued)
So, for example, if we want to be 85% certain of observing a problem that affects 31% of all users, then our usability study will need a sample size of:
𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) = ln (𝟏−𝟎.𝟖𝟓) ln (𝟏−𝟎.𝟑𝟏) = ln (𝟎.𝟏𝟓) ln (𝟎.𝟔𝟗) ≈ −𝟏.𝟖𝟗𝟕 −𝟎.𝟑𝟕𝟏 ≈𝟓
If we want to be 85% certain of observing a problem that affects 10% of all users:
𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) = ln (𝟏−𝟎.𝟖𝟓) ln (𝟏−𝟎.𝟏) = ln (𝟎.𝟏𝟓) ln (𝟎.𝟗) ≈ −𝟏.𝟖𝟗𝟕 −𝟎.𝟏𝟎𝟓 ≈𝟏𝟖
And to be 85% certain of observing a problem that affects only 1% of all users:
𝒏= ln (𝟏−𝒑 𝒙≥𝟏 ) ln (𝟏−𝒑) = ln (𝟏−𝟎.𝟖𝟓) ln (𝟏−𝟎.𝟏) = ln (𝟎.𝟏𝟓) ln (𝟎.𝟗𝟗) ≈ −𝟏.𝟖𝟗𝟕 −𝟎.𝟎𝟏 ≈𝟏𝟖𝟗
CS 321
Lesson Sixteen
Sample Size
Page 212

213. Weak Arguments For Large Samples
“If the population is large, then the sample needs to be large.”
The variance in statistical sampling is determined by the sample size, not the size of the overall population. The evaluation of a design element’s quality is independent of how many people are going to use it.
“The more features in the interface, the larger the sample size.”
When the interface is loaded with features, more tests are needed, not more users in each test. Test subjects will be overwhelmed if asked to evaluate too many features.
“The interface is being designed to accommodate many target audiences.”
This only requires larger sample sizes if the different target audiences will use the interface in very different ways (e.g., buyers vs. sellers, teachers vs. students, doctors vs. patients).
CS 321
Lesson Sixteen
Sample Size
Page 213