1. Summary of Test Sizing
Sample size calculations will use different equations and/or techniques, depending on the application.
Test sizing is a rational process
Test sizing is hard
But it’s important
But it’s hard

2. The Wise Approach to Sample Size
The situation drives the analytical approach to sample size - One size does NOT fit all
Analysts should have tools for your situation
Binomial data? (nomograph from Test 101 or AFOTEC program)
T-distributed data (see Montgomery’s* operating characteristic curves, p40)
ANOVA? (see Montgomery’s operating characteristic curves, p188, )
Etc.
Just to give a sense of the variety of situations, each with their different tools. I’ll probably pass out hard copies of a t distribution operating characteristic curve set, just to help walk them through an example – or at least as a visual aid.
*Douglas Montgomery. Design and Analysis of Experiments, 5th ed

3. The Wise Approach (Continued)
Understand basic stats. In particular, understand that test size depends on four quantities:
Alpha (Producers Risk)
Beta (Consumers Risk)
Sigma (inherent variability in MOE – standard deviation)
Delta (Operationally significant amount of improvement in MOE over spec. The Militarily Significant Difference.)
Get Alpha and Beta from decision-makers
Get Sigma and Delta from subject matter experts (or try to calculate Sigma from test data, if you have data)
Now SEEK PROFESSIONAL HELP
This is the rational approach. It amounts to asking your decisionmakers how much risk they can tolerate, determining in advance how much variability there will be, and finding out how much of a difference is operationally significant.
Your professionals are those who have a lot of experience and education in statistics. No, your typical engineer, scientist, or even mathematician may not be up on the approaches to test sizing and may have to hit the books or go get educated. No, this isn’t taught at AFIT or NPS. Yes, you still need that expertise, even though it’s hard to find.

4. Sample Size Calculation - JPATS Example
Sample Size controls magnitude of Producer’s (a) & Consumer’s (b) Risks
We will calculate sample size (n), based on a given “militarily significant difference” with desired values of a and b, for a given system.
Militarily Significant Difference – The difference (in the measured value) that is operationally significant to the warfighter.
The test objective is to determine whether JPATS speed meets the specification

5. JPATS Aircraft Example Sample Size Calculation
Calculate sample size needed to detect a KTAS militarily significant difference (D)
Normally Distributed Data
Assumed s = 15 KTAS
Let a (producer’s risk) = .05
Let b (consumer’s risk) = .10
KTAS = Knots True Airspeed
KTAS is Knots True air speed

6. JPATS Sample Size Calculations
m0 - mT = D = militarily significant difference
The 20 KTAS militarily significant difference means that a 20 KTAS improvement is significant to the warfighter.
m0 - mT
(s / n. 5 )
= Za + Zb
This example is for a one tailed case, that is, we are only interested in meeting or exceeding a value. The technique is similar for two tailed cases, except that the a and b values used are divided by 2, and there may be different thresholds for upper and lower limits. Showing the formulas for these cases is beyond the scope of the course.
In this example, the variance is small compared to the mean values and the difference between the objective and the threshold values. This yields a small sample size and a high confidence levels. The usual case, would have a higher variance compared to the mean value, which would cause the required sample size to increase greatly.
The equation on the left of the slide looks just like the equation we used for the central limit theorem as applied to the normal distribution, and for the t distribution. The denominator is the standard deviation of the sample mean, and the numerator is the physical distance between the values. The right hand side of the equation is simply the number of standard deviations the two physical measurements are apart.

7. Determining Za and Zb
Za and Zb are the statistical distance (number of standard deviations from the mean)
a = .05
b= .10
Assuming normal distribution, Za and Zb can be obtained from the normal distribution chart on the next slide.
Za = Z.05 = 1.65
Zb = Z.10 = 1.28

8. Cumulative Normal Distribution
Values of Zp Given P

9. JPATS Sample Size Calculations2
= ( )
15( )
(20)
m0 - mT
(s / n. 5 )
n = 4.83 ~ 5
= Za + Zb
This example is for a one tailed case, that is, we are only interested in meeting or exceeding a value. The technique is similar for two tailed cases, except that the a and b values used are divided by 2, and there may be different thresholds for upper and lower limits. Showing the formulas for these cases is beyond the scope of the course.
In this example, the variance is small compared to the mean values and the difference between the objective and the threshold values. This yields a small sample size and a high confidence levels. The usual case, would have a higher variance compared to the mean value, which would cause the required sample size to increase greatly.
The equation on the left of the slide looks just like the equation we used for the central limit theorem as applied to the normal distribution, and for the t distribution. The denominator is the standard deviation of the sample mean, and the numerator is the physical distance between the values. The right hand side of the equation is simply the number of standard deviations the two physical measurements are apart.
A sample size of 5 will give
the required protection against
Type I and Type II errors.
Z.05 = 1.65
Z.10 = 1.28

10. Distribution of Sample Averages
20 KTAS
5% Producers Risk
10% Consumers Risk
The true average is an unknown number
A sample size of 5 will allow the sample average to be within
20 KTAS of the true average, with a = .05 and b = .10
The math we used is -only- valid for normally distributed data.