Summary of Test Sizing Sample size calculations will use different equations and/or techniques, depending on the application. Test sizing is a rational process Test sizing is hard But it’s important But it’s hard

The Wise Approach to Sample Size The situation drives the analytical approach to sample size - One size does NOT fit all Analysts should have tools for your situation Binomial data? (nomograph from Test 101 or AFOTEC program) T-distributed data (see Montgomery’s* operating characteristic curves, p40) ANOVA? (see Montgomery’s operating characteristic curves, p188, 647-654) Etc. Just to give a sense of the variety of situations, each with their different tools. I’ll probably pass out hard copies of a t distribution operating characteristic curve set, just to help walk them through an example – or at least as a visual aid. *Douglas Montgomery. Design and Analysis of Experiments, 5th ed. 1997

The Wise Approach (Continued) Understand basic stats. In particular, understand that test size depends on four quantities: Alpha (Producers Risk) Beta (Consumers Risk) Sigma (inherent variability in MOE – standard deviation) Delta (Operationally significant amount of improvement in MOE over spec. The Militarily Significant Difference.) Get Alpha and Beta from decision-makers Get Sigma and Delta from subject matter experts (or try to calculate Sigma from test data, if you have data) Now SEEK PROFESSIONAL HELP This is the rational approach. It amounts to asking your decisionmakers how much risk they can tolerate, determining in advance how much variability there will be, and finding out how much of a difference is operationally significant. Your professionals are those who have a lot of experience and education in statistics. No, your typical engineer, scientist, or even mathematician may not be up on the approaches to test sizing and may have to hit the books or go get educated. No, this isn’t taught at AFIT or NPS. Yes, you still need that expertise, even though it’s hard to find.

Sample Size Calculation - JPATS Example Sample Size controls magnitude of Producer’s (a) & Consumer’s (b) Risks We will calculate sample size (n), based on a given “militarily significant difference” with desired values of a and b, for a given system. Militarily Significant Difference – The difference (in the measured value) that is operationally significant to the warfighter. The test objective is to determine whether JPATS speed meets the specification

JPATS Aircraft Example Sample Size Calculation Calculate sample size needed to detect a 20 KTAS militarily significant difference (D) Normally Distributed Data Assumed s = 15 KTAS Let a (producer’s risk) = .05 Let b (consumer’s risk) = .10 KTAS = Knots True Airspeed KTAS is Knots True air speed

JPATS Sample Size Calculations m0 - mT = D = militarily significant difference The 20 KTAS militarily significant difference means that a 20 KTAS improvement is significant to the warfighter. m0 - mT (s / n. 5 ) = Za + Zb This example is for a one tailed case, that is, we are only interested in meeting or exceeding a value. The technique is similar for two tailed cases, except that the a and b values used are divided by 2, and there may be different thresholds for upper and lower limits. Showing the formulas for these cases is beyond the scope of the course. In this example, the variance is small compared to the mean values and the difference between the objective and the threshold values. This yields a small sample size and a high confidence levels. The usual case, would have a higher variance compared to the mean value, which would cause the required sample size to increase greatly. The equation on the left of the slide looks just like the equation we used for the central limit theorem as applied to the normal distribution, and for the t distribution. The denominator is the standard deviation of the sample mean, and the numerator is the physical distance between the values. The right hand side of the equation is simply the number of standard deviations the two physical measurements are apart.

Determining Za and Zb Za and Zb are the statistical distance (number of standard deviations from the mean) a = .05 b= .10 Assuming normal distribution, Za and Zb can be obtained from the normal distribution chart on the next slide. Za = Z.05 = 1.65 Zb = Z.10 = 1.28

Cumulative Normal Distribution Values of Zp Given P

JPATS Sample Size Calculations 2 = ( ) 15(1.65 +1.28) (20) m0 - mT (s / n. 5 ) n = 4.83 ~ 5 = Za + Zb This example is for a one tailed case, that is, we are only interested in meeting or exceeding a value. The technique is similar for two tailed cases, except that the a and b values used are divided by 2, and there may be different thresholds for upper and lower limits. Showing the formulas for these cases is beyond the scope of the course. In this example, the variance is small compared to the mean values and the difference between the objective and the threshold values. This yields a small sample size and a high confidence levels. The usual case, would have a higher variance compared to the mean value, which would cause the required sample size to increase greatly. The equation on the left of the slide looks just like the equation we used for the central limit theorem as applied to the normal distribution, and for the t distribution. The denominator is the standard deviation of the sample mean, and the numerator is the physical distance between the values. The right hand side of the equation is simply the number of standard deviations the two physical measurements are apart. A sample size of 5 will give the required protection against Type I and Type II errors. Z.05 = 1.65 Z.10 = 1.28

Distribution of Sample Averages 20 KTAS 5% Producers Risk 10% Consumers Risk The true average is an unknown number A sample size of 5 will allow the sample average to be within 20 KTAS of the true average, with a = .05 and b = .10 The math we used is -only- valid for normally distributed data.