1. Measurement Systems Analysis
If you can’t measure it, you can’t reliably improve it

2. Continuous Data, Stable Characteristics
Fully Crossed Designs, Youden Plot
Continuous Data, Stable Characteristics

3. Experimental Procedure: Continuous Data
30 units that span the range of variation are selected.
Each unit is tested twice by each operator or by each operator/gage combination. The test order must be random.
The appraiser/equipment repeatability is calculated/plotted
The appraiser reproducibility is calculated/plotted
The system bias is calculated/plotted.
If the test is extremely difficult, time consuming or expensive a smaller sample size may be selected but caution must be exercised as this can effect the reliability of the results.
If the gage is a go, no-go test or is a visual inspection, a different study design is necessary.

4. Operator Repeatability
Each operator’s individual repeatability is calculated first.
Estimates of reproducibility and bias are not reliable until repeatability is established.

5. The Total Measurement Repeatability, se
Once each operator has passed individual repeatability, a combined repeatability can be calculated.
e is calculated in the same way as with one operator. The average range is calculated from the individual ranges, Rij for each unit/operator set.
i denotes the unit #, typically i varies from 1 to 30 for 30 units.
j denotes the operator number
A is the first measurement of any unit by an operator
B is the second measurement of any unit by an operator
Rij = | XAij - XBij |

6. Reproducibility Error, sO
𝑋 𝑖𝑗 = 𝑋 𝐴𝑖𝑗 + 𝑋 𝐵𝑖𝑗 2
The averages for each unit:
𝑋 𝑗 = 𝑋 𝑖𝑗 𝑘
The product average for each operator:
The Range of Operator Product Averages
𝑅 𝑂 =𝑚𝑎𝑥 𝑋 𝑗 −𝑚𝑖𝑛 𝑋 𝑗
𝜎 𝑂 = 𝑅 𝑂 𝑑 − 𝜎 𝑒 2 𝑛 𝑂
The Reproducibility, o

7. Statistical Significance of Reproducibility
Since every operator will have some inherent differences from other operators, it is essential to determine if the differences are significant and important.
In order to do this a small reproducibility decision chart can be constructed. The product averages for each operator are plotted against the following decision limits:

8. Operator to Operator Reproducibility
The Center Line = Grand Average of all of the part readings
The Upper & Lower Decision Limits, UDL:
These decision limits are derived from the hypothesis that there is no difference between operators.
If any values fall outside the decision limits, that operator(s) is significantly different from the other operators.

9. Example of Operator Reproducibility
An example of a reproducibility decision chart with a significant difference between operators is:
Remember, a statistically significant difference can exist between operators but there may be no practical importance if the reproducibility is a very small portion of the total variation.

10. Repeatability & Reproducibility Statistics
𝜎 𝑅&𝑅 = 𝜎 𝑒 2 + 𝜎 0 2
Repeatability & Reproducibility, R&R:
The R&R Intraclass Correlation Coefficient, rR&R:
𝜌 𝑅&𝑅 =1− 𝜎 𝑒 2 + 𝜎 𝜎 𝑇 2 + 𝜎 0 2
The percentage of the total variation which is due to
product variation is rR&R x 100.
The percentage of the total variation that is due to
R&R variation is (1-rR&R) x 100.

11. The Discrimination Ratio
The Discrimination Ratio, DR R&R
𝐷 𝑅 𝑅&𝑅 = 1+ 𝜌 𝑅&𝑅 1− 𝜌 𝑅&𝑅

12. Caution on Reproducibility Calculations
System repeatability, se has meaning even when multiple operators are used.
Operator reproducibility, so and the Reproducibility R&R number has no real meaning.
Reproducibility errors occur only in conjunction with repeatability errors. So we only calculate a total R&R number…We need to use the “Control Chart” approach to determine if there is a significant and important difference between operators...

13. Reproducibility: More Fake Math
The use of 3 operators is essentially a fixed effect.
If there is a difference between the operators it will be systemic by definition.
Standard deviations can only be calculated for random variation.

14. Reproducibility Example
The first check is to ensure that the individual operator repeatability is acceptable.
The Youden plot at left shows all operators.
The operator repeatability is visually similar.

15. Operator to Operator “Stability”
A quick analysis of the consistency of the measurement error of each operator (Operator repeatability) is done using a Range Control Chart.
If a statistically significant & important difference exists it must be corrected before reproducibility can be assessed.
Western Electric, "Statistical Quality Control Handbook", 1956, 2nd Edition, 10th Printing, May 1984, Part B3, pp
Formulas for the Range Chart:
Center Line = 𝑅
𝐿𝐶𝐿 𝑅 = 𝐷 3 𝑅 (For n=2, D3 = 0)
𝑈𝐶𝐿 𝑅 = 𝐷 4 𝑅 (For n=2, D4 = 3.267)
A complete table for D3 and D4 values is in the SPC Manual as well as the Western Electric SQC Handbook.
There is no statistically significant difference in repeatability.

16. Operator to Operator “Stability”
It is possible to fix both repeatability and any systemic bias at the same time.
The mathematical rule is that any reproducibility number is meaningless unless the operator repeatability is consistent and acceptable.

17. Operator to Operator Reproducibility
An ANOM chart is used to determine the statistical significance of the difference between operators.
The mean of all measurements for each operator is plotted against decision limits. Points outside of the decision limits indicate statistically different operators.
Ott, Ellis R., “Analysis of Means – A Graphical Procedure”, Journal of Quality Technology, vol. 15, No. 1, January 1983.
“Analysis of Means” Prentice Hall, David Levine
While the differences are statistically significant, we must also assess the importance of the differences.

18. Individual Operator Graphs
Always look at the individual operator Youden Plots to confirm a practical difference.
R&R = 6.4% of the total observed variation
In this example, checking the individual operator Youden Plots, the statistically significant differences are not substantial and are not practically different.

19. Example: Supplier vs. IDEXX IQA
Groovy VetTest Tip
A CTQ is the distance from the ‘fins’ to the end of the tip.
3 supplier inspectors and 1 IQA inspector are involved in the initial R&R study.
30 parts are measured twice by each inspector.
Fin to Tip Length

20. Results: Youden Plot

21. Results: Measurement Error Stability
Total Supplier Measurement Error
In this case, the statistical test is not helpful: Visually it is obvious that the IQA inspector has much better measurement error than the 3 supplier inspectors. The Range chart is limited by the small sample size which drives a zero lower limit. The upper limit is driven by the 3 supplier inspectors.
IQA Measurement Error

22. Results: Reproducibility Bias
Reproducibility between IQA and the Supplier
The ANOM chart shows that the bias is statistically significant with IQA reading substantially lower than the supplier inspectors.
The statistical analysis of this bias is redundant to the visually obvious bias on the Youden plot.

23. “Plot your data and think about your data”
Ellis Ott

24. The System Performance is not Acceptable
A this point the MSA obviously fails. There is no need for further quantification.
The graph points us to the best clue to understanding what is wrong.
What is different between IQA and the supplier?

25. Fixing the System Observations were made of the two inspector groups.
The supplier was measuring from a knit line just above the fin edge with a visual comparator.
IQA used a fixture that located the fins on a flat surface to measure the length with a height gauge.
Knit line
Bottom of
Fin to Tip
to Tip
The knit line was difficult to pick up on the comparator, did vary positionally in relationship to the tip, and a comparator is notoriously difficult to use accurately on small dimensions.
How the tip functions

26. After Fixing the System: The Youden Plot
The DR values are low because the product variation is low. The tips only span about 30% of the tolerance range.
This is typical for injection molded parts or any part fabricated with a tool that wears.
If we extrapolate the measurement error across the tolerance we can see that the error is small relative to the tolerance and is acceptable for use.
The data don’t span the full range of the tolerance, so a graphical extension of the variation can be used to demonstrate what would happen over the range. For this type of injection molded part it is impossible to produce parts that span the range at the beginning of the tool life. If the measurement error were to increase with longer tips, the SPC chart would detect it.
NDC or Discrimination ratios will also be very small due to the lack of variation in parts. We shouldn’t try to manufacture parts that span the range of expected variation just to get a number. Logic and engineering expertise help us avoid this waste.
Examples of when this occurs
Tool wear
Lot to lot variation of raw materials built in very large batches
Component degradation (stability or shelf life that takes months or years)

27. After Fixing the System: Stability & Bias
The stability chart demonstrates that the repeatability of each operator is nearly identical.
The bias chart demonstrates that while there is a statistically significant bias between operators is very small: ” vs. a tolerance range of 0.006”.