1. DSQR Training Measurement System Analysis
HEATING, COOLING & WATER HEATING PRODUCTS
DSQR Training Measurement System Analysis
Fred Nunez
Corporate Quality

2. Goals At the end of this section you’ll be able to:
describe the terms: accuracy, repeatability, reproducibility, stability and resolution of a measurement system
conduct a Gage R&R study for continuous data

3. Cause & Effect Diagram for a Measurement System

4. Improving a Measurement System
A measurement system consists of
Measuring devices
Procedures
Definitions
People
To improve a measurement system, you need to
Evaluate how well it works now (by asking “how much of the variation we see in our data is due to the measurement system?”).
Evaluate the results and develop improvement strategies.

5. Properties of Measurement Systems
Repeated measurements will disagree
Means of repeated measurements will disagree
Measurements made at different times, or by different operators,
or on different instruments will disagree
The measured value and the true value will disagree
Waste due to poor quality test data
Rejection of “good” material
Acceptance of “bad” material
Adjusting the process when not needed
Failure to adjust when needed
Loss of “goodwill” between production and test people 5

6. Common Problems with Measurements
Problems with the measurements themselves: 1. Bias or inaccuracy: The measurements have a different average value than a “standard” method. 2. Imprecision: Repeated readings on the same material vary too much in relation to current process variation. 3. Not reproducible: The measurement process is different for different operators, or measuring devices or labs. This may be either a difference in bias or precision. 4. Unstable measurement system over time: Either the bias or the precision changes over time. 5. Lack of resolution: The measurement process cannot measure to precise enough units to capture current product variation.
Greater organizational issues
Wrong quality is measured.
Measurements are not made, distributed, or acted on in a timely manner.
What measurements?
When you look at surveys/questionnaires (e.g. customer satisfaction surveys, employee loyalty questionnaires etc.) the issues are:
Reliability of the measurements (i.e. does the same respondent give the same answers repeatedly?) – this relates to 2. Imprecision
Validity of measurements (i.e .the same respondent gives the same answers on different way of formulating a question) – this relates to 3. Reproducibility
Note: The broader organizational issues and issues from surveys are often the most troublesome.

7. Desired Measurement Characteristic for Continuous Variables
1. Accuracy The measured value has little deviation from the actual value. Accuracy is usually tested by comparing an average of repeated measurements to a known standard value for that unit.
2. Repeatability The same person taking a measurement on the same unit gets the same result.
Good
repeatability
if variation is
small *
Data from repeated
measurement of
same item

8. Desired Measurement Characteristic for Continuous Variables, cont.
3. Reproducibility Other people (or other instruments or labs) get the same average result you get when measuring the same item or characteristic.
* Small relative to
a) product variation and
b) product tolerance (the width of the product specifications)

9. Desired Measurement Characteristic for Continuous Variables, cont.
4. Stability Measurements taken by a single person in the same way vary little over time.
* Small relative to
a) product variation and
b) product tolerance (the width of the product specifications)

10. Desired Measurement Characteristic for Continuous Variables, cont.
5. Adequate Resolution There is enough resolution in the measurement device so that the product can have many different values.
Good if 5 or more distinct values are observed

11. Ways to See if the Measurement System is Adequate
Accuracy
Calibration and Gage Linearity Study (not covered here)
Repeatability
Gage R&R Study (covered next)
Reproducibility
Stability
Control Chart (covered in the module “Patterns in data”)
Adequate Resolution
With above tests
* Note there are also more sophisticated methods to compare results between laboratories

12. Gage R&R
The Gage R&R study is a set of trials conducted to assess the repeatability and reproducibility of the measurement system.
Multiple operators measure multiple units a multiple number of times.
Example: 3 operators each measure 10 units 3 times each.
“Blindness” is extremely desirable. It is better that the operator not know which of the test parts they are currently measuring.
You analyze the variation in the study results to determine how much of it comes from differences in the operators, techniques, or the units themselves.
Assessing the accuracy, repeatability, and reproducibility of a continuous measurement system
A Gage R&R study is used to assess the measurement system for collecting continuous data.
It is typically used in manufacturing or other applications where “gages” or devices are used to measure important physical characteristics that are continuous.
Examples: thickness, viscosity, strength, stickiness

13. How a Gage R&R Study Works
Select units or items for measuring that represent the full range of variation typically seen in the process.
Measurement systems are often more accurate in some parts of the range than in others, so you need to test them over the full range.
Have each operator measure those items repeatedly.
In order to use Minitab to analyze the results, each operator must measure each unit the same number of times.
It is extremely desirable to randomize the order of the units and not let the operator know which unit is being measured.
Minitab looks at the total variation in the items or units measured.
Minitab then estimates the proportion of the total variation that is due to
1. Part-to-part variation: physical or actual differences in the units being measured.
2. Repeatability: Inconsistency in how a given person takes the measurement (lots of inconsistency = high variation = low repeatability).
3. Reproducibility: Inconsistency in how different people take the measurement (lots of inconsistency = high variation = low reproducibility).
4. Operator–part interaction: An interaction that causes people to measure different items in different ways (e.g., people of a particular height may have trouble measuring certain parts because of lighting, perspective, etc.).
If there is excessive variation in repeatability or reproducibility (relative to part-to-part variation), you must take action to fix or improve the measurement process.
The goal is to develop a measurement system that is adequate for your needs.

14. Adequate vs. Inadequate Measurement Systems•
Most of the variation is accounted for by physical or actual differences between the units. •
Variation in how the measurements are taken is high. –
You can’t tell if differences between units are due to the way they were measured, or are true differences –
What Minitab calls part-to-part variation will be relatively large –
All other sources of variation will be small –
You can’t trust your data and therefore shouldn’t react to perceived patterns, special causes, etc.—they may be false signals –
You can have higher confidence that actions you take in response to data are based on reality •
The measurement system has sufficient precision to distinguish at least four groups or “categories” of measurements. •
The measurements fall into less than four categories.
Why four or more categories for an adequate system?
Four categories are a bare minimum to distinguish parts that are likely good, from those that are likely fair.
Fair
Good

15. Measurement System Indices
%R&R
Describes the variation of the measurement system in comparison to the part variation of the process
%P/T or PTR
Describes the variation of the measurement system in comparison to the part tolerances 6
%R&R
Puts the measured standard deviation due to repeatability and reproducibility problems in the measurement system in relation to the overall measured variation (part to part + measurements system variation)
%P/T
Describes how much of the tolerances are “occupied” by the measurement system variation (6 s around the average represent 99.7% of the data in a normal distributed process)
See the following pages for a description how to calculate the measurement system variation.
General guidelines for interpreting Gage R&R results.
Unacceptable
Desired
Acceptable
Borderline 0%
10%
20%
30%
100%

16. Three different parts: 1, 2, & 3
The Impact of GR&R on Specification Limits
Three different parts: 1, 2, & 3
Each part is represented by a distribution representing known measurement variability, rather than an interval.
For judging conformance to spec, the GR&R will not be a factor for parts 1 & 3.
As in the previous example, the test result for sample #2 presents a distinct probability that the part may actually be out-of-specification, even though the test result is within spec.
Whenever a single test result falls directly on a spec limit, there is a 50:50 chance that the part may be in-spec or out-of-spec. This is true regardless of how “good” the GR&R is.
For these reasons, it is always best for the process average to be at or near the target value. 16

17. Data for a Gage R&R Study
Each operator measures each unit repeatedly.
Data must be balanced for Minitab—each operator must measure each unit the same number of times.
The units should represent the range of variation in the process.
Operators should randomly and “blindly” test the units; they should not know which unit they are measuring when they record their results.
Unit Number
Operator
Measurement 1
Joe
11.34 4
13.27
11.29
13.28
11.33
13.24
11.24
13.23
Sally
11.19
13.09
13.14
11.21
13.02
13.19 2
11.65 5
11.84
11.60
11.89
11.67
11.93
11.56
11.85
11.50
11.76
11.55
11.51
11.81
11.78 3
12.31
12.28
12.34
12.18
12.23
12.14
12.17
Here, two people each measured 5 units 4 times.
Part tolerances are 12+/- 2.

18. Plotting the Data From a Gage R&R Study
Use the following Minitab command to plot the data.
Open the worksheet: C:\SixSigma\Data\gagedemo.mtw
Stat > Quality Tools > Gage Study > Gage Run Chart…
Go to Minitab GRR Handout
Unit Number
Operator
Measurement 1
Joe
11.34 4
13.27
11.29
13.28
11.33
13.24
11.24
13.23
Sally
11.19
13.09
13.14
11.21
13.02
13.19 2
11.65 5
11.84
11.60
11.89
11.67
11.93
11.56
11.85
11.50
11.76
11.55
11.51
11.81
11.78 3
12.31
12.28
12.34
12.18
12.23
12.14
12.17
Here, two people each measured 5 units 4 times.
Part tolerances are 12+/- 2.

19. Plotting the Data From a Gage R&R Study
Use the following Minitab command to plot the data.
Open the worksheet: C:\SixSigma\Data\gagedemo.mtw
Stat > Quality Tools > Gage Study > Gage Run Chart…
Use the following Minitab command to plot the data.
Open the worksheet: C:\SixSigma\Data\gagedemo.mtw
Stat > Quality Tools > Gage Study > Gage Run Chart…

20. Using Gage Run Chart Using Gage Run Chart
Data columns from the worksheet
Data columns from the worksheet
You fill in this information
You fill in this information

21. Output From Gage Run Chart
Gage name:
Gage name:
Opacity Meter 3
Opacity Meter 3
Run chart of Measurement by Unit Number,
Run chart of Measurement by Unit Number,
Date of study:
Date of study:
8/19/98
8/19/98
Operator
Operator
Reported by:
Reported by:
GRR
GRR
Tolerance:
Tolerance:
Misc:
Misc:
Joe
Joe 13 13
Sally
Sally
Very little difference is seen in repeated measurements by the same person or between people
Very little difference is seen in repeated measurements by the same person or between people
Measurement
Measurement 12 12 11 11
Unit Number
Unit Number 1 1 2 2 3 3 4 4 5 5
Each dot represents a single measurement by a single person.
Repeated measurements by a single person are connected with a line.
To analyze the chart, look at the spread of points within a series connected by lines (repeatability) and the difference between data from different operators (reproducibility).
Here we see each unit (part number) has a tight spread of measurements with only a small operator effect.
Each dot represents a single measurement by a single person.
Repeated measurements by a single person are connected with a line.
To analyze the chart, look at the spread of points within a series connected by lines (repeatability) and the difference between data from different operators (reproducibility).
Here we see each unit (part number) has a tight spread of measurements with only a small operator effect.

22. Performing a Gage R&R Analysis
Use the following Minitab command to conduct a detailed Gage R&R analysis.
Stat > Quality Tools > Gage Study > Gage R&R Study (Crossed)…
Use the following Minitab command to conduct a detailed Gage R&R analysis.
Stat > Quality Tools > Gage Study > Gage R&R Study (Crossed)…

23. Using Gage R&R Study Using Gage R&R Study
Use the ANOVA method since it allows for more precision and a look at operator–part interactions
Use the ANOVA method since it allows for more precision and a look at operator–part interactions
Fill in “4” for Upper/Lower specs (the process specs are 12+/-2)
Fill in “4” for Upper Spec - Lower Spec (the process specs are 12+/-2)

24. Interpreting the Gage R&R Output From Minitab
Total Gage R&R plus Part-To-Part equals 100%
Interpreting the Gage R&R Output From Minitab
Gage R&R
%Contribution
Source VarComp (of VarComp)
Total Gage R&R
Repeatability
Reproducibility
Operator
Part-To-Part
Total Variation
StdDev Study Var %Study Var % Tolerance
Source (SD) (6*SD) (%SV) (SV/Toler)
Total Gage R&R
Repeatability
Reproducibility
Operator
Part-To-Part
Total Variation
Number of Distinct Categories = 12
Total Gage R&R plus Part-To-Part equals 100%
Should give the width of 99.73% confidence interval
A2+A3 = A1 A2 A3 A4 A5 A6
Gage R&R
%Contribution
Source VarComp (of VarComp)
Total Gage R&R
Repeatability
Reproducibility
Operator
Part-To-Part
Total Variation
StdDev Study Var %Study Var % Tolerance
Source (SD) (6*SD) (%SV) (SV/Toler)
Total Gage R&R
Repeatability
Reproducibility
Operator
Part-To-Part
Total Variation
Number of Distinct Categories = 12
Will not sum to 100
= 6 x st. dev.
A2+A3 =
Should give the width of 99.73% confidence interval
A2+A3 +A5= A1 A2 A3 A4 A5 A6
Will not sum to 100
= 6 x st. dev.
A2+A3 +A5=
These will match when there is no operator part interaction
%R&R
%P/T
Should be 4 or more for a healthy measurement system
%R&R
%P/T
These will match when there is no operator part interaction
Should be 4 or more for a healthy measurement system

25. Interpreting Gage R&R: Minitab’s Graphical Output
Conclusions
This is an adequate measurement system for the range of units tested.
B: No significant operator–part interaction
C: Measurement system can distinguish more than 4 categories
D: No special causes within an operator for the same part, as evidenced by the R- chart
If further measurement system refinement is desired:
A: Investigate the significant operator differences
Difference in values is part-to-part variation
Difference in values is part-to-part variation
Should be small relative to part-to-part if measurement system is good
Should be small relative to part-to-part if measurement system is good
Difference in horizontal alignment is operator effect
Difference in horizontal alignment is operator effect
Should always be in control, meaning the operators consistently measure the same way. If not, identify special cause.
Should always be in control, meaning the operators consistently measure the same way. If not, identify special cause.
For good measurement systems, this will be out of control, indicating the parts are more variable than the measurement system (which is good)
For good measurement systems, this will be out of control, indicating the parts are more variable than the measurement system (which is good)
Parallel lines if no interaction
Parallel lines if no interaction
Conclusions
This is an adequate measurement system for the range of units tested.
B: No significant operator–part interaction
C: Measurement system can distinguish more than 4 categories
D: No special causes within an operator for the same part, as evidenced by the R-chart
If further measurement system refinement is desired:
A: Investigate the significant operator differences

26. Using the Gage R&R ANOVA Method
Focus on the following: A) %P/T If > 30%, this indicates a poor measurement system to capture the actual process variation. B) Number of distinct categories If the number is < 4 it implies that the measurement variation is too large to adequately distinguish the part to part variation. C) The R chart by operator If it is stable, this tells that there are no special causes in the measurement process that could be throwing off our calculations .

27. Determining the “Tolerance”
Case 1 - Bilateral Specs:
Upper Spec Limit (USL) and Lower Spec Limit (LSL) available
Tolerance = USL minus LSL
Case 2A - Unilateral Specs
Target = 0; Have USL:
Tolerance = USL minus Zero
Case 2B - Unilateral Specs
Have USL or LSL but not both:
Tolerance = 2 * | Xbar – Spec Limit |
Case 3 – No Specs, but have production data:
Don’t use %P/T
Use only %GRR

28. Procedure for a GR&R Study
Select the measuring equipment to be studied.
Select three people for the study. Engineers and/or QA Technicians can be utilized to do preliminary studies to determine if gages, fixtures, and training are required to meet measurement requirements. Operators that will be doing the testing in production must be used for the final Gage R&R.
IMPORTANT: Prepare a stepwise procedure to be used for the study. Illustrate the procedure with photos or sketches if necessary to make it clear how to properly perform the testing. If this testing, or the gage, is new to them, it would be advisable to conduct training, then allow them to practice using the gage and procedure so that they can develop their technique and familiarity with the method. Failure to do so may result in an unacceptable %GR&R.
Select 10 parts to be measured. Number the parts 1 to 10. (See NOTE)
Allow each operator to measure each part in random order.
Repeat step 5 a second time. Note: the operator should not have access to previous results.
Repeat step 5 a third time. Note: the operator should not have access to previous results.
Analyze the data using Minitab or similar software.

29. Procedure for a GR&R Study (Continued)
Note 1: All 10 parts need not have the same part number if the target values are the same, or very similar in value. For example, we currently produce four different part numbers for skirt blank, but all four have the same 9.15” height dimension. If you cannot assemble the required 10 parts of the same part number, you could substitute different skirt blank parts to meet the requirement. Note 2: When selecting parts for a GR&R where multiple part numbers are available, e.g. 14”, 16”, 18” and 20” parts, where each has a different target dimension, select the part number with the dimension that will be the most difficult to measure accurately and precisely. For example, for the Skirt Blank, each part number has a different length. Select the part number with the longest length because this will generally be the most difficult one to measure. Note 3: When possible, select the parts that represent the actual variation of the process. Ideally, you would like the standard deviation of these 10 parts to be the same as the standard deviation of the process. Failure to select parts that represent the actual process variation will result in Gage R&R % of Study Variation values that may be of little use, with only the GR&R % of Tolerance value being useful.

30. 30

31. EXAMPLE 1 31

32. EXAMPLE 1
Minitab will also show the “Number of Distinct Categories”. This is the number of distinct categories of parts that the measurement process is currently able to distinguish. The lower the %GR&R, the higher this number will be. Ideally you should have 5 or more distinct categories. This example had only 2 distinct categories, but only because the variability of the 10 parts was small when compared to the allowable specifications.
If the GR&R % of Tolerance is acceptable and GR&R % of Study Variation is too high, it means that your parts are too uniform to use the % of Study Variation as a reliable measure of gage capability. 32

33. Example #2 (%P/T = 12.3%) 33

34. Example #3 (One Distinct Category, %P/T = 114%)
Handling Interaction
If the ANOVA table shows a statistically significant interaction, it may or may not be real. If you cannot get it to repeat, conclude that it is probably not a true interaction. 34