Gage Repeatability and Reproducibility (R&R) Studies An Introduction to Measurement System Analysis (MSA) Introduction This course is designed to give you a thorough explanation of gage studies, which are a specialized type of measurement system analysis. Gage R&R is a key tool within the Six Sigma methodology. This class is best taught using your company’s specific examples, allowing for interaction among the students Feel free to modify and change this course any way you like. Simply click View  Slide Master to edit the master slide so that all slides show your company logo and name. You may also teach it exactly as it is. We recommend about 4 hours to teach the course, but time may vary depending on your experience, class exercises, and knowledge of the class. This course is intended to teach an individual or group with basic math skills and little to no prior knowledge of the content. We do cover content related to control charts and capability analysis, which is available in course format from our website. We recommend that someone with prior experience and training with Gage R&Rs teach the course (usually Black Belt or Master Black Belt). If not being used as a course, valuable information can still be obtained by following through the presentation and reading the notes section. If you have any questions, please visit or contact us at http://www.biz-pi.com The primary reference for this course came from the AIAG manual for Measurement Systems Analysis: http://www.aiag.org/source/Orders/prodDetail.cfm?productDetail=MSA-4

Agenda Importance of data What is MSA? Measurement Error Sources of Variation Precision (Resolution, Repeatability, Reproducibility) Accuracy (Bias, Stability, Linearity) What is Gage R&R? Variable vs Binary Data Variable Gage R&R Criteria for % of Tolerance Type I and II error Attribute Agreement Analysis Criteria for Kappa Key Points More Resources Here is the agenda for the class You might want to assign times to it for your own benefit, based on your review of this presentation and what detail you plan to cover.

Real life MSA – Mortgage Loan Approval based on: Credit score Rental payment history Previous home ownership Job status and length Income to debt ratio Type of home Familiarity with applicant and their references MSA isn’t only used in manufacturing or technical industries (although it is most common in these environments) You can also experience measurement variation in real life. If you are applying for a mortgage loan, there is a decision being made about your ability to pay the loan, based upon numerous criteria. These criteria include credit score, rental payment history, previous home ownership, job status, income to debt ratio, type of home, familiarity with applicant, references, etc. You would hope that the process to determine if you are approved or not has been clearly defined and is accurate (MSA is good). If they have a perfect measurement process, then no matter who reviews your application, the same decision will be made. Otherwise, depending on when you submit your application, you might get different outcomes. Was someone at the bank in a bad mood? Did they discriminate based on information in your loan? Did the credit score show erroneous information that others ignored? Did they have access to all the relevant information? All of these questions might influence the final decision, which means their process has too much measurement error. Mortgage loans contain some data, but like many processes, there isn’t a single numeric outcome. It might be made up of different numbers, that all roll up into a decision that is reliant on a person, and may or may not be numeric. Even processes that have numeric scores often get reviewed by people, who can decide on how much they want to rely on the numbers. We will talk more about MSA applied to processes that do not have numeric outcomes later on in the course. What other examples does the class have to share, where they have experienced measurement error in real life or at work? Examples: Credit score Annual performance review at work Weight scales at work, gym, home Restaurant bills (same items priced differently) Speeding tickets from police (your speed vs radar/laser gun) Applying for job (resume review) Judging friends and co-workers competency Physical check-up at doctor’s office

Example: Reproducibility COMPARE AVERAGES OF SAME PART TO EACH OTHER REPRODUCIBLE NOT REPRODUCIBLE PERSON #1 0.0046 0.0057 0.0032 0.0039 0.0050 0.0030 0.0036 0.0056 AVERAGE 0.0043 PERSON #2 0.0048 0.0050 0.0034 0.0051 0.0037 0.0032 0.0046 0.0044 AVERAGE 0.0043 PERSON #1 0.0043 0.0052 0.0031 0.0033 0.0045 0.0034 0.0039 0.0052 AVERAGE 0.0041 PERSON #2 0.0034 0.0022 0.0021 0.0023 0.0035 0.0024 0.0029 0.0047 AVERAGE 0.0029 In this example, we have two people measure the same part over and over again. You can see on the right that the average reading for person 1 compared to person 2 differs from each other (0.0041 compared to 0.0029), which causes reproducibility variation. On the left, their average readings are the same, even though individual readings do not match exactly. Reproducibility is a comparison of the averages, not the individual readings. However, if the individual readings differ, then it will be less likely that the overall averages will be similar. GOOD BAD

Measurement Error Measured Value Actual Measurement Measurement Error Precision Accuracy Let’s go back to the diagram we showed earlier. Hopefully it makes more sense, now that you understand the different terms. Repeatability Reproducibility Stability Bias Linearity Resolution

Variable Gage R&R Example OPERATOR #1 OPERATOR #2 OPERATOR #3 48 total measurements 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 TRIAL 1 TRIAL 2 Let’s take a look at this study visually. Let’s also assume that we are using measurement (variable) data to assess the study (not attribute data). At the top, we have 3 different operators. Each operator will take measurements on all 8 parts, and they will repeat the measurement once (2 trials). Therefore, each operator will take 16 total measurements. The total study will be 48 measurements. Each measurement will be gathered randomly as much as possible, to remove the chance that people could remember their answers, or get in a routine that is not typical of the measurement process. Let’s look at how the calculations will be setup to assess repeatability and reproducibility in the study. 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

Measured Value = 5.5 (PASS) Type II Error MEASURING A PASS WHEN SHOULD HAVE FAILED – CONSUMER RISK LSL = 5 USL = 10 Measured Value = 5.5 (PASS) Actual Value = 4.9 (FAIL) Here is a visual picture of Type II error. Let’s assume we have an actual value is 4.9, which is a failing result (below 5). If we have a process with 33% of tolerance in the measurement process, then we have some probability that a measurement from the process would be above 5, which would result in a passing outcome (which would be an incorrect decision). This is also called consumer risk. Instead of rejecting the item and throwing it away, reworking it, rechecking or re-measuring it, we send it to the customer, who then has to deal with the problem when it does meet their needs. This is the worst case, as we have passed our problems to our customers, and hurt our reputation. Let’s look at Type I and II errors in another way… X X 4 5 6 7 8 9 10 11 33% measurement error

Measurement Error vs Uncertainty LSL USL LSL USL Gage R&R% of Tolerance 1% Gage R&R% of Tolerance 10% LSL USL LSL USL As you can see, when the Gage R&R % of tolerance increases, the uncertainty area gets wider. This increases the risk of Type I and Type II errors. This also shows how the variation in the measurements (green curve) can increase or decrease this risk of these false decisions. If the process is more “capable” (green curve is more narrow), then the probability of data points falling within the uncertainty area would be less. This is why measurement error cannot be evaluated completely on its own, we need to understand where the actual measurements are falling in relation to the limits. The closeness of the measurements to the limits is called Capability. Gage R&R% of Tolerance 30% Gage R&R% of Tolerance 80%

Attribute Agreement Analysis The overall results show that 38% of the operators agree with each other completely (3 out of 8, obtained from the last column). Next, we need to determine if this % of agreement is acceptable or not. 3 out of 8 assessments completely agree = 38%

Calculating Kappa for Coin Flips Ex: Flipping a coin 100 times 45 heads and 55 tails Po = 45/100 = .45 Pe = 50/100 = .50 Kappa = (0.45 – 0.50) / (1 – 0.5) Kappa = -0.05 / 0.5 = -0.1 Po – Pe 1 – Pe The formula for calculating kappa is the difference between the probability of occurrence and the expected probability, divided by 1 minus the expected probability. If we flip a coin 100 times, we would expect to get 50 heads and 50 tails. However, we won’t get that exactly due to random chance. Let’s assume we flip the coin 100 times, and get 45 heads. Therefore, our kappa would use Po = 45/100 = 0.45, and Pe = 50/100 = 0.50 Solving the formula, we get -0.1 for kappa. This is close enough to zero, so we assume that what we observed is close enough to chance, which meets our expectations. If our observed percentage was 90 heads (0.90), then you can walk the class through the exercise on what that would compute for kappa. (0.90 – 0.50) / (1-0.50) = 0.40 / 0.50 = 0.8, which would get us closer to a kappa value that differs significantly from chance. You would conclude that the coin or the flipper of the coin does not match random chance. This would lead us to believe that the flipper had a special technique to get the same result, or the coin has been modified to land on one side more often than 50% by chance. Kappa is near zero, so it matches close enough to our expectations

Key Points (cont’d) Clearly mark or control items, but don’t make markings visible to operators (blind study) Don’t let operators watch each other, so true behavior can be captured (depends on purpose of study) Use typical items seen in the process Measure to as finite a number as possible. Do not round Make detailed observations as the parts are being measured Treat each measurement as a new item (full setup and break down each time) Here are some more points about Gage R&Rs. Clearly mark the items, so you know which ones are which, but also so that the operator does NOT know which one they are measuring. We are trying to prevent them from memorizing the answers for future measurements, which will not simulate real-life conditions. The operators should think that each part is unique (part number is hidden), even though some of the measurements are repeats of the same part. In addition, don’t let the operators watch each other perform measurements, as they may be influenced by techniques, and change how they were planning to take the measurement. Don’t round any measurements, write down or capture as many decimal points as possible. Record detailed observations when items are being measured. This will help determine what inconsistencies exist, and what improvements need to be made. Treat each measurement as a new item. If the item requires any setup work (load software, calibrate equipment, change fixturing, documentation, etc) it should all be included in each measurement. Do not just re-measure the item, they need to treat each item like it is a brand new item.

Contact For more training materials and resources Capability Analysis Control Charts Lean Six Sigma Overview Root Cause Analysis 5S Cost of Poor Quality Templates and Diagrams Visit Business Performance Improvement at: http://biz-pi.com/store.asp As mentioned in the course, we have other training courses you might be interested in, such as: Capability Analysis Control Charts Lean Six Sigma Overview Root Cause Analysis 5S Cost of Poor Quality Templates and Diagrams Contact us at our website at Biz-PI.com

Business Performance Improvement http://www.biz-pi.com Additional Resources Business Performance Improvement http://www.biz-pi.com Contact us at http://www.biz-pi.com if you have any questions about this course, or would like some information about our training and/or consulting options