Measure Phase Measurement System Analysis Now we will continue in the Measure Phase with “Measurements System Analysis”.

Measurement System Analysis Basics of MSA Variables MSA Attribute MSA Wrap Up & Action Items Process Capability Measurement System Analysis Six Sigma Statistics Process Discovery Welcome to Measure Measurement System Analysis is one of those non-negotiable items! MSA is applicable in 98% of projects and it alone can have a massive effect on the success of your project and improvements within the company. In other words, LEARN IT & DO IT. It is very important.

Measurement System Analysis Introduction to MSA So far we have learned that the heart and soul of Six Sigma is that it is a data-driven methodology. How do you know that the data you have used is accurate and precise? How do know if a measurement is a repeatable and reproducible? How good are these? Measurement System Analysis or MSA In order to improve your processes, it is necessary to collect data on the "critical to" characteristics. When there is variation in this data, it can either be attributed to the characteristic that is being measured and to the way that measurements are being taken; which is known as measurement error. When there is a large measurement error, it affects the data and may lead to inaccurate decision-making. Measurement error is defined as the effect of all sources of measurement variability that cause an observed value (measured value) to deviate from the true value. The measurement system is the complete process used to obtain measurements, such as the procedures, gages and personnel that are employed to obtain measurements. Each component of this system represents a potential source of error. It is important to identify the amount of error and, if necessary, the sources of error. This can only be done by evaluating the measurement system with statistical tools. There are several types of measurement error which affect the location and the spread of the distribution. Accuracy, linearity and stability affect location (the average). Measurement accuracy describes the difference between the observed average and the true average based on a master reference value for the measurements. A linearity problem describes a change in accuracy through the expected operating range of the measuring instrument. A stability problem suggests that there is a lack of consistency in the measurement over time. Precision is the variability in the measured value and is quantified like all variation by using the standard deviation of the distribution of measurements. For estimating accuracy and precision, multiple measurements of one single characteristic must be taken. The primary contributors to measurement system error are repeatability and reproducibility. Repeatability is the variation in measurements obtained by one individual measuring the same characteristic on the same item with the same measuring instrument. Reproducibility refers to the variation in the average of measurements of an identical characteristic taken by different individuals using the same instrument. Given that reproducibility and repeatability are important types of error, they are the object of a specific study called a Gage Repeatability & Reproducibility study (Gage R&R). This study can be performed on either attribute-based or variable-based measurement systems. It enables an evaluation of the consistency in measurements among individuals after having at least two individuals measure several parts at random on a few trials. If there are inconsistencies, then the measurement system must be improved.

Measurement System Analysis MSA is a mathematical procedure to quantify variation introduced to a process or product by the act of measuring. Measurement Process Environment Measurement Item to be Measured Reference Equipment Procedure Operator The item to be measured can be a physical part, document or a scenario for customer service. Operator can refer to a person or can be different instruments measuring the same products. Reference is a standard that is used to calibrate the equipment. Procedure is the method used to perform the test. Equipment is the device used to measure the product. Environment is the surroundings where the measures are performed. Measurement System Analysis is the entire system, NOT just calibration or how good the measurement instrument is. We must evaluate the entire environment and Measurement System Analysis gives us a way to evaluate the measurement environment mathematically. All these sources of variation combine to yield a measurement that is different than the true value. It is also referred to as “Gage R&R” studies where R&R is: Repeatability & Reproducibility

What do I need to know? Measurement Purpose In order to be worth collecting, measurements must provide value - that is, they must provide us with information and ultimately, knowledge The question… What do I need to know? …must be answered before we begin to consider issues of measurements, metrics, statistics, or data collection systems Too often, organizations build complex data collection and information management systems without truly understanding how the data collected and metrics calculated actually benefit the organization. Measurement is a process within itself. In order to measure something you must go through a series of tasks and activities in sequence. Usually there is some from of set-up, there is an instrument that makes the measurement, there is a way of recording the value and it may be done by multiple people. Even when you are making a judgment call about something, there is some form of setup. You become the instrument and the result of a decision is recorded someway; even if it is verbal or it is a set of actions that you take. The types and sophistication of measurement vary almost infinitely. It is becoming increasingly popular or cost effective to have computerized measurement systems. The quality of measurements also varies significantly - with those taken by computer tending to be the best. In some cases the quality of measurement is so bad that you would be just as well off to guess at what the outcome should be. You will be primarily concerned with the accuracy, precision and reproducibility of measurements to determine the usability of the data.

The purpose of MSA is to assess the error due to measurement systems. The error can be partitioned into specific sources: Precision Repeatability - within an operator or piece of equipment Reproducibility - operator to operator or attribute gage to attribute gage Accuracy Stability - accuracy over time Linearity- accuracy throughout the measurement range Resolution Bias – Off-set from true value Constant Bias Variable Bias – typically seen with electronic equipment, amount of Bias changes with setting levels The purpose of conducting an MSA is to mathematically partition sources of variation within the measurement system itself. This allows us to create an action plan to reduce the biggest contributors of measurement error.

Accuracy and Precision Accurate but not precise - On average, the shots are in the center of the target but there is a lot of variability Precise but not accurate - The average is not on the center, but the variability is small Measurement systems, like all things, generate some amount of variation in the results/data they output. In measuring, we are primarily concerned with 3 characteristics: How accurate is the measurement? For a repeated measurement, where is the average compared to some known standard?. Think of the target as the measurement system, the known standard is the bulls eye in the center of the target. In the first example below you see the “measurements” are very dispersed, there is a lot of variability as indicated by the Histogram curve at the bottom. But on average, the “measurements” on target. When the average is on target, we say the measurement is accurate. However in this example they are not very precise. How precise is the measurement? For a repeated measurement, how much variability exists? As seen in the first target example, the “measurements” are not very precise, but on the second target they have much less dispersion, there is less variability as seen in the histogram curve. However, we notice that the tight cluster of “measurements” are off target, they are not very accurate. The third characteristic is how reproducible is the measurement from individual to another? What is the accuracy and precision from person to person. Here you would expect each person that performs the measurement to be able to reproduce the same amount of accuracy and precision as that of other persons performing the same measurement. Ultimately, we make decisions based on data collected from measurement systems. If the measurement system does not generate accurate or precise enough data, we will make the decisions that generates errors, waste and cost. When solving a problem or optimizing a process, we must know how good our data are, and the only way to do this is to perform a Measurement System Analysis.

MSA Uses MSA can be used to: Compare internal inspection standards with the standards of your customer. Highlight areas where calibration training is required. Provide a method to evaluate inspector training effectiveness as well as serves as an excellent training tool. Provide a great way to: Compare existing measurement equipment. Qualify new inspection equipment. The measurement system always has some amount of variation and that variation is additive to the actual amount of true variation that exists in what we are measuring. The only exception is when the discrimination of the measurement system is so poor, that it virtually sees everything the same. This means that you may actually be producing a better product or service than you think you are, provide that the measurement system is accurate meaning it does not have a bias, linearity or stability problem. It may also mean that your customer may be making the wrong interpretations about your product or service. The components of variation are statistically additive. The primary contributors to measurement system error are repeatability and reproducibility. Repeatability is the variation in measurements obtained by one individual measuring the same characteristic on the same item with the same measuring instrument. Reproducibility refers to the variation in the average of measurements of an identical characteristic taken by different individuals using the same instrument.

Measurement System Analysis is important to: Why MSA? Measurement System Analysis is important to: Study the % of variation in our process that is caused by our measurement system. Compare measurements between operators. Compare measurements between two (or more) measurement devices. Provide criteria to accept new measurement systems (consider new equipment). Evaluate a suspect gage. Evaluate a gage before and after repair. Determine true process variation. Evaluate effectiveness of training program. Why is MSA so important? MSA allows us to trust the data generated from our processes. When you charter a project you are taking on a significant burden which will require statistical analysis. What happens if you have a great project, with lots of data from measurement systems that produce data with no integrity?

Appropriate Measures are: Sufficient – available to be measured regularly Relevant –help to understand/isolate the problems Representative - of the process across shifts and people Contextual – collected with other relevant information that might explain process variability. Sufficient, means that measures are available to be measured regularly, if not it would take too long to gather data. Relevant, means that they will help to understand and isolate the problems. Representative measures mean that we can detect variation across shifts and people. Contextual means they are necessary to gather information on other relevant information that actually would help to explain sources of variation.

Poor Measures can result from: Poor or non-existent operational definitions Difficult measures Poor sampling Lack of understanding of the definitions Inaccurate, insufficient or non-calibrated measurement devices Measurement Error compromises decisions that affect: Customers Producers Suppliers It is very common while working projects to discover that the current measurement systems are poor. Have you ever come across a situation where the data from your customer or supplier doesn’t match yours? It happens often. It is likely a problem with one of the measurement systems. We have worked MSA projects across critical measurement points in various companies, it’s not uncommon for more than 80% of the measurements to fail in one way or another.

Examples of What to Measure Examples of what and when to measure: Primary and secondary metrics Decision points in Process Maps Any and all gauges, measurement devices, instruments, etc “X’s” in the process Prior to Hypothesis Testing Prior to modeling Prior to planning designed experiments Before and after process changes To qualify operators At this point you should have a fairly good idea of what to measure, listed here are some ideas to get you thinking… MSA is a Show Stopper!!!

Components of Variation Whenever you measure anything, the variation that you observe can be segmented into the following components… All measurement systems have error. If you don’t know how much of the variation you observe is contributed by your measurement system, you cannot make confident decisions. If you were one speeding ticket away from losing your license, how fast would you be willing to drive in a school zone? Accuracy Precision Repeatability Reproducibility Measurement System Error Unit-to-unit (true) Variation Observed Variation Stability Bias Linearity We are going to strive to have the measured variation be as close as possible to the true variation. In any case we want the variation from the measurement system to be a small as possible. We are now going to investigate the various components of variation of measurements.

Repeatability and Reproducibility = Gage R+R Precision A precise metric is one that returns the same value of a given attribute every time an estimate is made. Precise data are independent of who estimates them or when the estimate is made. Precision can be partitioned into two components: Repeatability Reproducibility The spread of the data is measured by Precision. This tells us how well a measure can be repeated and reproduced. Repeatability and Reproducibility = Gage R+R

Repeatability Repeatability is the variation in measurements obtained with one measurement instrument used several times by one appraiser while measuring the identical characteristic on the same part. For example: Manufacturing: One person measures the purity of multiple samples of the same vial and gets different purity measures. Transactional: One person evaluates a contract multiple times (over a period of time) and makes different determinations of errors. Y Repeatability Measurements will be different…expect it! If measurement are always exactly the same this is a flag, sometimes it is because the gauge does not have the proper resolution, meaning the scale doesn’t go down far enough to get any variation in the measurement. For example, would you use a football field to measure the gap in a spark plug?

Reproducibility Reproducibility is the variation in the average of the measurements made by different appraisers using the same measuring instrument when measuring the identical characteristic on the same part. For example: Manufacturing: Different people perform purity test on samples from the same vial and get different results. Transactional: Different people evaluate the same contract and make different determinations. Reproducibility Operator A Operator B Y Reproducibility will be present when it is possible to have more than one operator or more than one instrument measure the same part.

Time Estimate Exercise Exercise objective: Demonstrate how well you can estimate a 10 second time interval. Pair up with an associate. One person will say start and stop to indicate how long they think the 10 seconds last. Do this 6 times. The other person will have a watch with a second hand to actually measure the duration of the estimate. Record the value where your partner can’t see it. Switch tasks with partner and do it 6 times also. Record all estimates, what do you notice? Exercise.

Accuracy can be assessed in several ways: An accurate measurement is the difference between the observed average of the measurement and a reference value. When a metric or measurement system consistently over or under estimates the value of an attribute, it is said to be “inaccurate” Accuracy can be assessed in several ways: Measurement of a known standard Comparison with another known measurement method Prediction of a theoretical value What happens if we don’t have standards, comparisons or theories? True Average Accuracy Measurement Warning, do not assume your metrology reference is gospel. Accuracy and the average are related. Recall in the Six Sigma Statistics module we talked about the Mean and the variance of a distribution. Think of it this way….If the Measurement System is the distribution then accuracy is the Mean and the precision is the variance.

Accuracy Against a Known Standard In transactional processes, the measurement system can consist of a database query. For example, you may be interested in measuring product returns where you will want to analyze the details of the returns over some time period. The query will provide you all the transaction details. However, before you invest a lot of time analyzing the data, you must ensure the data has integrity. The analysis should include a comparison with known reference points. For the example of product returns, the transaction details should add up to the same number that appears on financial reports, such as the income statement. Please read the slide.

Accuracy vs. Precision ACCURATE PRECISE BOTH + = Accuracy relates to how close the average of the shots are to the Master or bull's-eye. Precision relates to the spread of the shots or Variance. Most Measurement Systems are accurate but not at all precise. NEITHER

Bias Bias is defined as the deviation of the measured value from the actual value. Calibration procedures can minimize and control bias within acceptable limits. Ideally, Bias can never be eliminated due to material wear and tear! Bias Bias is a component of Accuracy. Constant Bias is when the measurement is off by a constant value. A scale is a prefect example, if the scale reads 3 lbs when there is no weight on it then there is a 3lb Bias. Make sense?

Stability is Bias characterized as a function of time! Stability of a gauge is defined as error (measured in terms of Standard Deviation) as a function of time. Environmental conditions such as cleanliness, noise, vibration, lighting, chemical, wear and tear or other factors usually influence gauge instability. Ideally, gauges can be maintained to give a high degree of Stability but can never be eliminated unlike Reproducibility. Gage Stability studies would be the first exercise past calibration procedures. Control Charts are commonly used to track the Stability of a measurement system over time. Drift Stability is Bias characterized as a function of time! Stability just looks for changes in the accuracy or Bias over time.

Linearity Linearity is defined as the difference in Bias values throughout the measurement range in which the gauge is intended to be used. This tells you how accurate your measurements are through the expected range of the measurements. It answers the question, "Does my gage have the same accuracy for all sizes of objects being measured?" Linearity = |Slope| * Process Variation % Linearity = |Slope| * 100 Nominal High Low * Reference Value (x) B i a s (y) 0.00 + e - e y = a + b.x y: Bias, x: Ref. Value a: Slope, b: Intercept Linearity just evaluates if any Bias is consistent throughout the measurement range of the instrument. Many times Linearity indicates a need to replace or maintenance measurement equipment.

MSA’s fall into two categories: Attribute Variable Types of MSA’s MSA’s fall into two categories: Attribute Variable Transactional projects typically have Attribute based measurement systems. Manufacturing projects generally use Variable studies more often, but do use Attribute studies to a lesser degree. Attribute Pass/Fail Go/No Go Document Preparation Surface imperfections Customer Service Response Variable Continuous scale Discrete scale Critical dimensions Pull strength Warp Variable Data is always preferred over Attribute because it give us more to work with. Now we are gong to review Variable MSA testing.

Variable MSA’s SigmaXL® calculates a column of variance components (VarComp) which are used to calculate % Gage R&R using the ANOVA Method. Estimates for a Gage R&R study are obtained by calculating the variance components for each term and for error. Repeatability, Operator and Operator*Part components are summed to obtain a total Variability due to the measuring system. We use variance components to assess the Variation contributed by each source of measurement error relative to the total Variation. True Value Measured Value MSA’s use a random effects model meaning that the levels for the variance components are not fixed or assigned, they are assumed to be random.

Cheat Sheet Use % Study Var when you are interested in comparing the measurement system Variation to the total Variation. % Study Var is calculated by dividing each value in Study Var by Total Variation and Multiplying by 100. Study Var is calculated as 5.15 times the Standard Deviation for each source. (5.15 is used because when data are normally distributed, 99% of the data fall within 5.15 Standard Deviations.) Contribution of Variation to the total Variation of the study. % Contribution, based on variance components, is calculated by dividing each value in VarComp by the Total Variation then multiplying the result by 100. SigmaXL®’s default is 6 * StDev. Select “5.15 * StDev” in the dialog box. SigmaXL® uses this default to be consistent with 6 * StDev used in process capability . The 5.15 multiplier is common in the automotive industry (AIAG MSA Handbook). The worksheet used for this example is “Gage AIAG 2 -SigmaXL Template” . The above slides are for demonstration purposes, this dataset will be used in a later exercise.

SigmaXL® Report: Cheat Sheet When the process tolerance is entered in the system, SigmaXL® calculates % Tolerance which compares measurements system Variation to customer specification. This allows us to determine the proportion of the process tolerance that is used by the Variation in the measurement system. Notice the calculation method explained here for Distinct Categories. Traditionally NDC is truncated to an integer value, but SigmaXL® reports a more informative one decimal place. Distinct Categories (Rounded Down )

Number of Distinct Categories The number of distinct categories tells you how many separate groups of parts the system is able to distinguish.   Unacceptable for estimating process parameters and indices Only indicates whether the process is producing conforming or nonconforming parts Generally unacceptable for estimating process parameters and indices Only provides coarse estimates Recommended 1 Data Category 2 - 4 Categories Here is a rule of thumb for distinct categories. 5 or more Categories

AIAG Standards for Gage Acceptance Here are the Automotive Industry Action Group’s definitions for Gage acceptance. % Tolerance or % Study Variance % Contribution System is… 10% or less 10% - 20% 20% - 30% 30% or greater 1% or less 1% - 4% 5% - 9% 10% or greater Ideal Acceptable Marginal Poor   Please read the slide.

SigmaXL® Graphic Output Cheat Sheet Components of Variation The SigmaXL® report breaks down the variation in the measurement system into specific sources. The bar chart shown was created using Excel’s Clustered Column Bar Chart to graphically display the Components of Variation. Each cluster of bars represents a source of variation. In a good measurement system, the largest component of Variation is Part-to-Part variation. If instead you have large amounts of variation attributed to Gage R&R, then corrective action is needed. Note, this chart may be recreated by taking the following steps: 1. Copy the “% Total Variation (TV)” column. 2. Paste the column to the right of the “% Contribution of Variance Component” column. 3. Highlight the entire table which the “% Total Variation (TV)” was added. 4. From Excel, Insert> (Chart) Column>2-D Clustered Column. 5. Delete the “Variance Component “ column from this chart.

SigmaXL® Graphic Output Cheat Sheet SigmaXL® provides an R Chart and Xbar Chart by Operator. The R chart consists of the following: - The plotted points are the difference between the largest and smallest measurements on each part for each operator. If the measurements are the same then the range = 0. - The Center Line, is the grand average for the process. - The Control Limits represent the amount of variation expected for the subgroup ranges. These limits are calculated using the variation within subgroups. If any of the points on the graph go above the upper Control Limit (UCL), then that operator is having problems consistently measuring parts. The Upper Control Limit value takes into account the number of measurements by an operator on a part and the variability between parts. If the operators are measuring consistently, then these ranges should be small relative to the data and the points should stay in control. This chart may be found in the “Gage R&R - X-Bar & R (1)” worksheet.

SigmaXL® Graphic Output Cheat Sheet SigmaXL® provides an R Chart and Xbar Chart by Operator. The Xbar Chart compares the part-to-part variation to repeatability. The Xbar chart consists of the following: - The plotted points are the average measurement on each part for each operator. - The Center Line is the overall average for all part measurements by all operators. - The Control Limits (UCL and LCL) are based on the variability between parts and the number of measurements in each average. Because the parts chosen for a Gage R&R study should represent the entire range of possible parts, this graph should ideally show lack-of-control. Lack-of-control exists when many points are above the Upper Control Limit and/or below the Lower Control Limit. In this case there are several points out of control which indicates the measurement system is adequate. This chart may be found in the “Gage R&R - X-Bar & R (1)” worksheet.

SigmaXL®’s Gage R&R Multi-Vari Output The Multi-Vari Charts show each Part as a separate graph. Each Operator’s response readings are denoted as a vertical line with the top tick corresponding to the Maximum value, bottom tick is the Minimum, and the middle tick is the Mean. The horizontal line across each graph is the overall average for each part. Ideally the connected means red line should be horizontal (i.e., small reproducibility) and the vertical lines should be short (small repeatability). SigmaXL® produces these Multi-Vari Charts as part of the Gage R&R report. Select the “Gage R&R - X-Bar & R (1)” worksheet.

SigmaXL® Graphic Output Cheat Sheet Pattern Means… Lines are virtually identical Operators are measuring the parts the same One line is consistently higher or lower than the others That operator is measuring parts consistently higher or lower than the others Lines are not parallel or they cross The operators ability to measure a part depends on which part is being measured (an interaction between operator and part) Using the SigmaXL® two-way ANOVA tool creates an interaction chart that shows the average measurements taken by each operator on each part in the study, arranged by part. Each line connects the averages for a single operator. Ideally, the lines will follow the same pattern and the part averages will vary enough that differences between parts are clear. Currently the Gage R&R report in SigmaXL® does not include an Interaction Plot. The following steps show how to create the Interaction Plot using Two-Way ANOVA: 1. Select the data which will be used for the chart. 2. Select “SigmaXL>Statistical Tools>Two-Way ANOVA” 3. This chart was generated from the “Gage AIAG2 - SigmaXL Format” worksheet. Select “Response” as “Numerical Data Variable (Y)”, “Part” as “Group Category Factor (X1)”, and “Operator” as “Group Category Factor (X2)”. The operator by part interaction plot is given in the Two-Way ANOVA output worksheet.

SigmaXL® Graphic Output Cheat Sheet The “By Part” Multi-Vari Chart allows us to analyze all of the measurements taken in the study arranged by part. The measurements are represented by dots; the means by the middle bar. The red line connects the average measurements for each part. Ideally, Multiple measurements for each individual part have little variation (the dots for one part will be close together) Averages will vary enough that differences between parts are clear Currently the Gage R&R report in SigmaXL® does not include a Multi-Vari Chart showing all parts. To create the above chart, use SigmaXL®’s Multi-Vari tool: Select the data which will be used for the chart. Select “SigmaXL>Graphical Tools>Multi-Vari Chart” 3. This chart was generated from the “Gage AIAG2 - SigmaXL Format” worksheet. Select “Response” as “Numerical Data Variable (Y)”, “Part” as “Group Category Factor (X1)”, and “Operator” as “Group Category Factor (X2)”.

SigmaXL® Graphic Output Cheat Sheet The “By Operator” Multi-Vari Chart is created by modifying the X’s from parts to operator. This helps us determine whether the variability in measurements are consistent across operators. The by operator graph shows all the study measurements arranged by operator. Dots represent the measurements; the middle bars represent the means. The red line connects the average measurements for each operator. You can also assess whether the overall Variability in part measurement is the same using this graph. Is the spread in the measurements similar? Or is one operator more Variable than the others? If the red line is … Then… Parallel to the x-axis The operators are measuring the parts similarly Not parallel to the x-axis The operators are measuring the parts differently This Multi-Vari Chart was created with SigmaXL®’s Multi-Vari tool: Select the data which will be used for the chart. Select “SigmaXL>Graphical Tools>Multi-Vari Chart” 3. This chart was generated from the “Gage AIAG2 - SigmaXL Format” worksheet. Select “Response” as “Numerical Data Variable (Y)”, “Operator” as “Group Category Factor (X1)”, and “Part” as “Group Category Factor (X2)”.

Practical Conclusions For this example, the measuring system contributes little to the overall Variation, as confirmed by both the Gage R&R table and graphs. The Variation due to the measurement system, as a percent of study (Total) Variation is causing 16.80% of the Variation seen in the process. By AIAG Standards this gage should be used. By all standards, the data being produced by this gage is acceptable, and valid for analysis. % Tolerance or % Study Variance % Contribution System is… 10% or less 10% - 20% 20% - 30% 30% or greater 1% or less 1% - 4% 5% - 9% 10% or greater Ideal Acceptable Marginal Poor Please read the slide.

Repeatability and Reproducibility Problems Repeatability Problems: Calibrate or replace gage. If only occurring with one operator, re-train. Reproducibility Problems: Measurement machines Similar machines Ensure all have been calibrated and that the standard measurement method is being utilized. Dissimilar machines One machine is superior. Operators Training and skill level of the operators must be assessed. Operators should be observed to ensure that standard procedures are followed. Operator/machine by part interactions Understand why the operator/machine had problems measuring some parts and not others. Re-measure the problem parts Problem could be a result of gage linearity Problem could be fixture problem Problem could be poor gage design For Repeatability Problems: If all operators have the same repeatability and it is too big, the gage needs to be repaired or replace. If only one operator or in the case where there are no operators, but several gages and only one gage is showing repeatability problems, re-train the one operator or replace the one gage. For Reproducibility Problems: In the case where only machines are used and the multiple machines are all similar in design, check the calibration and ensure that the standard measurement method is being used. One of the gages maybe performing differently than the rest, the graphs will show which one is performing differently. It may need to go in for repair, or it may simply be a setup or calibration issue. If dissimilar machines are used it typically means that one machine is superior. In the case where multiple operator are the graphs will show who will need additional training to perform at the same level as the rest. The most common operator/machine interactions are either someone misread a value, recorded the value incorrectly or that the fixture holding the part is poor.

Design Types Crossed Design Nested Design A Crossed Design is used only in non-destructive testing and assumes that all the parts can be measured multiple times by either operators or multiple machines. Gives the ability to separate part-to-part Variation from measurement system Variation. Assesses Repeatability and Reproducibility. Assesses the interaction between the operator and the part. Nested Design A Nested Design is used for destructive testing (we will learn about this in MBB training) and also situations where it is not possible to have all operators or machines measure all the parts multiple times. Destructive testing assumes that all the parts within a single batch are identical enough to claim they are the same. Nested designs are used to test measurement systems where it is not possible (or desirable) to send operators with parts to different locations. Do not include all possible combinations of factors. Uses slightly different mathematical model than the Crossed Design. Crossed Designs are the workhorse of MSA. They are the most commonly used design in industries where it is possible to measure something more than once. Chemical and biological systems can use Crossed Designs also as long as you can assume that the samples used come from a homogeneous solution and there is no reason they can be different. Nested Designs must be used for destructive testing. In a Nested Design, each part is measured by only one operator. This is due to the fact that after destructive testing, the measured characteristic is different after the measurement process than it was at the beginning. Crash testing is an example of destructive testing. If you need to use destructive testing, you must be able to assume that all parts within a single batch are identical enough to claim that they are the same part. If you are unable to make that assumption then part-to-part variation within a batch will mask the measurement system variation. If you can make that assumption, then choosing between a Crossed or Nested Gage R&R Study for destructive testing depends on how your measurement process is set up. If all operators measure parts from each batch, then use Gage R&R Study (Crossed). If each batch is only measured by a single operator, then you must use Gage R&R Study (Nested). In fact, whenever operators measure unique parts, you have a Nested Design. Your Master Black Belt can assist you with the set-up of your design.

NO, not that kind of R&R! Gage R&R Study Is a set of trials conducted to assess the Repeatability and Reproducibility of the measurement system. Multiple people measure the same characteristic of the same set of multiple units multiple times (a crossed study) Example: 10 units are measured by 3 people. These units are then randomized and a second measure on each unit is taken. A Blind Study is extremely desirable. Best scenario: operator does not know the measurement is a part of a test At minimum: operators should not know which of the test parts they are currently measuring. A Gage R&R, like any study, requires careful planning. The common way of doing an Attribute Gage R&R consists of having at least two people measure 20 parts at random, twice each. This will enable you to determine how consistently these people evaluate a set of samples against a known standard. If there is no consistency among the people, then the measurement system must be improved, either by defining a measurement method, training, etc. You use an Excel spreadsheet template to record your study and then to perform the calculations for the result of the study. NO, not that kind of R&R!

Variable Gage R & R Steps Step 1: Call a team meeting and introduce the concepts of the Gage R&R Step 2: Select parts for the study across the range of interest If the intent is to evaluate the measurement system throughout the process range, select parts throughout the range If only a small improvement is being made to the process, the range of interest is now the improvement range Step 3: Identify the inspectors or equipment you plan to use for the analysis In the case of inspectors, explain the purpose of the analysis and that the inspection system is being evaluated not the people Step 4: Calibrate the gage or gages for the study Remember Linearity, Stability and Bias Step 5: Have the first inspector measure all the samples once in random order Step 6: Have the second inspector measure all the samples in random order Continue this process until all the operators have measured all the parts one time This completes the first replicate Step 7: Repeat steps 5 and 6 for the required number of replicates Ensure there is always a delay between the first and second inspection Step 8: Enter the data into SigmaXL® and analyze your results Step 9: Draw conclusions and make changes if necessary The parts selected for the MSA are not random samples. We want to be sure the parts selected represent the overall spread of parts that would normally be seen in manufacturing. Do not include parts that are obviously grossly defective, they could actually skew your mathematical results and conclude that the MSA is just fine. For example, an engine manufacturer was using a pressure tester to check for leaks in engine blocks. All the usual ports were sealed with plugs and the tester was attached and pressure was applied. Obviously, they were looking for pin hole leaks that would cause problems later down the line. The team performing the MSA decided to include an engine block that had a hole in the casting so large you could insert your entire fist. That was an obvious gross defect and should never been included in the MSA. Don’t be silly saying that once in a while you get a part like that and it should be tested. NO IT SHOULDN’T - you should never have received it in the first place and you have got much bigger problems to take care of before you do an MSA.

Part Allocation From Any Population Gage R & R Study Part Allocation From Any Population 10 x 3 x 2 Crossed Design is shown A minimum of two measurements/part/operator is required Three is better! Trial 1 Operator 1 Trial 2 Parts Trial 1 Operator 2 1 2 3 4 5 6 7 8 9 10 This is the most commonly used Crossed Design. 10 parts are each measured by 3 different operators 2 different times. To get the total number of data points in the study simply multiply these numbers together. In this study we have 60 measurements. Trial 2 Trial 1 Operator 3 Trial 2

Create a data collection sheet for: 10 parts 3 operators 2 trials The next few slides show how to create a data collection table in SigmaXL®.

The Data Collection Sheet

Open the worksheet “Gage AIAG2 - SigmaXL Format”. Gage R & R Open the worksheet “Gage AIAG2 - SigmaXL Format”. Variables: Part Operator Response Please read the slide.

Gage R & R Use 1.0 for the tolerance. We will now repeat the analysis of the previous Gage R&R data with a Standard Deviation Multiplier of 6 and a Tolerance value of 1. Recall that the previous analysis used a Standard Deviation Multiplier of 5.15. Change Alpha to remove interaction to 0.25. This will prevent SigmaXL® from removing the part by operator interaction term. Select “SigmaXL>Measurement Systems Analysis > Analyze Gage R&R (Crossed)” and enter the values as shown above.

Part to Part Variation needs to be larger than Gage Variation Graphical Output Looking at the “Components of Variation” chart, the Part to Part Variation needs to be larger than Gage Variation. If in the “Components of Variation” chart the “Gage R&R” bars are larger than the “Part-to-Part” bars, then all your measurement Variation is in the measuring tool i.e.… “maybe the gage needs to be replaced”. Part to Part Variation needs to be larger than Gage Variation Note, this chart may be recreated by taking the following steps: 1. Copy the “% Total Variation (TV)” column. 2. Paste the column to the right of the “% Contribution of Variance Component” column. 3. Copy the “% Tolerence” column. 4. Paste the column to the right of the “% Total Variation (TV)” column. 5. Highlight the entire table which the “% Total Variation (TV)” was added. 6. From Excel, Insert> (Chart) Column>2-D Clustered Column. 7. Delete the “Variance Component “ column from this chart.

Graphical Output The same concept applies to the “Response by Operator” chart. If there is extreme Variation within operators, then the training of the operators is suspect. Operator Error This Multi-Vari Chart was created with SigmaXL®’s Multi-Vari tool: Select the data which will be used for the chart. Select “SigmaXL>Graphical Tools>Multi-Vari Chart” 3. This chart was generated from the “Gage AIAG2 - SigmaXL Format” worksheet. Select “Response” as “Numerical Data Variable (Y)”, “Operator” as “Group Category Factor (X1)”, and “Part” as “Group Category Factor (X2)”.

I can see clearly now! Session Window The ANOVA table values are utilized to calculate % Contribution and Standard Deviation. To calculate % study variation and % tolerance, you will need to know values for the Standard Deviation and tolerance ranges. SigmaXL® defaults to a value of 6 (the number of Standard Deviations within which about 99.7 % of your values should fall). Tolerance ranges are based on process tolerance and are business values specific to each process.

If the Variation due to Gage R & R is high, consider: Session Window If the Variation due to Gage R & R is high, consider: Procedures revision? Gage update? Operator issue? Tolerance validation? 20 % < % Tol GRR < 30%  Gage Unacceptable 10 % < % Tol GRR < 20 %  Gage Acceptable 1 % < % Tol GRR < 10 %  Gage Preferable This output tells us that the part to part variation exceeds the allowable tolerance. This gage is acceptable.

Uses average of repeat measurements. Signal Averaging Signal Averaging can be used to reduce Repeatability error when a better gage is not available. Uses average of repeat measurements. Uses Central Limit theorem to estimate how many repeat measures are necessary. Signal Averaging is a method to reduce Repeatability error in a poor gage when a better gage is not available or when a better gage is not possible. Please read the slide.

Signal Averaging Example Suppose SV/Tolerance is 35%. SV/Tolerance must be 15% or less to use gage. Suppose the Standard Deviation for one part measured by one person many times is 9.5. Determine what the new reduced Standard Deviation should be. Here we have a problem with Repeatability, not Reproducibility so we calculate what the Standard Deviation should be in order to meet our desire of a 15% gage. The 35% represents the biggest problem, Repeatability. We are assuming that 15% will be acceptable for the short term until an appropriate fix can be implemented. The 9.5 represents our estimate for Standard Deviation of population of Repeatability.

Signal Averaging Example Determine sample size: Using the average of 6 repeated measures will reduce the Repeatability component of measurement error to the desired 15% level. This method should be considered temporary! We now use it in the Central Limit Theorem equation to estimate the needed number of repeated measures to do this we will use the Standard Deviation estimated previously.

Paper Cutting Exercise Exercise objective: Perform and Analyze a variable MSA Study. 1. Cut a piece of paper into 12 different lengths that are all fairly close to one another but not too uniform. Label the back of the piece of paper to designate its “part number” 2. Perform a variable gage R&R study as outlined in this module. Use the following guidelines: Number of parts: 12 Number of inspectors: 3 Number of trials: 5 3. Create a SigmaXL® data sheet and enter the data into the sheet as each inspector performs a measurement. If possible, assign one person to data collection. 4. Analyze the results and discuss with your mentor. Exercise.

Attribute Gage Error Repeatability Reproducibility Calibration Attribute MSA A methodology used to assess Attribute Measurement Systems. They are used in situations where a continuous measure cannot be obtained. It requires a minimum of 5x as many samples as a continuous study. Disagreements should be used to clarify operational definitions for the categories. Attribute data are usually the result of human judgment (which category does this item belong in). When categorizing items (good/bad; type of call; reason for leaving) you need a high degree of agreement on which way an item should be categorized. Attribute Gage Error Repeatability Reproducibility Calibration The Discrete Measurement Study is a set of trials conducted to assess the ability of operators to use an operational definition or categorize samples, an Attribute MSA has: 1 . Multiple operators measure (categorize) multiple samples a multiple number of times. For example: 3 operators each categorize the same 50 samples, then repeat the measures at least once. 2. The test should be blind. It is difficult to run this without the operator knowing it is a calibration test, but the samples should be randomized and their true categorization unknown to each operator. The test is analyzed based on correct (vs. incorrect) answers to determine the goodness of the measuring system.

The purpose of an Attribute MSA is: Attribute MSA Purpose The purpose of an Attribute MSA is: To determine if all inspectors use the same criteria to determine “pass” from “fail”. To assess your inspection standards against your customer’s requirements. To determine how well inspectors are conforming to themselves. To identify how inspectors are conforming to a “known master,” which includes: How often operators ship defective product. How often operators dispose of acceptable product. Discover areas where: Training is required. Procedures must be developed. Standards are not available. An Attribute MSA is similar in many ways to the continuous MSA, including the purposes. Do you have any visual inspections in your processes? In your experience how effective have they been? When a continuous MSA is not possible an Attribute MSA can be performed to evaluate the quality of the data being reported from the process.

Visual Inspection Test Take 60 Seconds and count the number of times “F” appears in this paragraph? The Necessity of Training Farm Hands for First Class Farms in the Fatherly Handling of Farm Live Stock is Foremost in the Eyes of Farm Owners. Since the Forefathers of the Farm Owners Trained the Farm Hands for First Class Farms in the Fatherly Handling of Farm Live Stock, the Farm Owners Feel they should carry on with the Family Tradition of Training Farm Hands of First Class Farmers in the Fatherly Handling of Farm Live Stock Because they Believe it is the Basis of Good Fundamental Farm Management. Have each person in the class count the “F’s” in the paragraph above. Tally the answers? Did everyone get the same answer? Did anyone get 36? That’s the right answer! Why not? Does everyone know what an “F” (defect) looks like? Was the lighting good in the room? Was it quite so you could concentrate? Was the writing clear? Was 60 seconds long enough? This is the nature of visual inspections! How many places in your process do you have visual inspection? How good do you expect them to be?

How can we Improve Visual Inspection? Visual Inspection can be improved by: Operator Training & Certification Develop Visual Aids/Boundary Samples Establish Standards Establish Set-Up Procedures Establish Evaluation Procedures Evaluation of the same location on each part. Each evaluation performed under the same lighting. Ensure all evaluations are made with the same standard. Look closely now! Please read the slide.

Attribute Agreement Analysis Attribute MSA (Binary) Attribute MSA is also known as Attribute Agreement Analysis. The response must be binary (e.g. Pass/Fail, Good/Bad, G/NG, Yes/No). Open the worksheet Attribute MSA – AIAG. This is an example from the AIAG MSA Reference Manual, 3rd Edition, page 127. Note that the worksheet data must be in stacked column format Click SigmaXL > Measurement Systems Analysis > Attribute MSA (Binary). Ensure that the entire data table is selected. Click Next. Select Part, Appraiser, Assessed Result and Reference as shown. Check Report Information and enter AIAG Example, Page 127. Click OK. The results are shown on the next slide. Note, SigmaXL® version 6 only supports binary attribute MSA. Ordinal attribute MSA will be included in Version 7. SigmaXL® also includes a template for Attribute MSA. Select “SigmaXL>Measurement Systems Analysis> Basic MSA Templates> Attribute MSA”

Attribute MSA (Binary) Fleiss’ Kappa statistic is a “correlation coefficient” for discrete data. Kappa ranges from -1 to +1. A Kappa value of +1 indicates perfect agreement. If Kappa = 0, then agreement is the same as would be expected by chance. If Kappa = -1, then there is perfect disagreement. Kappa values > 0.9 indicate a very good measurement system; Kappa values > 0.7 indicate an acceptable measurement system. The Between Appraiser Agreement and All Appraisers vs. Standard Agreement are also known as “System Effectiveness Scores”, with > 95% considered very good, 90-95% acceptable, 80 to < 90 % marginal, and < 80 % unacceptable. Clearly this measurement system needs to be improved, but we should not be quick to judge Appraiser C. The confidence intervals are quite wide and overlap. It is a good practice to “blame the process not the people”. Look for unclear or confusing operational definitions, inadequate training, operator distractions or poor lighting. Consider the use of pictures to clearly define a defect. Use Attribute MSA as a way to “put your stake in the ground” and track the effectiveness of improvements to the measurement system.

M&M Exercise Exercise objective: Perform and Analyze an Attribute MSA Study. You will need the following to complete the study: A bag of M&Ms containing 50 or more “pieces” The attribute value for each piece. Three or more inspectors. Judge each M&M as pass or fail. The customer has indicated that they want a bright and shiny M&M and that they like M’s. Pick 50 M&Ms out of a package. Enter results into SigmaXL®'s Attribute MSA Template and draw conclusions. The instructor will represent the customer for the Attribute score. Part Attribute Number M&M 1 2 3 Pass Fail To complete this study you will need, a bag of M&Ms containing 50 or more “pieces”. The attribute value for each piece, which means the “True” value for each piece, in addition to being the facilitator of this study you will also serve as the customer, so you will have the say as to if the piece is actually a Pass or Fail piece. Determine this before the inspectors review the pieces. You will need to construct a sheet as shown here to keep track of the “pieces” or “parts” in our case M&Ms it is important to be well organized during these activities. Then the inspectors will individually judge each piece based on the customer specifications of bright and shiny M&M with nice M’s.

At this point, you should be able to: Summary At this point, you should be able to: Understand Precision & Accuracy Understand Bias, Linearity and Stability Understand Repeatability & Reproducibility Understand the impact of poor gage capability on product quality Identify the various components of Variation Perform the step by step methodology in Variable and Attribute MSA’s Please read the slide.

Get the latest products at… The Certified Lean Six Sigma Green Belt Assessment The Certified Lean Six Sigma Green Belt (CLSSGB) tests are useful for assessing Green Belt’s knowledge of Lean Six Sigma. The CLSSGB can be used in preparation for the ASQ or IASSC Certified Six Sigma Green Belt (CSSGB) exam or for any number of other certifications, including private company certifications. The Lean Six Sigma Green Belt Course Manual Open Source Six Sigma Course Manuals are professionally designed and formatted manuals used by Belt’s during training and for reference guides afterwards. The OSSS manuals complement the OSSS Training Materials and consist of slide content, instructional notes data sets and templates. Get the latest products at… www.OpenSourceSixSigma.com 63