1. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 1 MER301: Engineering Reliability LECTURE 16: Measurement System Analysis and Uncertainty Analysis-Part 1

2. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 We must submit the output from our design process to a second (measurement) process Measurement Process Outputs Measurements Process InputsOutputs Inputs Parts (Example) Measurement as a Process 2

3. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 3 Measurement System Concerns.. l How big is the measurement error? l What are the sources of measurement error? l Are the measurements being made with units which are small enough to properly reflect the variation present? l Is the measurement system stable over time? l How much uncertainty should be attached to a measurement system when interpreting data from it? l How do we improve the measurement system?

4. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 4 Measurement System Analysis  Total Error in a measurement is defined as the difference between the Actual Value and Observed Value of Y  Two general categories of error – Accuracy or Bias and Precision Accuracy or Bias of Measurement System is defined as the difference between a Standard Reference and the Average Observed Measurement Precision of a Measurement System is defined as the standard deviation of Observed Measurements of a Standard Reference Total Error = Bias Error + Precision Error for independent random variables  Measurement System Error is described by Average Bias Error (Mean Shift)and a statistical estimate of the Precision Error (Variance) Measurement System Analysis is a Fundamental Part of Every Experiment

5. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 5 Not Accurate, Not PreciseAccurate, Not Precise Not Accurate, PreciseAccurate, Precise Precision and Accuracy

6. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 6 Measurement System Analysis  Bias or Accuracy error is a constant value and is dealt with by calibrating the measurement system  Variation or Precision error is a random variable which depends on the measurement equipment(the instruments used) and on the measurement system repeatability and reproducibility. Instrument Capability Analysis, Test/retest (repeatability)and Gage R&R studies are used to quantify the size of these errors.

7. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 Actual Defects LSL USL = Product Std. Dev. = Product Mean LSL USL Observed Defects Measurement system variance Product variance Impact of Measurement System Variation on Variation in Experimental Data 7

8. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 8 Example 16.1-Effect of Measurement System Variation  Calculate the effect of measurement system variation on the acceptance rate for a part with USL and LSL at Z= +/-1.96 respectively. If then what is the percentage of acceptable parts that will be rejected? If on the other hand what is the percentage of acceptable parts that will be rejected?

9. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 9 Impact of Measurement System Variation on Variation in Experimental Data…

10. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 10 Example 16.1(con’t: )  For the product, the spec limits of +/-1.96 mean that the 2.5% of parts in each tail are out of spec. Thus  For the observed standard deviation is and Then the acceptable parts now rejected are

11. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 11 Example 16.1(con’t: )  For the product, the spec limits of +/-1.96 mean that the 2.5% of parts in each tail are out of spec. Thus  For the observed standard deviation is and Then the acceptable parts now rejected are

12. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 12 Example 16.1(con’t) Set Measurement System Requirements Based on the Process Variation Unacceptable Acceptable

13. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 Gage Repeatability & Reproducibility ---GRR or GR&R---  Gage Repeatability & Reproducibility compares measurement system variation and product variation  The term is the size of an interval containing 99% of the measured values made on a specific item  The Tolerance- often equal to - is the size of the interval where a product has acceptable dimensions, performance, or other characteristics 13

14. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 Gage Performance relative to required Tolerance Band  R&R less than 10% - Measurement system is acceptable.  R&R 10% to 30% - Maybe acceptable - make decision based on classification of characteristic, hardware application, customer input, etc.  R&R over 30% - Not typically acceptable. Find the problem using root cause analysis(fishbone), remove root causes GRR is a measure of “noise” in the data 14

15. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 15 Effect of Gage R&R on Variation  GRR <10% means < 0.7% of the variation in the experimental data is from the measurement system  GRR> 30% means that > 5.9% of the variation in the experimental data is from the measurement system

16. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 16 GRR Example 16.2  The GRR values for the previous Example 16.1 are  The capabilities of two (or more) measurement systems can be compared by comparing the GRR’s for each. Since GRR 2 <GRR 1, the second measurement system is more capable than the first. The observed standard deviations quantify how much better….

17. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 17 GRR Example 16.2  The GRR values for the previous Example 16.1 are at best marginally acceptable(GRR 2 ) or not acceptable(GRR 1 )  For a GRR value equal to 10% (0.10) there results

18. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 18 Example 16.2 ( so )  For the product, the spec limits of +/-1.96 mean that the 2.5% of parts in each tail are out of spec. Thus  For the observed standard deviation is and Then the acceptable parts now rejected are

19. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 19 Summarizing how it all fits together…..  When a set of measurements are made, the results are always observed values,  If the actual mean and standard deviation are known then the measurement system bias and variance can be calculated If the item being measured is a standard reference  If the measurement system bias and variance are known then the actual mean and actual variance can be calculated

20. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 Total Variation made up of Actual Process Variation and Measurement System Variation Total Variation made up of Actual Process Variation and Measurement System Variation Sources of Measurement System Error Process Inputs Outputs Inputs Measurement Process Outputs Observations Measurements Long-term Process Variation Actual Process Variation Accuracy (Bias) Measurement Variation Observed Process Variation Short-term Process Variation Variation due to gauge Variation due to operator Precision (Pure Error) Stability (time dependent) Linearity (value dependent) Repeatability Reproducibility within sample variation Measurement System Repeatability Resolution 20

21. L Berkley Davis Copyright 2009 Engineering Reliability Lecture 16 21 Measurement System Errors Repeatability (precision) Reproducibility Operator B Operator A Stability Time 1 Time 2 Observed Average Accuracy (Bias) True Average True Average Accuracy (Low End) Accuracy (High End) Observed Average (Low End) Observed Average (High End) Linearity

22. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 22 Elements that contribute to Accuracy and Precision Errors  Instrument Capability Resolution Gage Repeatability Linearity  Measurement System - Short Term (ST) Instrument Capability Equipment Calibration(Bias) Test/Re-Test Study(Repeatability)  Measurement System - Long Term (LT) Use Measurement System - Short Term Use Reproducibility Stability First Two are Entitlement….Third is Reality

23. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 23 Elements that contribute to Precision or Variation Errors  Instrument Capability Resolution Gage Repeatability Linearity  Measurement System- Short Term (ST) Use Instrument Capability Equipment Calibration(Bias) Test/Re-Test Study(Repeatability)  Measurement System - Long Term (LT) Use Measurement System - Short Term Use (ST) Reproducibility(Gage R&R) Stability(Gage R&R) First Two are Entitlement….Third is Reality

24. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 24 Measurement System Analysis From pages 119-120…

25. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 25 Updating how variances all fit together  When a set of measurements are made, the results are always observed values,  If the actual mean and standard deviation are known then the measurement system bias and variance can be calculated If the item being measured is a standard reference  If the measurement system bias and variance are known then the actual mean and actual variance can be calculated

26. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 26 Emissions Sampling NOx Instrument Y actual - NOx from Gas turbine Cal/Zero Gases Y obs - NOx Reading Heated Sampling Line Calibration Gas Sample Conditioning

27. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 27 Elements that contribute to Accuracy and Precision Errors  Instrument Capability Resolution Gage Repeatability Linearity Measurement System- Short Term(ST) Use Instrument Capability Equipment Calibration Test/Re-Test Study Measurement System- Long term (LT) Use Measurement System -Short Term(ST) Use Reproducibility Stability

28. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 28  Establish magnitude and sources of measurement system error due to bias and precision errors  Tools Instrument Capability Analysis Test/Re-test – system precision/repeatability Calibration - bias “Continuous Variable” Gage R&R (Gage Reproducibility and Repeatability) Attribute Variable Gage R&R Destructive Gage R&R How Can we Address Accuracy and Precision Errors?

29. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 29 Measurement System Analysis  Instrument Capability Analysis…..  Resolution-smallest increment that the gage can resolve in the measurement process. Gage should be able to resolve tolerance band into ten or more parts. Resolution Uncertainty =  Instrument Accuracy- measure of instrument repeatability or instrument “noise”.. Found by repeated measurements of the same test item. Uncertainty =  Linearity- consistency of the measurement system across the entire range of the measurement system. Linearity Uncertainty =  The variations are combined as follows

30. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 30 Instrument Capability Analysis…..  Variation for any one instrument equals the sum of the resolution, repeatability and linearity terms  The Variation for “n” instruments equals the sum of the variations for each individual instrument Each of the “n” instruments has resolution, repeatability, and linearity terms that must be taken into account

31. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 31 Instrument Capability Analysis - Resolution of Instruments/Sensors  The measurement uncertainty due to resolution is generally taken as a specified fraction of the smallest increment an instrument can resolve, ie as a fraction of the smallest scale division  General Rule: assign a numerical value for the mean value of equal to one half of the instrument resolution. This means that half of the smallest scale division is assumed to equal a 95% Confidence Interval ( a wide band) for variation due to resolution

32. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 32 Instrument Capability Analysis Repeatability and Linearity  The manufacturer of an instrument will provide information on the capability of the instrument in the specification sheets provided with the instrument  The numerical values given for Instrument or Sensor Accuracy and Linearity are almost always uncertainties Let = uncertainty due to the equipment accuracy/repeatability error where Let = uncertainty due to linearity error where  The inherent capability/uncertainty of the instrument/sensor is then estimated as:

33. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 33 Example 16.3-Instrument Capability  The Capability of a force measuring instrument is described by catalogue data. Calculate an estimate of the variation attributable to this instrument. Express the result both in dimensional terms (N) and in dimensionless terms for a reading R=50N Resolution0.25N Range0 to 100N Linearitywithin 0.20N over range Repeatabilitywithin 0.30N over range

34. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 34 Example 16.3(con’t)  An estimate of the instrument uncertainty depends on the combined uncertainties due to resolution, repeatability and linearity The instrument uncertainty is then

35. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 35 Measurement System Analysis  Instrument Capability Analysis Summary…..  Resolution-smallest increment that the gage can resolve in the measurement process. Gage should be able to resolve tolerance band into ten or more parts. Resolution Uncertainty =  Instrument Accuracy- measure of gage repeatability or gage “noise”.. Found by repeated measurements of the same test item. Uncertainty =  Linearity- consistency of the measurement system across the entire range of the measurement system. Linearity Uncertainty =  The variations are combined as follows

36. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 36 Measurement System Analysis  Measurement System Short Term Use Includes Instrument Capability Repeatability - variation when one operator repeatedly makes the same measurement with the same measuring equipment Test/Re-test Study Calibration/Bias  Measurement System-Long Term Use Includes Measurement System –Short Term Use Reproducibility- variation when two or more operators make same measurement with the same measuring equipment Stability-variation when the same operator makes the same measurement with the same equipment over an extended period of time

37. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 37 Test/Retest Example 16.4  Test/Retest (Repeatability) Study on a Measurement System. Thirty repeat measurements were taken on a Standard Reference Item with a thickness of 50mils The tolerance band for the application is 20mils(+/-10).  Data, in mils 53,45,52,47,54,52,52,55,52,48,48,53,55,51,47,52,47, 35,45,54,48,51,53,44,52,52,55,59,53,53

38. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 38 Example 16.4(con’t)  Objective is to establish the precision and accuracy of the measurement system  Precision-Repeatability In a good Measurement System, 99% of the measurements of a given item should fall within a band less than 1/10 of tolerance band  Accuracy/Bias Bias = sample mean- true value

39. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 39 Example 16.4 Run Chart and Histogram These results look bad to the eye… there are outliers and mean is high

40. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 40 Test/Retest Study Example 16.4 Summary  Descriptive Statistics Variable N Mean Median StDev SE Mean C1 30 50.567 52.000 4.561 0.833 Variable Minimum Maximum Q1 Q3 C1 35.000 59.000 47.750 53.000  Conclusions Given the tolerance band of 20 mils,there is an unacceptable level of device precision Given the Reference Test item had a known thickness of 50mils, the bias(inaccuracy) is: bias = 50.57 – 50.0 = 0.57mils

41. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 41 Measurement System Analysis  Measurement System-Short Term Use  Repeatability-variation when one operator repeatedly makes the same measurement with the same measuring equipment Test/Re-test Study  Measurement System - Long Term Use  Reproducibility- variation when two or more operators make same measurement with the same measuring equipment  Stability-variation when the same operator makes the same measurement with the same equipment over an extended period of time

42. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 42 Elements that contribute to Accuracy and Precision Errors  Instrument Capability Resolution Gage Repeatability Linearity  Measurement System - Short Term (ST) Instrument Capability Equipment Calibration(Bias) Test/Re-Test Study(Repeatability)  Measurement System - Long Term (LT) Use Measurement System - Short Term Use Reproducibility Stability First Two are Entitlement….Third is Reality

43. L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 16 43 Measurement System Analysis: A Summary of the Basic Equations