slide 1 Six Sigma in Measurement Systems: Evaluating the Hidden Factory Scrap Rework Hidden Factory NOT OK OperationInputs Inspect First Time First TimeCorrect OK Time, cost, people Bill Rodebaugh Director, Six Sigma GRACE

slide 2 Objectives l The Hidden Factory Concept  What is a Hidden Factory?  What is a Measurement System’s Role in the Hidden Factory? l Review Key Measurement System metrics including %GR&R and P/T ratio l Case Study at W. R. GRACE  Measurement Study Set-up and Minitab Analysis  Linkage to Process  Benefits of an Improved Measurement System l How to Improve Measurement Systems in an Organization

slide 3 The Hidden Factory -- Process/Production Scrap Rework Hidden Factory NOT OK OperationInputs Inspect First Time First TimeCorrect OK Time, cost, people What Comprises the Hidden Factory in a Process/Production Area? Reprocessed and Scrap materials -- First time out of spec, not reworkable Over-processed materials -- Run higher than target with higher than needed utilities or reagents Over-analyzed materials -- High Capability, but multiple in-process samples are run, improper SPC leading to over-control

slide 4 The Hidden Factory -- Measurement Systems Waste Re-test Hidden Factory NOT OK Lab Work Sample Inputs Inspect Production OK Time, cost, people What Comprises the Hidden Factory in a Laboratory Setting? Incapable Measurement Systems -- purchased, but are unusable due to high repeatability variation and poor discrimination Repetitive Analysis -- Test that runs with repeats to improve known variation or to unsuccessfully deal with overwhelming sampling issues Laboratory “Noise” Issues -- Lab Tech to Lab Tech Variation, Shift to Shift Variation, Machine to Machine Variation, Lab to Lab Variation

slide 5 The Hidden Factory Linkage l Production Environments generally rely upon in- process sampling for adjustment l As Processes attain Six Sigma performance they begin to rely less on sampling and more upon leveraging the few influential X variables l The few influential X variables are determined largely through multi-vari studies and Design of Experimentation (DOE) l Good multi-vari and DOE results are based upon acceptable measurement analysis

slide 6 Objectives l The Hidden Factory Concept  What is a Hidden Factory?  What is a Measurement System’s Role in the Hidden Factory? l Review Key Measurement System metrics including %GR&R and P/T ratio l Case Study at W. R. GRACE  Measurement Study Set-up and Minitab Analysis  Linkage to Process  Benefits of an Improved Measurement System l How to Improve Measurement Systems in an Organization

slide 7 Possible Sources of Process Variation We will look at “repeatability” and “reproducibility” as primary contributors to measurement error

slide 8 LSLUSL Actual No Actual process variation - No measurement error Observed With Observed process variation - With measurement error LSLUSL How Does Measurement Error Appear?

slide 9 Measurement System Terminology l Discrimination - Smallest detectable increment between two measured values l Accuracy related terms  True value - Theoretically correct value  Bias - Difference between the average value of all measurements of a sample and the true value for that sample l Precision related terms  Repeatability - Variability inherent in the measurement system under constant conditions  Reproducibility - Variability among measurements made under different conditions (e.g. different operators, measuring devices, etc.) l Stability - distribution of measurements that remains constant and predictable over time for both the mean and standard deviation l Linearity - A measure of any change in accuracy or precision over the range of instrument capability

slide 10 Measurement Capability Index - P/T l Precision to Tolerance Ratio percent of the tolerance l Addresses what percent of the tolerance is taken up by measurement error l Includes both repeatability and reproducibility  Operator x Unit x Trial experiment l Best case: 10% Acceptable: 30% Usually expressed as percent Note: 5.15 standard deviations accounts for 99% of Measurement System (MS) variation. The use of 5.15 is an industry standard.

slide 11 Measurement Capability Index - % GR&R percent of the Observed Process Variation l Addresses what percent of the Observed Process Variation is taken up by measurement error l %R&R is the best estimate of the effect of measurement systems on the validity of process improvement studies (DOE) l Includes both repeatability and reproducibility l As a target, look for %R&R < 30% Usually expressed as percent

slide 12 Objectives l The Hidden Factory Concept  What is a Hidden Factory?  What is a Measurement System’s Role in the Hidden Factory? l Review Key Measurement System metrics including %GR&R and P/T ratio l Case Study at W. R. GRACE  Measurement Study Set-up and Minitab Analysis  Linkage to Process  Benefits of an Improved Measurement System l How to Improve Measurement Systems in an Organization

slide 13 Case Study Background l Internal Raw Material, A1, is necessary for Final Product production  Expensive Raw Material to produce – produced at 4 locations Worldwide  Cost savings can be derived directly from improved product quality, CpKs  Internal specifications indirectly linked to financial targets for production costs are used to calculate CpKs  If CTQ1 of A1 is too low, then more A1 material is added to achieve overall quality – higher quality means less quantity is needed – this is the project objective l High Impact Six Sigma project was chartered to improve an important quality variable, CTQ1 l The measurement of CTQ1 was originally not questioned, but the team decided to study the effectiveness of this measurement  The %GR&R, P/T ratio, and Bias were studied  Each of the Worldwide locations were involved in the study l Initial project improvements have somewhat equalized performance across sites. Small level improvements are masked by the measurement effectiveness of CTQ1

slide 14 CTQ1 MSA Study Design (Crossed) Site 1 Lab 6 analyses/site/sample 2 samples taken from each site 2*4 Samples should be representative Each site analyzes other site’s sample. Each plant does 48 analyses 6*8*4=196 analyses Site 1 Sample 1 Site 1 Sample 2 Op 1 Op 2Op 3 T1 T2 Site 2 LabSite 3 Lab Site 4 Lab Site 2 Sample 1…..

slide 15 CTQ1 MSA Study Results (Minitab Output)

slide 16 CTQ1 MSA Study Results (Minitab Session) Source DF SS MS F P Sample Operator Operator*Sample Repeatability Total %Contribution Source VarComp (of VarComp) Total Gage R&R Repeatability Reproducibility Operator Operator*Sample Part-To-Part Sample, Operator, & Interaction are Significant

slide 17 CTQ1 MSA Study Results Site%GRR P/T Ratio R-bar Equal Variances within Groups Mean Differences (Tukey Comp.) All 94.3 (78.6 – 100)* No (0.004)Only 1,2 No Diff. Site (30.0 – 47.6) Yes (0.739)All Pairs No Diff. Site (70.7 – 100) Yes (0.735)Only 1,2 Diff. Site (60.8 – 94.8) Yes (0.158)All Pairs No Diff. Site (64.8 – 100) Yes (0.346)Only 2,3 No Diff. *Conf Int not calculated with Minitab, Based upon R&R Std Dev

slide 18 CTQ1 MSA Study Results (Minitab Output) Site 1 Site 2 Site 3 Site 4 Dotplot of All Samples over All Sites

slide 19 CTQ1 MSA Study Results (Minitab Session) Analysis of Variance for Site Source DF SS MS F P Site Error Total Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev Site (---*---) Site (---*---) Site (---*---) Site (---*---) Pooled StDev = Site and Operator are closely related

slide 20 CTQ1 MSA Study Results (Minitab Output) X-bar R of All Samples for All Sites Most of the samples are seen as “noise” Discrimination Index is “0”, however can probably see differences of 5

slide 21 CTQ1 MSA Study Results (Minitab Output) Mean differences are seen in X-bar area Most of the samples are seen as “noise” X-bar R of All Samples for Site 4

slide 22 CTQ1 MSA Study Results – Process Linkage Site 2 Example 2002 Historical Process Results with Mean = MSA Study Results with Mean = Selected Samples are Representative

slide 23 CTQ1 MSA Study Results – Process Linkage Site 2 Example 2002 Historical Process Results with Range = Calc for pt to pt MSA Study Results with Range = 17.92, Calc for Subgroup When comparing the MSA with process operation, a large percentage of pt-to-pt variation is MS error (70%) --- a back check of proper test sample selection

slide 24 CTQ1 MSA Study Results – Process Linkage Site 2 Example Use Power and Sample Size Calculator with and without impact of MS variation. Lack of clarity in process improvement work, results in missed opportunity for improvement and continued use of non-optimal parameters l Key issue for Process Improvement Efforts is “When will we see change?”  Initial Improvements to A1 process were made  Control Plan Improvements to A1 process were initiated  Site 2 Baseline Values were higher than other sites  Small step changes in mean and reduction in variation will achieve goal l How can Site 2 see small, real change with a Measurement System with 70+% GR&R?

slide 25 CTQ1 MSA Study Results – Process Linkage Site 2 Example Simulated Reduction of Pt to Pt variation by 70% decreases time to observe savings by over 9X. 2-Sample t Test Alpha = 0.05 Sigma = Sample Target Actual Difference Size Power Power Sample t Test Alpha = 0.05 Sigma = 6.67 Sample Target Actual Difference Size Power Power

slide 26 CTQ1 MSA Study Results – Process Linkage Site 2 Example Benefits of An Improved MS l Realized Savings for a Process Improvement Effort  For A1, an increase of 1 number of CTQ1 is approximately $1 per ton  Change of 10 numbers, 1000 Tons produced in 1 month (832  842)  $1 * 10 * 1000 = $10,000 l More trust in all laboratory numbers for CTQ1 l Ability to make process changes earlier with R-bar at 6.67  Previously, it would be pointless to make any process changes within the 22 point range. Would you really see the change? l As the Six Sigma team pushes the CTQ1 value higher, DOEs and other tools will have greater benefit

slide 27 Objectives l The Hidden Factory Concept  What is a Hidden Factory?  What is a Measurement System’s Role in the Hidden Factory? l Review Key Measurement System metrics including %GR&R and P/T ratio l Case Study at W. R. GRACE  Measurement Study Set-up and Minitab Analysis  Linkage to Process  Benefits of an Improved Measurement System l How to Improve Measurement Systems in an Organization

slide 28 Measurement Improvement in the Organization l Initial efforts for MS improvement are driven on a BB/GB project basis  Six Sigma Black Belts and Green Belts Perform MSAs during Project Work  Lab Managers and Technicians are Part of Six Sigma Teams  Measurement Systems are Improved as Six Sigma Projects are Completed l Intermediate efforts have general Operations training for lab personnel, mostly laboratory management  Lab efficiency and machine set-up projects are started  The %GR&R concept has not reached the technician level l Current efforts enhance technician level knowledge and dramatically increase the number of MS projects  MS Task Force initiated (3 BBs lead effort)  Develop Six Sigma Analytical GB training  All MS projects are chartered and reviewed; All students have a project  Division-wide database of all MS results is implemented

slide 29 Measurement Improvement in the Organization l Develop common methodology for Analytical GB training

slide 30 Final Thoughts l The Hidden Factory is explored throughout all Six Sigma programs l One area of the Hidden Factory in Production Environments is Measurement Systems l Simply utilizing Operations Black Belts and Green Belts to improve Measurement Systems on a project by project basis is not the long term answer l The GRACE Six Sigma organization is driving Measurement System Improvement through:  Tailored training to Analytical Resources  Similar Six Sigma review and project protocol  Communication to the entire organization regarding Measurement System performance  As in the case study, attaching business/cost implications to poorly performing measurement systems