2016/6/12 Discussion on DUT Uncertainty at CCM-WGFF Chun-Min Su CMS/ITRI, Chinese Taipei

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (1) Paragraph N5 of MRA-D-04 says “Contributions to the uncertainty stated on the calibration certificate include the measured performance of the device under test during its calibration at the NMI or accredited laboratory. CMC uncertainty statements anticipate this situation by incorporating agreed-upon values for the best existing devices.” BED

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (2) Recommendations in DOCUMENT JCRB-8/9 “Uncertainty contributions of the device under calibration or measurement” by Ad hoc JCRB Working Group CMC Uncertainties: 1.It is recommended to exclude contributions to the CMC uncertainty caused by the customer device before or after its calibration or measurement at the institute. 2.It is recommended to include contributions to the CMC uncertainty caused by the best ordinarily available customer device during its calibration or measurement at the institute. In general, these values will be published in Appendix C of the CIPM MRA. For individual certificates the actual characteristics of the customer device must be considered.

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (3) ILAC Policy for Uncertainty in Cal. (Clause 5.4) –Calibration laboratories shall provide evidence that they can provide calibrations to customers in compliance with 5.1 b) so that measurement uncertainties equal those covered by the CMC. In the formulation of CMC, laboratories shall take notice of the performance of the “best existing device” which is available for a specific category of calibrations. –A reasonable amount of contribution to uncertainty from repeatability shall be included and contributions due to reproducibility should be included in the CMC uncertainty component, when available. There should, on the other hand, be no significant contribution to the CMC uncertainty component attributable to physical effects that can be ascribed to imperfections of even the best existing device under calibration or measurement. ILAC-P14:12/2010

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (4) –It is recognized that for some calibrations a “best existing device” does not exist and/or contributions to the uncertainty attributed to the device significantly affect the uncertainty. If such contributions to uncertainty from the device can be separated from other contributions, then the contributions from the device may be excluded from the CMC statement. For such a case, however, the scope of accreditation shall clearly identify that the contributions to the uncertainty from the device are not included. –NOTE: The term “best existing device” is understood as a device to be calibrated that is commercially or otherwise available for customers, even if it has a special performance (stability) or has a long history of calibration. ILAC-P14:12/2010

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (5) ILAC Policy for Uncertainty in Calibration (Clause 6.4) –Contributions to the uncertainty stated on the calibration certificate shall include relevant short-term contributions during calibration and contributions that can reasonably be attributed to the customer’s device. Where applicable the uncertainty shall cover the same contributions to uncertainty that were included in evaluation of the CMC uncertainty component, except that uncertainty components evaluated for the best existing device shall be replaced with those of the customer’s device. Therefore, reported uncertainties tend to be larger than the uncertainty covered by the CMC. Random contributions that cannot be known by the laboratory, such as transport uncertainties, should normally be excluded in the uncertainty statement. If, however, a laboratory anticipates that such contributions will have significant impact on the uncertainties attributed by the laboratory, the customer should be notified according to the general clauses regarding tenders and reviews of contracts in ISO/IEC ILAC Policy for Uncertainty in Calibration (Clause 6.5) –As the definition of CMC implies, accredited calibration laboratories shall not report a smaller uncertainty of measurement than the uncertainty of the CMC for which the laboratory is accredited. ILAC-P14:12/2010

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (6) At it 10 th meeting, WGFF resolved “DUT uncertainty will be agreed upon by the WGFF, based on uncertainties of KC transfer standards, and included in the CMC uncertainty value.” The document “Proposed Definition of DUT Uncertainty” was generated on Dec. 6, 2011 and circulated among members of the WGFF

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (7) The same values of DUT uncertainties for different instruments, derived primarily from the KC reports, was proposed Measurand / Device Under Test DUT Uncertainty (k=2)UnitsComments K1 Water flow0.02% Based on CCM.ff-K1 report figures 5 to 8 and figures 19 to 22. K2 Hydrocarbon liquid flow 0.02% Based on CCM.ff-K2 report figures 2 and 4 to 7 showing repeated cals of TS at NEL. K3 Air speed /u(m/s ) % Based on APMP.ff-K3 TS reproducibility, see "Air Speed" worksheet. K5 High-pressure gas flow 0.05% Based on CCM.ff-K5b report table 2. K6 Low-pressure gas flow 0.05% From Wright Flomeko 2007 paper on the uncertainty of the CCM K6 TS.

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Background (8) Feedbacks to the Dec. 6, 2011 proposal –The numbers suggested by WGFF are sensible values but the inclusion of fixed figure is not acceptable. –Development of guidance on the expression of uncertainty in CMCs and how to include repeatability, reproducibility and inter-comparison contributions to uncertainty, and uncertainty budget template. –… 2016/6/129

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Ongoing Discussion in WGFF “WGFF Guidelines for CMC Uncertainty and Calibration Report Uncertainty” –Jun. 11, 2012 version Read and discussed in 2012 WGFF Colorado Springs, CO, USA Agreed on establishment of the document and most of the content Some revisions required and a review team formed –Jul. 24, 2012 version Circulated among members of the review team Comments were given –Sep. 19, 2012 version Circulated among members of the review team Comments being given 2016/6/1210

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (1) Define Calibration and Measurement Capability uncertainty (U CMC ) as the root-sum-of-squares (RSS) of: 1.the base uncertainty of the reference standard based on the GUM methodology; 2.the repeatability of calibration results measured using the best existing device. 2016/6/1211 U CMC represents the 95 % confidence level uncertainty of the average of n measurements from the reference standard. The number of measurements n should match the normal procedures routinely followed during calibration of a customer’s device.

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial The best way is to use the Welch-Satterthwaite Method and a coverage factor k 95 that is based on the effective degrees of freedom How to Calculate U CMC ?

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial To Welch-Satterthwaite or not to W-S? 4 methods were compared –Following Annex G of the GUM by using the W-S Method –Using the approach that EURAMET proposed –Using the formula –assuming that both components have infinite DoF 2016/6/1213 “Comparison of W-S and t-Value Methods” prepared by Dr. John Wright

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial To W-S or not to W-S: Comparison Result 2016/6/ /6/1214

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial To W-S or not to W-S: Recommendation The Guidelines call for the use of the Welch- Satterthwaite method or the U t95/2 method –The W-S method follows the specific GUM recommendations –The U t95/2 method should be allowed since it is simple to understand and the uncertainty values it produces are conservative 2016/6/1215 “Comparison of W-S and t-Value Methods” prepared by Dr. John Wright

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (2) CMC Uncertainty –Base uncertainty uncertainty of the quantity of fluid contained within, delivered, or passed through a device under test That each identified measurement, instrument, and influence factor is assessed for repeatability, stability, reproducibility, and historical performance is expected –Repeatability of the BED By the nature of the test, the result will be a combination of the repeatability of the chosen DUT and that of the facility 2016/6/1216

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (3) Procedure for Assessing Repeatability for a flow cal. 1.Establish the range of flows, temperatures, and other conditions listed in the scope of the facility to be covered. 2.Define a number of test flows and conditions which within the normal working practices, should demonstrate the least stable conditions and one or two which should demonstrate the most stable conditions. (The least stable results are generally found at the highest or lowest ends of the reference standard range.) 3.Identify the BED to use for each test. 2016/6/1217

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (4) 4.Working within the operating procedures of the laboratory to ensure each test can proceed without interruption within a working day, perform multiple calibrations under stable flows and conditions to produce performance indicator values for the BED. The number of repeats should match the normal practices used when calibrating a customer’s device. If procedures dictate less than twenty points, the effect of the smaller sample should be fully recognised in the analysis by using the Welch-Satterthwaite or t-value approaches described below. 2016/6/1218 Performance indicator may be error, meter factor, K-factor, discharge coefficient, etc.

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (5) 5.For each set of points, examine the data to determine the distribution to establish if a ‘normal’ or Gaussian distribution can be assumed. 6.If the distribution is approximately normal, calculate the standard deviation of the mean. If the sample size is twenty or greater, the standard deviation of the mean can be combined by RSS with the base uncertainty and expanded using a coverage factor of 2. If the sample is smaller than twenty then, 1) use the Welch-Satterthwaite method to determine the degrees of freedom and the coverage factor necessary to achieve a 95 % confidence level, or 2) multiply the standard deviation by the 95 % confidence level t-value for n-1, divide by 2 to give the standard uncertainty for RSS with other components, and use a coverage factor of 2 on the RSS to obtain the expanded uncertainty. If the distribution of the results is not normal or Gaussian, then further consideration and assessment is necessary to determine the correct statistical approach. 2016/6/1219

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (6) Procedure for Assessing Repeatability for a volume cal. 1.Establish the volume range and type of volumetric equipment covered in the scope of the calibration system. 2.Select the instruments (BED) to be tested. 3.Perform n tests of each instrument at the chosen volume, where n is the number of repeated measurements normally performed on a customer’s device. 4.Work within the operating procedures of the laboratory to ensure each test can proceed without interruption within a working day. 5.Take the standard deviation of the mean of the repeated measurements and 1) use the Welch-Satterthwaite method to determine the degrees of freedom and the coverage factor necessary to achieve a 95 % confidence level, or 2) multiply the standard deviation by the 95% confidence level t-value for n-1, divide by 2 to give the standard uncertainty for RSS with other components, and use a coverage factor of 2 for the expanded uncertainty. 2016/6/1220

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (7) Reported Uncertainty for the Performance Indicator (U PI ) –Include additional components over u base due to: instrumentation and characteristics associated with the DUT fluid properties (if applicable) repeatability or short-term reproducibility for the customer’s DUT 2016/6/1221 Note that Note: U PI must be > U CMC

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (8) –Associated Instrumentation and Property Uncertainties –Repeatability or Short-Term Reproducibility of the Customer’s DUT Acceptable methods for quantifying u reprd,DUT or repeatability are (in all cases, the method must be clearly stated in the report) –The Welch-Satterthwaite method and effective degrees of freedom applied to multiple measurements at each set point. –The standard deviation of the mean of multiple measurements at each set point ( ) with statistical corrections using the t-value for the finite sample size. –The standard deviation of multiple measurements at each set point, with weighting based on the t-value at 95 % confidence for the number of points and then divided by 2 to give the standard uncertainty for RSS with other components. 2016/6/1222

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial Introduction to the Guidelines (9) Correlation Methods –In some reference standards, two devices can be installed in series and calibrated at the same time. This allows the application of correlation techniques or Youden analysis to separate the facility and BED repeatability. –Separating the repeatability by correlation methods is acceptable and the facility portion can be used instead of u repeat,BED to calculate U CMC. In this case, a statement in the comments section is required, such as: “Contributions to the uncertainty from the device are not included.” 2016/6/1223 Note that

Copyright 2012 ITRI 工業技術研究院 建議字型：中文微軟正黑體，英文 Arial 2016/6/1224 Thank you for your attention!