ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève 1 A review of the state of the art, present norms and future trends in the field of measurement uncertainty Walter Bich INRIM – Istituto nazionale di ricerca metrologica Torino (Italia) JCGM-WG1

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 2 Uncertainty of measurement 1st PACMAN Workshop 2-4 February 2015, CERN Genève measurement uncertainty uncertainty of measurement uncertainty parameter characterizing the dispersion of the values being attributed to a quantity, based on the information used standard measurement uncertainty standard uncertainty standard deviation of the random variable describing the state of knowledge about a quantity

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 3 Some dates 1st PACMAN Workshop 2-4 February 2015, CERN Genève BIPM questionnaire on uncertainties 1980 Recommendation INC Establishment of WG3 on uncertainties under ISO TAG4: BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, OIML 1981 Recommendation CI Recommendation CI Guide to the expression of uncertainty in measurement (GUM) 1995 Reprint with minor corrections 1997 Establishment of the Joint Committee for Guides in Metrology, JCGM. ILAC joins in 1998

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 4 Joint Committee for Guides in Metrology 1st PACMAN Workshop 2-4 February 2015, CERN Genève Present Chair: the BIPM Director The JCGM has two working groups (WGs) WG 1 “Expression of uncertainty in measurement”, has the task “to promote the use of the GUM and to prepare Supplements for its broad application” WG 2 “on International vocabulary of basic and general terms in metrology”, has the task “to revise and promote the use of the VIM” See also In the following, emphasis will be on documents from WG1

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 5 Published documents Common banner: Evaluation of measurement data 1st PACMAN Workshop 2-4 February 2015, CERN Genève Supplement 1 to the “Guide to the expression of uncertainty in measurement” — Propagation of distributions using a Monte Carlo method. JCGM 101:2008 Guide to the expression of uncertainty in measurement. JCGM 100:2008 (GUM 1995 with minor modifications) Supplement 2 to the “Guide to the expression of uncertainty in measurement” — Extension to any number of output quantities. JCGM 102:2011

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 6 Published documents II 1st PACMAN Workshop 2-4 February 2015, CERN Genève Role of measurement uncertainty in conformity assessment. JCGM 106:2012 An introduction to the “Guide to the expression of uncertainty in measurement” and related documents. JCGM 104:2009

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 7 Documents in preparation 1st PACMAN Workshop 2-4 February 2015, CERN Genève Examples of uncertainty evaluation. JCGM 110:201X Supplement 3 to the “Guide to the expression of uncertainty in measurement” — Modelling. JCGM 103:20XX Guide to uncertainty in measurement. JCGM 100:201X

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 8 Documents planned 1st PACMAN Workshop 2-4 February 2015, CERN Genève Concepts and basic principles of uncertainty evaluation Bayesian methods Applications of the least-squares method

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 9 JCGM 104:2009 1st PACMAN Workshop 2-4 February 2015, CERN Genève It should also help the reader to decide which of the JCGM documents is appropriate for his specific application. As the title says, it is a «soft» introduction to the GUM and related documents, but also to the topic of uncertainty. Formulae are kept to a minimum.

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 10 JCGM 106:2012 1st PACMAN Workshop 2-4 February 2015, CERN Genève It deals with items having a single scalar property with a requirement given by one or two tolerance limits, and a binary outcome in which there are only two possible states of the item, conforming or non-conforming, and two possible corresponding decisions, accept or reject. This is not a guidance document on evaluating uncertainty, but on using uncertainty in a specific application, conformity assessment based on measurement result.

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 11 JCGM 106:2012 1st PACMAN Workshop 2-4 February 2015, CERN Genève This document addresses the technical problem of calculating the conformance probability and the probabilities of the two types of incorrect decisions, given a probability density function (PDF) for the measurand, the tolerance limits and the limits of the acceptance interval. It adopts a Bayesian view of measurement uncertainty. It considers topics such as decision rules, and the different types of risk.

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 12 GUM – JCGM 100:2008 and 100:201X First publication in st PACMAN Workshop 2-4 February 2015, CERN Genève Until now, a large number of documents based on the GUM has been written. The GUM has been translated into many languageslanguages Reprint in 1995 with some corrections JCGM 100:2008 (free of charge) GUM 1995 with minor modifications In addition, the GUM has been adopted as a standard, in some cases as a law, in many countriesstandard

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève 13 Courtesy BIPM

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève 14 t.aspx

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 15 Caution 1st PACMAN Workshop 2-4 February 2015, CERN Genève The same uppercase symbol is used with two meanings: A (physical, chemical…) quantity The associated random variable used to describe the experimental outcome Estimates (i.e., measured values), are denoted by the corresponding lowercase symbol

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève The GUM method 16 A measurement model is requested, relating the measurand Y to the input quantities X i

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève The GUM method II 17

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève The GUM method III 18 The method provides reliable standard uncertainties in the vast majority, not in the totality of cases

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 19 Drawbacks of the GUM method No guidance on the (frequent) case of many measurands Poor guidance on the construction of a coverage interval (emphasis is on standard uncertainty), limited to a situation optimistically considered as frequently occurring Other (comparatively minor) weak sides, such as poor consideration to –non-symmetric distributions –non-linear measurement models The cases above are difficult, probably they had not been considered in the first edition on purpose 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 20 Are the cases not covered in the GUM of practical importance? Any calibration of a set of artefacts, be they weights, capacitors, gauge blocks or similar, is a multivariate case 1st PACMAN Workshop 2-4 February 2015, CERN Genève The CIPM MRA asks for CMCs at the 95 % coverage probability, i.e, CMCs are coverage intervals

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 21 Remedies Difficult problems typically imply difficult solutions Coverage interval (and more): JCGM 101:2008 (Supplement 1) Multivariate case: JCGM 102:2011 (Supplement 2) 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 22 Supplement 1 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 23 Supplement 1 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 24 Supplement 1 1st PACMAN Workshop 2-4 February 2015, CERN Genève The (numerical) mechanism is the Monte Carlo method:

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 25 Supplement 1 1st PACMAN Workshop 2-4 February 2015, CERN Genève Extensive guidance given on: assigning PDFs based on available knowledge, and sampling at random from these PDFs

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 26 Example (from Supplement 1) 1st PACMAN Workshop 2-4 February 2015, CERN Genève Red dotted: GUM Solid blue: S1

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 27 Supplement 1 1st PACMAN Workshop 2-4 February 2015, CERN Genève Asymmetric PDFs, as well as coverage intervals asymmetric about the mean, arise naturally

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 28 Supplement 2 1st PACMAN Workshop 2-4 February 2015, CERN Genève Generalizes to the multivariate case: The law of propagation of uncertainties, and

ISTITUTO NAZIONALE DI RICERCA METROLOGICA Side effects of remedies 29 1st PACMAN Workshop 2-4 February 2015, CERN Genève The GUM and its Supplements are now inconsistent Why didn’t we write Supplements consistent with the GUM?

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 30 The GUM is ambiguous 1st PACMAN Workshop 2-4 February 2015, CERN Genève parameter characterizing the dispersion of the values being attributed to a quantity, based on the information used This is an intrinsically Bayesian view of uncertainty – uncertainty concerns the measurand The definition contrasts with the way in which uncertainty is obtained, essentially frequentist – uncertainty concerns the measurand estimate, and is itself uncertain The definition of uncertainty in the GUM is

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 31 Supplements are unambiguous In Supplement 1, PDFs (probability density functions) are used to describe the state of knowledge about each input quantity Accordingly, the state of knowledge about the measurand is described by a PDF obtained from those of the input quantities through the measurement model (in a way that is not relevant here) This is an intrinsically Bayesian attitude, and is consistently adopted throughout the Supplements No alternative was possible! 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 32 JCGM 100:201X Main purpose: to make the GUM consistent with its Supplements 1st PACMAN Workshop 2-4 February 2015, CERN Genève Secondary purposes: to make it consistent as much as possible with VIM3 to broaden its applicability to “new” needs to minimize notational and terminological ambiguities

ISTITUTO NAZIONALE DI RICERCA METROLOGICA Alignment with Supplements Uncertainties (and estimates) are: –estimates of moments of frequency distributions, in the current GUM (they have degrees of freedom) –exact moments of state-of-knowledge distributions, in the Supplements (no degrees of freedom) In the revised GUM JCGM 100:201X, uncertainties (and estimates) will be exact moments of state-of-knowledge distributions, as in the Supplements 33 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA Practical impact on standard uncertainty 34 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA Practical impact on coverage intervals In the revised GUM there will be mostly generic guidance on the construction of coverage intervals, this task being given to Supplement 1 Distribution-free coverage intervals, based on Chebyshev or Gauss inequalities, will be given Expanded uncertainty de-emphasized Greater consideration to non-symmetric coverage intervals Possible impact on KCDB, Appendix C 35 1st PACMAN Workshop 2-4 February 2015, CERN Genève

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 36 Cosmetic changes Suffix «c» in the combined standard uncertainty u c dropped (as in JCGM 101, JCGM 102 and JCGM 106) 1st PACMAN Workshop 2-4 February 2015, CERN Genève New notation u x allowed as an alternative to u(x)

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 37 Further notable features Increased guidance on the evaluation of input uncertainties 1st PACMAN Workshop 2-4 February 2015, CERN Genève Guidance on the evaluation of input covariances and on their effect on the output uncertainty Enhanced examples. Examples concerning the GUM and its Supplements will be collected in a separate document, JCGM 110. Help from users needed! Clarification of the meaning of loose expressions such as «uncertainty of…» or «covariance between quantities» Increased guidance on reporting and recording a measurement result

ISTITUTO NAZIONALE DI RICERCA METROLOGICA If your experiment needs statistics, you ought to perform a better experiment (Lord Rutherford) 1st PACMAN Workshop 2-4 February 2015, CERN Genève 38

ISTITUTO NAZIONALE DI RICERCA METROLOGICA 1st PACMAN Workshop 2-4 February 2015, CERN Genève Thank you for your attention 39