User’s Guide to the ‘QDE Toolkit Pro’ National ResearchConseil national Council Canadade recherches Excel Tools for Presenting Metrological Comparisons by B.M. Wood, R.J. Douglas & A.G. Steele Chapter 9. Graphing and Pooling Measurement Distributions (ii) Sept 5, 2003 In this chapter, we present the use of the QDE Toolkit Pro’s facilities for graphing distributions and pooled distributions. We also present some summary statistics, calculated by the QDE Toolkit Pro, that are useful even before a KCRV is chosen. Ch 9: 111

Similarly, the x-axis range, etc. can be adjusted. There is a very wide range in x of data points available (~1000 uniformly spaced points across the default graph range and ~1000 more on an expanding mesh to cover ~200 x the breadth (for integrating Student distribution tails). After running the macro tk_pool_PlotBuilder, the graphs can be adjusted with Excel. Here’s a quick review of some things you will want to do: Point at a graph, left mouse click to select it and pop it to the top of the stacked graphs. Point at a corner or edge of a selected graph and drag (left mouse button) to resize. Double-click on the selected chart’s y axis (the vertical line at x=0) to get a dialog box, select the scale tab, enter the new y maximum... Tip: We find Excel’s undo capability is very useful in reversing unwanted formatting of graphs. Select Edit|Undo with the mouse, or ‘control Z’ from the keyboard. QDE Toolkit Pro Graphs - working with these Pooled Distributions Ch 9: 112

After running the macro tk_pool_PlotBuilder, the graphs’ data can be edited or recalculated with Excel (for example to convolute with either a uniform or a “goalpost” distribution). The data (~2000 rows x ~10-20 columns) are extensive enough so that some Excel tricks (shift Edit|Paste Picture, or using F9 to convert a graph series into an explicit formula array) for taking a frozen picture of a graph no longer always work well. It is often very helpful to take a frozen snapshot of a graph, and going beyond screen resolution can be a challenge! After getting the Excel graph about right, one trick seems to give the best for-printing conversion to Windows metafile form… (continued on next page) QDE Toolkit Pro Graphs - working with these Pooled Distributions Prior to Excel 97 SR2, Excel had a memory leak when creating lots of charts that could give a “not enough memory” error and required restarting Windows occasionally. Ch 9: 113

For Office XP, we copy the selected graph to the clipboard – Control-C Paste it into Microsoft Word – Control-V, then click on the clipboard icon beside the pasted graph to select the “Paste Options” select “Picture of Chart (smaller file size)” Press the Esc key to get rid of the icon Select and copy the graph in Word Paste it back into Excel as a “frozen picture”. For Office 97, or for Office 2000, we copy the selected graph to the clipboard, and do a “Edit |Paste Special” into Microsoft Word as a “Picture (Enhanced Metafile)”. Then, copying from this graph in Word, it can be pasted (not Paste Special this time) back into Excel as a “frozen picture”. QDE Toolkit Pro Graphs – freezing a graph (as a picture) Ch 9: 114

The macro tk_pool_PlotBuilder can also graph the PDFs of the candidate KCRVs. The RVs are graphed with thick lines. In column A of the input table, the block of reference values starts beneath the block of Lab comparison data, with the first name containing the character pair RV (case insensitive). It ends with the first blank in column A. QDE Toolkit Pro Graphs - working with Reference-Value Distributions Here, two RVs are graphed. One is labeled the KCRV, with no uncertainty given. (This is sometimes done to avoid profound difficulties, for example by the CCT.) This first RV (its value happens to be the simple mean of the 8 pooled Labs) is plotted as a delta-function (really.001 of the initial graph width). The second RV is an inverse- variance weighted mean of the 8 pooled labs, with its formal standard uncertainty (the same as the product of PDFs, since they are all normal) and correlation coefficients with the contributor labs (which are not used in plotting the RV’s PDF). Ch 9: 115

After running the macro tk_pool_PlotBuilder, there is a block of supplementary information in the 16 columns to the right of the correlation coefficient matrix. We will discuss them in turn, but here’s the overview: QDE Toolkit Pro Graphs - working with the “supplementary information” Col after r ij 1 Lab names: comments hold reduced chi-square of differences with all other pooled Labs. 2-4 Input Comparison Data for Labs, similar statistics for pools 5-6  u/u for degrees of freedom: variance-based and tail-based. 7-8 Coverage Factors for 68.0% and 95.0% confidence (from trapezoidal integration of the distributions). Mostly of interest for the pooled distributions Error in the symmetric vs rigorous confidence intervals. Mostly of interest for quantifying the un- importance of this effect for the asymmetric pooled distributions Mean, Median and (first) Mode of each distribution. Mostly of interest for the pooled distributions, 16 Lab Names, just for convenience Tip: the Toolkit Pro’s output italic numbers are unitless, and regular font numbers have units. Ch 9: 116

QDE Toolkit Pro Graphs - col 1 of the “supplementary information” Col after r ij 1 The comment in the top row of the column contains the date and time of creation of the block of supplementary information. The column is of Lab names for each row, but some really useful information is in the comments of n “in-pool” labs, which give the reduced chi- square statistical information about their rms En with all other (n-1) “in-pool” Labs: (n-1) terms, between 1 and (n-1) degrees of freedom. The bottom (“Pair Difference”) name has a comment that gives the all-pairs chi-squared for the pool: there are n(n-1) terms to sum, with an obvious 2:1 redundancy: Despite having n(n-1) terms, the degrees of freedom is still (n-1). Note that this chi-square is independent of any choice of KCRV. Its use is illustrated in: Hill, Steele and Douglas, Metrologia 39, 269 (2002). It is also discussed in a bit more detail on the following page. It uses the Excel statistical function ChiDist to convert from an all-pairs-variance to a probability that it could be exceeded by chance. 1 Ch 9: 117

QDE Toolkit Pro Graphs - VERY USEFUL INFORMATION about Lab j Ch 9: 118 The chi-squared of each of N in-pool Labs, with respect to the N-1 other Labs, is calculated and put into this column as comment boxes. This is the Lab’s “rms E n ”:  j 2 = (N-1) -1  i=1 N i≠j (x i – x j ) 2 / (u i 2 + u j 2 - 2r ij u i u j ) If the differences are independent and normally distributed about zero, then this is EXACTLY a reduced chi- squared statistic with N-1 degrees of freedom. It is perhaps the best test as to whether Lab j agrees with the “rest of the world”, and is independent of any choice of KCRV. In practice, the differences are only independent when the other Labs’ uncertainties dominate, and the degrees of freedom for  j 2 may be as small as 1 when u j dominates. If the differences were independent and there was perfect agreement, within the stated uncertainties, for Lab j, this value of  j 2 is expected to be exceeded by chance with only this probability: a minimum probability that may be a substantial underestimate if the differences’ lack of independence is considered. In the Toolkit Version 2.07, the probability, between 0 and 1, is given in scientific number format.

QDE Toolkit Pro Graphs - VERY USEFUL INFORMATION about comparison Ch 9: 119 The reduced chi-squared Labs j with respect to the N-1 other Labs, is:  j 2 = (N-1) -1  i=1 N i≠j (x i – x j ) 2 / (u i 2 + u j 2 - 2r ij u i u j ) By averaging the N  j 2 ’s,  2 = (N) -1  i=1 N  j 2 the all-pairs variance. It is perhaps the best test as to whether these Labs agree with each other within their uncertainties, and is independent of any choice of KCRV. The degrees of freedom of this reduced chi-squared is N-1. If there were perfect agreement, within the stated uncertainties, for all Labs, this value of  2 is expected to be exceeded by chance with only this probability. If this probability is very small, then the measured differences are not described very well by the stated uncertainties. In the Toolkit Version 2.07, the probability, between 0 and 1, is given in scientific number format.

QDE Toolkit Pro Graphs - VERY USEFUL INFORMATION – a TIP Ch 9: 120 There are so many comment boxes that the workbook needs to have its comments hidden unless mouse is pointing to a commented cell. (Tools | Options |View | and select “Comment indicator only”. ) To keep one or more comment boxes on display, Right-mouse-click on the cell Select “Show comment” To hide one or more comment boxes that is on display, Right-mouse-click on the cell Select “Hide comment”

QDE Toolkit Pro - Using Pair Differences The bilateral pair differences of laboratories can incorporate all knowledge about the pairs, including correlations, so the reduced chi- square of all pair differences is an appropriate tool to consider for characterizing a comparison model without having to choose a specific reference value. The “all-pairs variance” or APV of a comparison with N laboratories can be calculated APV =  [   i =1 N  j =1 N (x j - x i ) 2 / (u i 2 + u j r ij u i u j ) ] / (N(N-1)) for normal PDF’s for the x i ’s, the APV is distributed as a reduced chi- square with N-1 degrees of freedom. Aside: Here we use the name APV here since it sometimes may be appropriate to interpret the APV in terms of its model distribution, which is not necessarily a reduced chi-squared if we include the effects of the stated effective degrees of freedom for each laboratory. In the QDE Toolkit Pro Versions 2.04…2.07, the analysis is all in terms of the normal distribution limit and the reduced chi-squared. Ch 9: 121

QDE Toolkit Pro - Using Pair Differences The “all-pairs variance” or APV of a comparison with N laboratories is APV =  [  i =1 N  j =1 N (x j - x i ) 2 / (u i 2 + u j r ij u i u j ) ] / (N(N-1)) for normal PDF’s for the x i ’s, the APV is distributed as a reduced chi- square with N-1 degrees of freedom. Ch 9: 122 At the left, the Monte Carlo simulation of the distribution of the APV of a 12-Lab comparison is compared with the reduced chi-squared curves having 10, 11 and 12 degrees of freedom. Although the APV sum is over 132 non- zero terms, with 66 terms that look distinct, the distribution of APVs is just the reduced chi-squared distribution with 11 degrees of freedom. The APV has been constructed to be independent of any change of reference value. Although we may have not yet chosen a specific KCRV, one degree of freedom is used by this possibility.

QDE Toolkit Pro - Using Pair Differences The “all-pairs variance” or APV of a comparison with N laboratories is the master chi-squared for the comparison. This is true in the sense that … if a comparison has failed the APV chi-squared test, then NO choice of reference value can, by itself, rescue the comparison… This can be demonstrated by considering what happens to the pair differences when the reference value is changed: the pair differences are invariant, and so the APV chi squared statistic is also invariant. Ch 9: 123

QDE Toolkit Pro Graphs - col 2-4 of the “supplementary information” Col after r ij 2-4 Input Comparison Data for Labs: value, uncertainty, degrees of freedom. In columns 2 and 3, similar stastistical information is calculated for the pools, and explained in comments. Degrees of freedom is just as input, with a default to “normal. 4 Ch 9: 124 3

QDE Toolkit Pro Graphs - col 5-6 of the “supplementary information” Col after r ij 5  u/u for degrees of freedom: estimate based on the chi distribution’s variance- about-the-mean based and tail-based. The overly broad tails of the Student distribution at low degrees of freedom arise from the small arguments (and large value) of the chi distribution. Approximate fractional uncertainty in the standard uncertainty from variance of the chi-square family of distributions that are highly asymmetric for degrees of freedom less than 10. See ISO Guide Eq.E-7 & Table E-1. 6  u/u for degrees of freedom Improved estimate  u for the fractional uncertainty in the standard uncertainty, derived from the correct inverse-chi distribution's interval [0,u+  u] with 84% confidence (the same  u as Col 5 in the limit of large degrees of freedom). See discussion on pages of this Guide, in the context of discussing Tables of Equivalence. 5 6 Two definitions for degrees of freedom? No. The variance-based method is simply a bad approximation for degrees of freedom < 10, if the degrees of freedom is to be used for evaluating coverage factors from Student distributions. Nonetheless, we suggest using the symbol S for a degrees of freedom aimed at describing the tails of the Student distribution. Fortunately, in precision metrology, usually > 10 and there is no difficulty. Ch 9: 125

QDE Toolkit Pro Graphs - col 7-8 of the “supplementary information” Col after r ij 7-8 Coverage Factors for 68.0% and 95.0% confidence (from trapezoidal integration of the distributions). Mostly of interest for the pooled distributions. Because the product PDF is narrow, there may be only a few tens of samples within ± . An easy way of monitoring this effect is to include a normal RV of about the same value and width. Any variation of the coverage factor from the expected norm (here +0.1%) is an indication of the accuracy if the product were over Student distributions instead of normal distributions. If the accuracy is not sufficient, in the Visual Basic code in module QDE_Toolkit_PlotBuilder, in subroutine tk_pool_PlotBuilder_With_Anchor, near comment line ‘C140, change poolpoints = 1002 to a larger number, such as poolpoints = (Excel will limit you to ~60000)... near comment line ‘C110 in the same subprogram Confidence_k1 = 0.68 Confidence_k2 = 0.95 can be edited to the values of your choice. These do not affect the MRA value of 95% confidence. 7 8 Ch 9: 126

QDE Toolkit Pro Graphs - col 9-12 of the “supplementary information” Col after r ij 9-12 Error in the symmetric vs rigorous confidence intervals. For the symmetric input data, you see the effects of round-off in the trapezoidal integration. These columns are mostly of interest for quantifying the un-importance of asymmetry for the asymmetric pooled distributions: “How close are the intervals {mean-u, mean+u] and [mean-U, mean+U] to the true asymmetric confidence intervals, starting for example at X where CDF(X)=0.025 and running to X’ where CDF(X’)= Ch 9: 127

QDE Toolkit Pro Graphs - col of the “supplementary information” Col after r ij Mean, Median and (rightmost) Mode (the zero-slope peak) determined from the table of each distribution. Now this is mostly of interest for the pooled distributions, So as you see, there’s lots of information included in the “supplementary information” columns that is eminently ignore-able, most of the time. If and when it is wanted, it will be waiting! Ch 9: 128