1. Reference Value Estimators in Key Comparisons Margaret Polinkovsky Advisor: Nell Sedransk NIST May 2004

2. Statistics, Metrology and Trade Statistics Estimation for measurements (1 st moment) Attached Uncertainty (2 nd moment) Incredible precision in National Metrology Institute (NMI) Superb science Exquisite engineering Statistical analysis

3. What are Key Comparisons? Each comparability experiment Selected critical and indicative settings – “Key” Tightly defined and uniform experimental procedures Purpose Establish degree of equivalence between national measurement standards Mutual Recognition Arrangement (MRA) 83 nations Experiments for over 200 different criteria

4. Seller a Buyer b SpecificationsRequirements Measurements NMI b NMI a Comparability NMI: National Metrology Institute products money Equivalence

5. Elements of Key Comparisons Key points for comparisons Experimental design for testing Participating NMIs Measurement and procedure for testing Statistical design of experiments Analysis of target data Statistical analysis of target data Scientific review of measurement procedure

6. Issues for Key Comparisons Goals: To estimate NMI-NMI differences To attach uncertainty to NMI-NMI differences To estimate Key Comparison Reference Value (KCRV) To establish individual NMI conformance to the group of NMIs To estimate associated uncertainty Complexity Artifact stability; Artifact compatibility; Other factors Pilot NMI NMIs

7. Statistical Steps Step 1 Design Experiment (statistical) Step 2 Data collected and statistically analyzed Full statistical analysis Step 3 Reference value and degree of equivalence determined Corresponding uncertainties estimated

8. Present State of Key Comparisons No consensus among NMIs on best choice of procedures at each step Need for a statistical roadmap Clarify choices Optimize process

9. Outcomes of Key Comparisons Idea “True value” Near complete adjustment for other factors Model based, physical law based Non-measurement factors Below threshold for measurement Precision methodology assumptions Highly precise equipment used to minimize variation Repetition to reduce measurement error

10. Outcomes of Key Comparisons Each NMI: Observation= “True Value”+ measurement +non-measurable error error same for all NMIs varies for NMI varies for NMI (after adjustment,if any ) data based estimate different expert for each NMI common artifact “statistical uncertainty“ “non-statistical or physical event uncertainty” Combined uncertainty Goal Estimate “True Value”: KCRV Estimated combined uncertainty and degrees of equivalence

11. Problems to Solve Define Best Estimator for KCRV Data from all NMIs combined Many competing estimators Unweighted estimators Median ( for all NMIs) Simple mean Weighted by Type A Weighted by 1/Type A (Graybill-Deal) Weighted by both Type A and Type B Weighted by 1/(Type A + Type B) (weighted sum) DerSimonian-Laird Mandel-Paule

12. Role of KCRV Used as reference value “95% confidence interval” Equivalence condition for NMI

13. Research Objectives Objectives Characterize behavior of 6 estimators Examine differences among 6 estimators Identify conditions governing estimator performance Method Define the structure of inputs to data Simulate Analyze results of simulation Estimator performance Comparison of estimators

14. Model for NMI i  Reference value: KCRV   (Laboratory/method) bias : Type B   (Laboratory/method) deviation : Type B   Measurement error: Type A 

15. Process  2 =  2 +   2 (Type B)  = systematic bias   2 = extra-variation NMI y =  +  +   * +  s 2 = data-based variance estimate (Type A)  = experiment-specific bias   * = experiment-specific deviation  = random variation Data y s 2 Expertise  2 Sources of Uncertainty NMI Process

16. Translation to Simulation  Scientist: unobservable  Systematic Bias ~N(, ) Simulate random  Extra-variation Simulate random  Data: observable  Observed “Best Value”  Variance estimate s 2 ~ 1/df( ) Simulate random  Experimental Bias Simulate random  Experimental Deviation Simulate random Uncertainty Type A: s 2 Type B:

20. Conclusions and Future Work Conclusions Uncertainty affects MSE more than Bias Estimators performance Graybill-Deal estimator is least robust Dersimonian-Laird and Mandel-Paule perform well When 1 NMI is not exchangeable the coverage is effected Number of labs changes parameters Future work Use on real data of Key Comparisons Examine other possible scenarios Further study degrees of equivalence Pair-wise differences

21. Looking Ahead Use on real data of Key Comparisons Examine other possible scenarios Further study degrees of equivalence Pair-wise differences

22. References R. DerSimonian and N. Laird. Meta-analysis in clinical trials. Controlled Clinical Trials, 75:789-795, 1980 F. A. Graybill and R. B. Deal. Combining unbiased estimators. Biometrics, 15:543-550, 1959 R. C. Paule and J. Mandel. Consensus values and weighting factors. Journal of Research of the National Bureau of Standards, 87:377-385 P.S.R.S. Rao. Cochran’s contributions to the variance component models for combining estimators. In P. Rao and J. Sedransk, editors. W.G. Cochran’s Impact on Statistics, Volume II. J. Wiley, New York, 1981 A. L. Rukhin. Key Comparisons and Interlaboratory Studies (work in progress) A. L. Rukhin and M.G. Vangel. Estimation of common mean and weighted mean statistics. Jour. Amer. Statist. Assoc., 73:194-196, 1998 J.S. Maritz and R.G. Jarrett. A note on estimating the variance of the sample median. Jour. Amer. Statist. Assoc., 93:303-308, 1998 SED Key Comparisons Group (work in progress)