1. Rick Walker Evaluation of Out-of-Tolerance Risk 1 Evaluation of Out-of-Tolerance Risk in Measuring and Test Equipment Rick Walker Fluke - Hart Scientific

2. Rick Walker Evaluation of Out-of-Tolerance Risk 2 Outline Introduction Individual risk Conditional probability Variables Equations Examples Check

3. Rick Walker Evaluation of Out-of-Tolerance Risk 3 Assumption Customers need their measuring and test equipment to perform in tolerance Risk that MTE is out of tolerance should be low

4. Rick Walker Evaluation of Out-of-Tolerance Risk 4 Definition Out-of-tolerance risk (for MTE): probability it is out of tolerance given that testing indicates in tolerance

5. Rick Walker Evaluation of Out-of-Tolerance Risk 5 Why out of tolerance? Calibration has error Electronic components drift Environment varies Measuring between calibration points

6. Rick Walker Evaluation of Out-of-Tolerance Risk 6 Tradition Rigorous calibration uncertainty analysis Test uncertainty ratio required (4:1) Calibration guard band applied (80%)

7. Rick Walker Evaluation of Out-of-Tolerance Risk 7 Problems TUR might be higher than is necessary (wasted cost) What guard band is best? What about additional error while in service, such as drift? What will be the pass yield? Still don’t know what the OTR will be

8. Rick Walker Evaluation of Out-of-Tolerance Risk 8 Balancing act

9. Rick Walker Evaluation of Out-of-Tolerance Risk 9 Better approach Evaluate uncertainties and variables Calculate out-of-tolerance risk Calculate pass yield Adjust guard band as necessary

10. Rick Walker Evaluation of Out-of-Tolerance Risk 10 Possible benefits Manufacturer –Avoid overdesign –Reduce cost –Improve specifications –Predict and control pass yield Calibration lab –Reduce required TUR –Find the optimal guard band –Fewer failures Customer –Better reliability

11. Rick Walker Evaluation of Out-of-Tolerance Risk 11 Test results MTE is calibrated and shown to be in tolerance Calibration has uncertainty May actually be out of tolerance Greater observed error means higher probability of being out of tolerance

12. Rick Walker Evaluation of Out-of-Tolerance Risk 12 OTR vs. observed error SN. A Error: 50% OTR: 0.1% SN. B Error: 90% OTR: 10% Is it acceptable that a few customers receive equipment like SN. B?

13. Rick Walker Evaluation of Out-of-Tolerance Risk 13 Individual risk Risk can be evaluated for individual MTE, depending on calibration results Do not integrate over the population! –Otherwise some high-risk equipment may be delivered

14. Rick Walker Evaluation of Out-of-Tolerance Risk 14 Guard band OTR can be controlled by limiting the acceptable calibration results Specification x guard band (< 1) OTR is calculated for the maximum acceptable error, at the guard band limit

15. Rick Walker Evaluation of Out-of-Tolerance Risk 15 What if... Observed error is at the specification limit Test has uncertainty DUT is either in tolerance or out You must guess which Which is more likely? What is the OTR? 50%?

16. Rick Walker Evaluation of Out-of-Tolerance Risk 16 The answer More likely to be in tolerance OTR is 31% –(4:1 TUR, 2  specification) Why?

17. Rick Walker Evaluation of Out-of-Tolerance Risk 17 Other relevant information MTE is designed, manufactured, and aligned to be well in tolerance Most devices pass first calibration Those that have large error usually do better when retested A priori distribution:

18. Rick Walker Evaluation of Out-of-Tolerance Risk 18 Conditional probability OTR

19. Rick Walker Evaluation of Out-of-Tolerance Risk 19 Electrical analogue

20. Rick Walker Evaluation of Out-of-Tolerance Risk 20 Mean and standard deviation Assumptions: normal distributions

21. Rick Walker Evaluation of Out-of-Tolerance Risk 21 OTR calculation  (x) is the cumulative normal distribution function No integrals! OTR is easily calculated in a spreadsheet

22. Rick Walker Evaluation of Out-of-Tolerance Risk 22 Immediate risk Maximum OTR immediately after calibration Depends on... –Calibration error, varying (noise) –Calibration error, constant –Calibration error, alignment –Specification limit –Guard band DUT is tested, aligned, then retested with the same system

23. Rick Walker Evaluation of Out-of-Tolerance Risk 23 Immediate risk equations

24. Rick Walker Evaluation of Out-of-Tolerance Risk 24 Field risk What really matters is that the MTE is in tolerance during use –It is less accurate in the field than during calibration –Drift or aging and other product characteristics must be taken into account –Environmental conditions vary

25. Rick Walker Evaluation of Out-of-Tolerance Risk 25 Field risk equations

26. Rick Walker Evaluation of Out-of-Tolerance Risk 26 Pass yield Probability MTE will pass calibration (error is within the guard band) Pass yield must be high Guard band must not be too low

27. Rick Walker Evaluation of Out-of-Tolerance Risk 27 Pass yield equations

28. Rick Walker Evaluation of Out-of-Tolerance Risk 28 Other risk measures Retest yield –Probability MTE appears in tolerance when retested at the end of the calibration interval Retest risk –Probability MTE is out of tolerance though it appears in tolerance during retest

29. Rick Walker Evaluation of Out-of-Tolerance Risk 29 Spreadsheet (inputs)

30. Rick Walker Evaluation of Out-of-Tolerance Risk 30 Spreadsheet (outputs)

31. Rick Walker Evaluation of Out-of-Tolerance Risk 31 Example 1 (parameters) Specification: 25 ppm Calibration, varying: 1.2 Calibration, constant: 2.8 Alignment: 6.0 Drift, mean: 1.6 Drift, std dev: 2.6 Environment: 1.4 Guard band: 75%

32. Rick Walker Evaluation of Out-of-Tolerance Risk 32 Example 1 (results) Immediate risk: 1.1% First-pass yield: 99.7% Field risk: 10.4% (too high) Retest pass yield: 73.5% (too low) 75% guard band is inadequate 55% guard band gives acceptable results (0.8% field risk, 97% FPY)

33. Rick Walker Evaluation of Out-of-Tolerance Risk 33 Example 2 (parameters) Specification: 0.4°C Calibration, varying: 0.028 (2:1 TUR) Calibration, constant: 0.094 (2:1 TUR) Alignment: 0.02 Drift, mean: 0.038 Drift, std dev: 0.052 Environment: 0.032 Guard band: 50%

34. Rick Walker Evaluation of Out-of-Tolerance Risk 34 Example 2 (results) Immediate risk: 0.19% First-pass yield: 99.999% Field risk: 1.7% Retest pass yield: 96.3% 2:1 TUR is sufficient, with 50% guard band

35. Rick Walker Evaluation of Out-of-Tolerance Risk 35 Check Monte Carlo computer simulation Generate random numbers with probability distributions Add to simulate total errors Compare with limits Repeat many times Count out-of-tolerance occurrences

36. Rick Walker Evaluation of Out-of-Tolerance Risk 36 Example 1 comparison MetricCalculationSimulation Immediate risk1.1% First-pass yield99.7% Field risk10.4% Retest pass yield73.5%73.3%

37. Rick Walker Evaluation of Out-of-Tolerance Risk 37 Conclusion OTR should be evaluated and controlled Product drift should be factored in Best way to decide guard bands TUR need not be absolute

38. Rick Walker Evaluation of Out-of-Tolerance Risk 38 Thank you Questions?