COP pass/fail decisions Norbert E. Ligterink

pass-fail decision schemes number of tests failures in PASS failures in FAIL 3 - 4 1 … COP pass/fail decisions pass-fail decision schemes sequential scheme based on: producer and consumer risks level of confidence (representativeness versus test burden) Schemes can be adapted to match the general criteria General criteria: minimal number of tests: all-pass with 3 or 5 tests maximal number of tests: for example: 11 or 17 tests symmetry between producer and consumer risks: 11 tests: 5 or less faulty  pass, 6 or more faulty  fail asymmetry: higher producer risk: e.g. 3 or less faulty  pass  scheme can be adapted to meet criteria example “all-pass” with 3 tests means, e.g.: less than 38% faulty with 95% confidence less than 47% faulty with 90% confidence

COP pass/fail decisions effect of the criteria minimum number of tests leads to high confidence (easy to fulfil) maximal number of tests is critical: (difficult to fulfil) confidence increases slowly with the number of tests: pass-with-40%-faulty with symmetric risk: (thus corresponding to fail-with-60% faulty) pass at 11 tests with less than 6 faulty  75% confidence pass at 17 tests with less than 9 faulty  80% confidence asymmetry might be appropriate: (e.g. EPA) e.g. pass 17 tests less than 4 faulty (higher producer risk) Decision on min-max-symmetry determines scheme. recommendation: minimal 3 tests, maximal 11 tests, pass with “maximally-4-fails-in-11” asymmetry

background on pass-fail scheme statistics COP pass/fail decisions background on pass-fail scheme statistics calculate the probability that a test is passed contrary to expectation based on the fraction of faulty (pfaulty)  P(k failures out of n tests) = (n!/(k!(n-k!)) * pfaultyk *(1-pfaulty)n-k (similar for fail decision with a fraction of non-faulty) pass decision on at most K failures: P(k=0) + P(k=1) + … + P(k=K) < 1 – confidence level confidence level pfaulty higher confidence levels for lower faulty fractions