1. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org The International Seed Testing Association GMO PT Program Design and Analysis AIEC Fall Meeting October 15-16 2014 New Orleans, Louisana

2. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Outline Reference material purity requirements Rating labs for ability to detect presence Rating labs for ability to quantify

3. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Reference material purity requirements Seed Lot Purity = 99% + - + -

4. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Reference material purity requirements To make “sure” that the distributed samples contain the targeted GMO levels – GM seed is required to have purity greater than 99.25% with 95% confidence – Conventional seed is required to have impurity less than 0.01% with 95% confidence

5. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Binomial Distribution: 1.Success – With probability p 2.Failure – With probability q = 1-p How Do I Do That Tim?

6. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Example 1: Probability of X heads out of 10 coin flips How many head would you expect to see? Would you be shocked if all 10 coins came up heads?

7. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Coin Tossing: binomial probability of X heads in 10 coin tosses

8. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org If I tell you 1% of the beads are red how many would you red beads would you expect to see? Would you be surprised if you took a scoop and no red beads were observed? Example 2: Probability of X red beads out of 300

9. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org

10. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org I have another bag of beads This bag could have any number of red beads I let you take 1 and only 1 sample In the sample you observe no red beads If you had to say something about the purity of the new bag what would you say?

11. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Confidence Claim If the bag had 1% red beads, a scoop of 300 would contain at least 1 red bead 95% of the time I took a scoop of 300 beads and observed 0 red Thus, I am 95% confident the percent red beads is less than 1%.

12. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Using the same logic in our case To demonstrate GM seed purity greater than 99.25% with 95% confidence requires 400 seeds tested with 0 negatives To demonstrate conventional seed impurity less than 0.01% with 95% confidence requires 30,000 seeds tested with 0 positives

13. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Can we do this cheaper The cost of testing reference material for PT program was becoming too large Needed to find a way to reduce cost and still make “sure” that the distributed samples contain the targeted GMO levels

14. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Father of Bayesian Methods Reverend Thomas Bayes 1702 –1761, London England Nonconformist minister in Tunbridge Kent, England and a mathematician on the side

15. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org The seed provider has already performed purity tests By taking into account this information using Bayesian statistics, the final assessment of seed purity performed by ISTA requires less testing efforts  $$$$ savings Sequential updates: Prior updated using seed provider’s data Then update the posterior Prior (given y 1 )Likelihood (given y 1 ) : proportionality sign; replaces a constant normalizer

16. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Results : Updated prior : Formulas for individual testing (GM seeds) : Formulas for group testing (conventional seeds) : Likelihood : 1, 2 : false negative rates;  1,  2 : false positive rates; m 1, m 2 : # of seeds; y 1, y 2 : # of negative seeds observed 1, 2 : false neg. rates;  1,  2 : false pos. rates; n 1, n 2 : # of pools ; m 1, m 2 : # of seeds/pool; y 1, y 2 : # of neg. pools observed. Purity requirements fulfilled. $$$$ savings

17. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Rating a laboratory in a given proficiency test We need to compute objectively, and transparently to the participants, criteria to rate laboratories for each proficiency test on: –Presence/absence –Quantification

18. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Proficiency tests rating system ANo problem has been detected in this test BThere are small problems, but no specific look or action is suggested to the participant CProblems, ISTA indicates there might be things to consider by the laboratory to explain or correct things BMPBelow Minimum of Performance, ISTA indicates that the results were poor and the laboratory need to find explanations and to improve/correct

19. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org One test rating One test Score Value Overall rating on 6 tests Range on 6 tests A5 pointsA28 – 30 points B4 pointsB21 – 27 points C3 pointsC16 – 20 points BMP0 points BMPbelow 16 points each proficiency testand a run of 6 tests Example 1: 5A rating and 1 BMP 5*5points + 1*0point =25 points Overall rating is B Example 2: 4 B =16 points, no overall BMP

20. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Rating ability to detect presence True value is known 2 types of error can occur: –Laboratory report as negative a positive sample –Laboratory report as positive a negative sample The number of mistakes is used as a basis for the rating computations

21. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Presence/absence Rating system RatePercentage of misclassified samples A0% - 5% B>5% - 10% C>10% - 20% BMP>20% Example with 12 samples in a Test 3 mistakes 3/12=25%

22. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Presence/absence Rating systems RatePercentage of misclassified samples A0 errors B1 error C2 errors BMPMore than 2 errors Example with 12 samples in a Test

23. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Confidence intervals are rather wide: If 20 test and all correct 95% confident error rate less than 17% Many more samples (over 300) would be necessary to estimate error rates with 1% precision => PT does not check error rates

24. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Rating quantification True value is known, and/or can be estimated from all results received Too many sample results are too far from truth ->BMP Average results by spiking level are not accurate ->C Too many inaccurate sample results ->B Otherwise rate A

25. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Z-Scores The rating system for quantification is based on Z-scores Before we get to far into the rating system lets talk really quick about Z-scores

26. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Example: Birth Weights are Normally Distributed

27. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org z-scores Consider a value x from a distribution with mean  and standard-deviation . The formula for converting x into its corresponding z-score is:  Indicates how far and in what direction x deviates from , in units of   z-scores distribution’s mean = 0 z-scores distribution’s std-deviation = 1  When the distribution of reference is normal, the z- scores distribution is also normal and the probability to have a z-score in the interval [-2 ; +2] is ~ 0.95

28. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org

29. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org

30. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Consider the following z-scores computed for each Lab and each spiking-level: We can then define the rejection region at the 0.01 level: 3 spiking levels: if sum of absolute z-scores > 5.25  C rating 1%

31. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Consider the following z-scores computed for each (Lab x spiking-level x sample) combination: A z-score outside [-2 ; +2] has low probability (~ 0.05) to occur for a laboratory providing accurate sample results.

32. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org. For each Lab k and each spiking level i, compute the variance of the sample results:. Compute the mean of the variances for a given spiking level i (except the one identified with the Cochran’s test): Estimate of the intra-Lab standard deviation. Carry out the Cochran’s test at the 95% level to see if the Lab with the highest variance has an outlying spread of replicates.. The intra-Lab standard-deviation estimate is then:

33. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Statistical evaluation of the accuracy of the ISTA GMO PT quantitative rating system Recent PTs have smaller intra-laboratory standard-deviations

34. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Statistical evaluation of the accuracy of the ISTA GMO PT quantitative rating system Leading to more marginal results for the PTs having smaller intra-laboratory standard-deviations

35. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Estimated from the data Characterize the precision of the results The better the precision, the higher chance to get “bad” ratings! Particularly true for late PTs: Change implemented (starting PT18) : If the relative reference intra-laboratory standard-deviation is lower than 25%, use 0.25 x true GM content as the reference intra- laboratory standard-deviation.

36. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Conclusion Reference purity requirements are checked using Bayesian methods which utilized provider test results Labs ability to detect presence/absence rated based on the percent misclassified samples, not intended to estimate error rates Labs ability to quantify rated based on z- scores, as labs have gotten more precise some adjustments have needed to be made

37. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Thanks to: Jean-Louis Laffont Kirk Remund

38. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Thank you for your attention www.seedtest.org

39. INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org Questions ?