1. Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics College of Engineering Rochester Institute of Technology

2. CQAS 2 Sensitivity Experiments u ASTM method D 1709–91 u Impact resistance of plastic film by free-falling dart method

3. CQAS 3 Objectives Engineer  Specify a probability of failure  – 0.50, 0.10, …  Find dart weight x =  such that Prob(F;  )=  Statistician  Find a strategy for selecting weights { x i } so that  is estimated as precisely as possible Darts are dropped one at a time. Weight of i th dart may depend on results obtained up to date

4. CQAS 4 Data Collection Possibilities Non-sequential  Specify n and all the { x i } before any Pass-Fail data { Y i } are obtained. Find dose  of drug at which 5% of mice develop tumors Group-sequential u Example: two-stage.  Specify n 1 and the { x 1i }. Obtain data { Y 1i }  Use this info to specify n 2 and the { x 2i }. Obtain data { Y 2i } l Same mice example, but with more time. (Fully) Sequential  Use all prior knowledge: x 1  Y 1  x 2  Y 2  x 3  Y 3  x 4  Y 4 l Dart-weight example. One machine, one run at a time.

5. CQAS 5 Model and Objectives Objective: Example  Estimate weight  at which 10% of the samples fail  So, try to set the { x i } to minimize

6. CQAS 6 Our Interest u (Fully) Sequential experiments  Estimating a  corresponding to a given , e.g. 0.10. u The real problem.   = 0.50?   = 0.001?

7. CQAS 7 A Quick Tour of Some Past Work u Up-Down Method. Dixon and Mood (1948) Only for  =0.50 u Robbins-Munro (1951) wanted { x i } to converge to . l Like Up-Down, but with decreasing increments  far from 0.50  convergence is too slow

8. CQAS 8 A Quick Tour of Some Past Work u Wu’s (1985) Sequential-Solving Method l Similar in spirit to the R-M procedure Collect some initial data to get estimates of  and  l l Better than R-M, much better than Up-and-Down l Performance depends somewhat heavily on initial runs l Asymptotically optimal, in a certain sense

9. CQAS 9 Some Non-Sequential Bayesian Results u Tsutakawa (1980) How to create design for estimation of  for a given . Certain priors on  and  l Some approximations l Assumed constant number of runs made at equally spaced settings. u Chaloner and Larntz (1989) Includes how to create design for estimation of  for a given  l Some reasonable approximations used l Not restricted to constant number of runs or equally spaced settings.

10. CQAS 10

11. CQAS 11

12. CQAS 12 This Talk. Bayesian Sequential Design u A way to specify priors  Measures of what we are learning about , , and  —A II and Information u Specifying the next setting, with some insights u Some examples and comparisons u Rethinking the priors

13. CQAS 13 Specifying Priors u Consider the related tolerance-distribution problem  The r.v. X i represents the (unobservable) speed at which the i th sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)

14. CQAS 14 Specifying Priors u Two-parameter distribution u Could specify priors on ( ,  ) ( ,  ) ( ,  ) u For simplicity, want to assume independence so only need to specify marginals of each parameter ( ,  )

15. CQAS 15 Specifying Priors  Instead of ( ,  ) …  Consider  =0.10 example u Consider  distance from  to  = 2.2  u Easier for engineer to understand ( ,  )

16. CQAS 16 Specifying Priors u Ask engineer for Best guess and 95% range for  v 5.0 ± 3.0 Best guess and 95% range for  –  distance  6.6  /  2.0  Translate  –  =2.2  into  terms: 3.0  /  2.0  Translate into normal, independent, priors on  and ln(  )  We used a discrete set of 15  15=225 values as prior distribution of ( ,  ) ( ,  )

17. CQAS 17 Specifying Priors  More natural for engineer to think about priors on  and . We let engineer do this as follows. u We created 27 combinations of prior distributions:  best guess—10  uncertainty (95% limits)— ± 2, 4, 6.  best guess— 1, 3, 5  uncertainty (95% limits)—  /  2, 4, 6.  We graphed these in terms of ( ,  ) ( ,  )

18. CQAS 18 Example of Prior Distributions of  =10± 4

19. CQAS 19 Finding the next setting x n+1 to run

20. CQAS 20 AII Measure

21. CQAS 21 Simple Example  Objective: find the  corresponding to  =0.10  Prob 80.25 90.50 100.25  Prob 10.25 20.50 30.25  Pr 815.8.0625 823.6.1250 831.4.0625 916.8.1250 924.6.2500 932.4.1250 1017.8.0625 1025.6.1250 1033.4.0625

22. CQAS 22 Simple Example

23. CQAS 23 Simple Example  Finding the AII for various x settings

24. CQAS 24 How AII “Thinks”

25. CQAS 25 First Simulation  ( ,  )=(8,1.82). Makes  =4.0 Setting increment = 1

26. CQAS 26 Example with a More Diffuse Prior   =10 ± 4,  =5  /  6  Simulation again done with  =8,  =4 

27. CQAS 27 Example with a More Diffuse Prior

28. CQAS 28 Behavior of AII after 0, 2, 10, 20, 60 runs

29. CQAS 29 Information on , ,  =  2.2   A serious problem—all the information on  was obtained through   The simulation trusted the relative tight prior on  … u Another problem: more objective methods of estimation, such as MLE, will likely not work well u Are there other ways to specify priors that might be better? Two methods…

30. CQAS 30 Equal-Contribution Priors  For  =  2.2 , restrict original prior so that Var 0 (  )=Var o (2.2  ) u Results of another simulation  Problem: fails for case  =  : Var 0 (  )=Var o (0  )?

31. CQAS 31 Relative Priors u Consider the tolerance-distribution problem  The r.v. X i represents the (unobservable) speed at which the i th sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)

32. CQAS 32 Relative Priors  We observe only the ( x, Y x ) ’s  If we could observe the X ’s, the problem would be a simple one-sample problem of finding the 100  percentile of a distribution.  Assume the distribution of the X ’s has a finite fourth moment.

33. CQAS 33 Relative Priors  Using delta-method to find Var( s ) and m-1  m u So, to a good approximation  After m runs, observing X 1, X 2, …, X m, we have

34. CQAS 34 Relative Priors  So, with k 1 and k 2 known,  So, in this sense it is defensible to specify only the prior precision with which  is know, and base the prior precision of  upon it. u Now assume tolerance distribution is symmetric and its shape is know, e.g. logistic. Then

35. CQAS 35 Logistic Example

36. CQAS 36 Summary  AII as a useful measure of value of making next run at x. Combination of shift in posterior mean & probability that a failure will occur at x  Informal comparison to non-Bayesian methods  Bayesian x -strategy is more subtle u Danger of simply using any prior, and recommended way to set priors