1. Brian L. Strom, M.D., M.P.H. Chair and Professor, Department of Biostatistics and Epidemiology Director, Center for Clinical Epidemiology and Biostatistics George S. Pepper Professor of Public Health and Preventive Medicine Professor of Biostatistics and Epidemiology, Medicine, and Pharmacology University of Pennsylvania School of Medicine CCEB

2. “Sampling” 1.What is an appropriate sample size of respondents to best determine risk of sound and look alike proprietary names in the prescription drug study group? In a focus group? In a survey document? 2.Should sample be randomly selected? Is it important to have a statistical significance for this type of evaluation?

4. CCEB “Sampling” Introduction Very general principles of sample size calculations and sampling Application of general principles to this situation Recommendations for research, to guide the future

5. CCEB Options in Research Design Analytic Studies –Experimental Study –Prospective Cohort Study –Retrospective Cohort Study –Case-Control Study Descriptive Studies –Analyses of Secular Trends –Case Series –Case Reports

6. Case-Control Studies Disease Cohort Studies Factor Present (cases) Absent (not exposed) Present (exposed) Absent(controls)A DC B

7. CCEB Determinants of Needed Sample Size Alpha Beta Variability (SD) Delta--how small a difference do you want to be able to detect?

8. CCEB Determinants of Needed Sample Size: Cohort Study Alpha Beta Incidence in the unexposed control group Delta--how small a RR do you want to be able to detect?

9. CCEB Determinants of Needed Sample Size: Case-Control Study Alpha Beta Prevalence in the undiseased control group Delta--how small a RR do you want to be able to detect?

10. CCEB Sampling Study Sample Conclusion About a Population ( Association ) Conclusion About Scientific Theory (Causation) Statistical Inference Biological Inference

11. CCEB Application of General Principles to This Situation THE Central Principle of Research Design: The question is, what is the question??

12. CCEB Application of General Principles to This Situation In this situation, there are no a priori hypotheses being tested to be able to consider sample size calculations or questions of sampling What is being performed is essentially qualitative research

13. CCEB Recommendations for Research, to Guide the Future **Evaluate the current process in a quantitative fashion**

14. CCEB General Strategy for Evaluation Standardize procedure Test for reliability/ reproducibility Test for validity Make changes in the procedure accordingly

15. CCEB Standardize Procedure Choose among the current possible approaches a “standard” to be evaluated more rigorously

16. CCEB Test for Reliability/Reproducibility Evaluate the same drug names in the same process with multiple different groups of survey prescribers and experts, to look for whether there is adequate agreement If no reliability, validity is impossible and procedure should be abandoned

17. CCEB Test for Validity Gold standard needed: –Drug names rejected in initial FDA review –Drug names withdrawn due to problems –Direct measurement of error rate

18. CCEB Retrospective: Drug Names Rejected in Initial FDA Review Problem: was the initial review decision correct?

19. CCEB Retrospective: Drug Names Withdrawn Due to Problems Knowledge of reviewers could be problematic –Other countries –Years ago, and using new pharmacists Was the withdrawal decision a correct one?

20. CCEB Prospective: Direct Measurement of Error Rate Simulate real life situation in a study setting Choose good and bad options for new names Enter possible new names into prescription entry computer program Ask large numbers of docs to write orders Ask large numbers of pharmacists to “fill” each rx, entering into prescription entry computer program Directly measure resulting error rate

21. CCEB Make Changes in the Procedure Accordingly Determine appropriate cutpoint for expert ratings, ROC curve vs. gold standard Determine appropriate sample sizes through simulation, ie how many are needed to achieve results consistent with the gold standard Modify processes accordingly

22. CCEB Potential Sources of Support FDA extramural funds AHRQ patient safety funds NIA pharmacology program NIGMS pharmacology program

23. CCEB Conclusions Applying a quantitative approach to evaluating what has so far been a qualitative one, could lead to major changes in the procedure, and major improvements in the net results

24. Statitsically incorrect