1. Evaluating Drug Name Confusion Using Expert Panels Drug Safety and Risk Management Advisory Committee Meeting Silver Spring, Maryland September 19, 2003 R. F. Shangraw, Jr., Ph.D. Project Performance Corporation

2. Topics Proposed Design Validation Approaches Strengths and Weaknesses

3. Proposed Design Establish Credentials for Experts Establish Voting/Ranking Guidelines Select Panel & Moderator Conduct Round 1 Conduct Round 2 Collate & Consolidate Results Report Confusing Names

4. Establish Credentials for Experts Establish baseline qualifications Review potential conflicts Evaluate personalities (protect against domineering personalities) Seek diversity Establish Credentials for Experts Establish Credentials for Experts Establish Voting/Ranking Guidelines Establish Voting/Ranking Guidelines Select Panel & Moderator Select Panel & Moderator Conduct Round 1 Conduct Round 1 Conduct Round 2 Conduct Round 2 Collate & Consolidate Results Collate & Consolidate Results Report Confusing Names Report Confusing Names

5. Establish Voting & Ranking Guidelines Set clear guidelines for ranking choices Set clear guidelines for collating results Determine voting thresholds Establish Credentials for Experts Establish Credentials for Experts Establish Voting/Ranking Guidelines Establish Voting/Ranking Guidelines Select Panel & Moderator Select Panel & Moderator Conduct Round 1 Conduct Round 1 Conduct Round 2 Conduct Round 2 Collate & Consolidate Results Collate & Consolidate Results Report Confusing Names Report Confusing Names

6. Select Panel & Moderator Select from 8 – 12 participants Identify a moderator with experience in facilitating small-group discussions KEY ISSUE The experience of the moderator can influence the outcome of the process Establish Credentials for Experts Establish Credentials for Experts Establish Voting/Ranking Guidelines Establish Voting/Ranking Guidelines Select Panel & Moderator Select Panel & Moderator Conduct Round 1 Conduct Round 1 Conduct Round 2 Conduct Round 2 Collate & Consolidate Results Collate & Consolidate Results Report Confusing Names Report Confusing Names

7. Conduct Round 1 Provide proposed name (and other factors) with instructions via email Instructions should outline a process for reviewing the proposed name Ask panelists for a ranked list of 5 potential look-alike and 5 potential sound-alike names Establish Credentials for Experts Establish Credentials for Experts Establish Voting/Ranking Guidelines Establish Voting/Ranking Guidelines Select Panel & Moderator Select Panel & Moderator Conduct Round 1 Conduct Round 1 Conduct Round 2 Conduct Round 2 Collate & Consolidate Results Collate & Consolidate Results Report Confusing Names Report Confusing Names KEY ISSUE The number of potential names and the ranking process can influence the outcome of the process

8. Consolidate Results from Round 1 Consolidate ranked lists from panelists Develop an aggregated ranked list for use in Round 2 Establish Credentials for Experts Establish Credentials for Experts Establish Voting/Ranking Guidelines Establish Voting/Ranking Guidelines Select Panel & Moderator Select Panel & Moderator Conduct Round 1 Conduct Round 1 Conduct Round 2 Conduct Round 2 Collate & Consolidate Results Collate & Consolidate Results Report Confusing Names Report Confusing Names KEY ISSUE The method for aggregating the ranked individual results can influence the outcome of the process

9. Conduct Round 2 Discuss and vote on the list of ranked names from Round 1 Discussion facilitated by moderator Round 2 can be conducted in-person (more expensive) Can be conducted via teleconference with the assistance of a web-based application for real-time voting Use predetermined voting rules to report potentially confusing names Establish Credentials for Experts Establish Credentials for Experts Establish Voting/Ranking Guidelines Establish Voting/Ranking Guidelines Select Panel & Moderator Select Panel & Moderator Conduct Round 1 Conduct Round 1 Conduct Round 2 Conduct Round 2 Collate & Consolidate Results Collate & Consolidate Results Report Confusing Names Report Confusing Names

10. Possible Validation Methods Reliability Convene 3-5 different panels Provide each panel with the same proposed name and use identical instructions Evaluate the correlation in outcomes Predictive Validity Provide the same panel with 5-10 names with known confusions Evaluate the correlation between the panel findings and the known confusions Control for history, if possible

11. Strengths of Design Human opinion on potential confusions may yield more “real life” examples Facilitated expert discussion may yield more insight into the reasons for the confusion Design is easy to understand and implement

12. Weaknesses of Design Panels are susceptible to domineering participants or partial moderators Difficult to validate design and outcomes Consensus may not be achievable with certain names or the voting threshold may be difficult to determine Design does not adequately address confusion due to dialect Potential for wide variability in results due to the expertise of the individual panel members Concern about confidentiality using electronic methods