1 Audit of the Columbia Suicidality Classification Project September 13, 2004 Meeting of Psychopharmacological Drugs Advisory Committee and Pediatric Advisory Committee Solomon Iyasu, M.D., M.P.H. Medical Team Leader DPDD/OCTAP/CDER Food and Drug Administration

2 Outline Background Objectives of the FDA audit FDA audit method, process Results of the audit Limitations/strengths Conclusion

3 Objective of the FDA Audit Assess the reproducibility and reliability of the Columbia suicidality classification methodology and process

4 The FDA Audit Methods and Process: Event Narratives Columbia reviewed and classified ALL sponsor- submitted event narratives FDA audit team received a computerized line listing of these event narratives from the Div. Neuro- pharmacological Drug Products Drew a sample of events for review

5 The FDA Audit Methods and Process Grouped event narratives into 4 predefined strata Selected event narratives for review via a stratified simple random sampling strategy Over-sampled difficult to classify and reclassified events

6 The FDA Audit Methods and Process: Strata and Definitions Stratum 1: Events reclassified by Columbia to non-suicidal or other events (n=2) Stratum 2: Events newly identified and classified by Columbia as possibly suicide related or other categories (n=29) Stratum 3: Events that were difficult to classify defined as events with discordant initial independent ratings by Columbia reviewers (n=56) Stratum 4: Events that are straight forward cases defined as events with concordant initial ratings by Columbia reviewers (n=336)

7 The FDA Audit Methods and Process: Sample Size and Selection Sample size: 64 of the 423 event narratives selected for review (15%) –Stratum 1: All selected ( n=2) –Stratum 2: 1/3 rd selected (n=10)* –Stratum 3: 1/3 rd selected (n=19) –Stratum 4: 1/10 th selected (n=33) Events from stratum 2 – 4 were selected by a simple random technique using a random number table * Four of the sampled records were also ‘difficult to classify”

8 Composition of the FDA Audit Team Planning Group Div. of Pediatric Drug Development, OCTAP, CDER –Solomon Iyasu, MD, M.P.H., Audit Team Leader –Susan Cummins, M.D., M.P.H. –Rosemary Addy, Project Manager –Kristin Phucas, Project Manager Div. of Neuro-pharmacological Drug Products, CDER –Thomas Laughren, M.D. OND –Armando Oliva, M.D. Clinical Reviewers Div. of Pediatric Drug Development, OCTAP, CDER –Hari Sachs, M..D., (Pediatrician) –ShaAvhree Buckman, M..D., Ph.D. (Pediatrician) Div. of Neuro-pharmacological Drug Products, CDER –Cara Alfaro, PharmD. (Pharmacist) –Robert Levin, M..D. (Psychiatrist) Consensus meeting facilitator Div of Scientific Investigations –Robert Stasko, M.D. (psychiatrist), None of the clinical reviewers had previously reviewed the event narratives

9 FDA Audit Team Training FDA Audit Team received a 2-hour teleconference training by Kelly Posner about the suicidality classification scale, method and process including –A review of the classification scale, categories and construct –A review of examples of case narratives for each of the classification categories –Independent classification of test event narratives by participants to evaluate the training

10 The FDA Audit Methods and Process: Review Assignments Each sampled event randomly assigned to three of four reviewers for independent and blinded review Each reviewed and rated 48 events (total of 192 reviews) Reviewers blinded to treatment assignment, sponsor, diagnosis and to the Columbia ratings

11 The FDA Audit Methods and Process Review Procedures/Instructions A memo outlining the procedures of the audit was prepared and provided to the audit team Reviewers were not allowed to discuss the events among themselves or with colleagues during the independent review Allowed reviewers to call Kelly Posner, Columbia University to obtain clarification on the classification scale but no discussion of the specifics of any case was allowed Required reviewers to record on rating form if they consulted with Columbia during the review process Modified and pre-coded rating form used

12

13 The FDA Audit Methods and Process Reviewers returned completed rating forms in sealed envelopes Ratings double key entered into an Excel database Discordant ratings identified and discussed during a consensus meeting facilitated –by a board certified child and adolescent psychiatrist external to DNDP –previously not involved in the review of these records or had knowledge of the Columbia ratings score Final consensus ratings entered into an Excel database and compared to the final Columbia ratings. Discordant ratings between FDA and Columbia discussed with Columbia to better understand the reason for the differences

14 Criteria for Evaluation of Concordant Ratings Columbia Definition of Concordance –Categories 1, 2, 3, 6, and 10 required exact match –Categories 4, 5 or 11 all describe self-injurious behavior with no suicidal intent and considered equivalent rating –Categories 7, 8, 9 or 12 all describe “other” (non-suicidal and non-self-injurious behavior) and are considered equivalent rating

15 Results: Concordance among FDA reviewers 47 of 64 initial ratings were concordant 17 of 64 initial ratings were discordant –Event narratives discussed during the FDA reviewer’s consensus meeting –Final consensus ratings reached for all 17 events

16 Results: Comparison of Final FDA and Columbia Ratings Concordance on 57 of 64 final event ratings –Percent agreement of 89% ( k=0.84) Discordance analysis (n=7) –Severity Hierarchy (1 or 2) > 6 > 3 > (4 or 5 or 11) > 10 –Compared to Columbia, the FDA audit team classified six events with higher severity (3 of these events were classified as ‘not enough information’ vs. ‘other’) one event with lower severity

17 Limitations Neither the quality of the narratives nor the clinical source material for the narratives were evaluated Validity of the Columbia classification method not assessed –there is no gold standard

18 Strengths Achieved a high level of concordance between the two independent review teams despite –Differences in expertise and experience –Short FDA timeline for training and review –Intentional over-sampling of difficult to classify events for review

19 Conclusion The Columbia Suicidality classification methodology is robust and reproducible.

20 Acknowledgments OCTAP/CDER: Hari Sachs, ShaAvhree Buckman, Rosemary Addy, Kristin Phucas, Susan Cummins DNDP/CDER: Robert Levin, Cara Alfaro, Thomas Laughren, Alice Hughes, Tarek Hammad DSI: Robert Stasko OND: Armando Oliva Columbia University: Kelly Posner

21 Thank you!