Audit of the Columbia Suicidality Classification Project September 13, 2004 Meeting of Psychopharmacological Drugs Advisory Committee and Pediatric Advisory Committee

1
Audit of the Columbia Suicidality Classification
Project September 13, 2004Meeting of
Psychopharmacological Drugs Advisory Committee
andPediatric Advisory Committee

Solomon Iyasu, M.D., M.P.H.Medical Team
Leader DPDD/OCTAP/CDERFood and Drug
Administration
2
Outline

• Background
• Objectives of the FDA audit
• FDA audit method, process
• Results of the audit
• Limitations/strengths
• Conclusion

3
Objective of the FDA Audit

• Assess the reproducibility and reliability of the
  Columbia suicidality classification methodology
  and process

4
The FDA Audit Methods and ProcessEvent
Narratives

• Columbia reviewed and classified ALL
  sponsor-submitted event narratives
• FDA audit team received a computerized line
  listing of these event narratives from the Div.
  Neuro-pharmacological Drug Products
• Drew a sample of events for review

5
The FDA Audit Methods and Process

• Grouped event narratives into 4 predefined strata
• Selected event narratives for review via a
  stratified simple random sampling strategy
• Over-sampled difficult to classify and
  reclassified events

6
The FDA Audit Methods and ProcessStrata and
Definitions

• Stratum 1 Events reclassified by Columbia to
  non-suicidal or other events (n2)
• Stratum 2 Events newly identified and classified
  by Columbia as possibly suicide related or other
  categories (n29)
• Stratum 3 Events that were difficult to classify
  defined as events with discordant initial
  independent ratings by Columbia reviewers (n56)
• Stratum 4 Events that are straight forward cases
  defined as events with concordant initial ratings
  by Columbia reviewers (n336)

7
The FDA Audit Methods and ProcessSample Size
and Selection

• Sample size 64 of the 423 event narratives
  selected for review (15)
• Stratum 1 All selected ( n2)
• Stratum 2 1/3rd selected (n10)
• Stratum 3 1/3rd selected (n19)
• Stratum 4 1/10th selected (n33)
• Events from stratum 2 4 were selected by a
  simple random technique using a random number
  table

Four of the sampled records were also
difficult to classify
8
Composition of the FDA Audit Team

• Planning Group
• Div. of Pediatric Drug Development, OCTAP, CDER
• Solomon Iyasu, MD, M.P.H., Audit Team Leader
• Susan Cummins, M.D., M.P.H.
• Rosemary Addy, Project Manager
• Kristin Phucas, Project Manager
• Div. of Neuro-pharmacological Drug Products, CDER
• Thomas Laughren, M.D.
• OND
• Armando Oliva, M.D.

• Clinical Reviewers
• Div. of Pediatric Drug Development, OCTAP, CDER
• Hari Sachs, M..D., (Pediatrician)
• ShaAvhree Buckman, M..D., Ph.D. (Pediatrician)
• Div. of Neuro-pharmacological Drug Products, CDER
• Cara Alfaro, PharmD. (Pharmacist)
• Robert Levin, M..D. (Psychiatrist)
• Consensus meeting facilitator
• Div of Scientific Investigations
• Robert Stasko, M.D. (psychiatrist),

None of the clinical reviewers had previously
reviewed the event narratives
9
FDA Audit Team Training

• FDA Audit Team received a 2-hour teleconference
  training by Kelly Posner about the suicidality
  classification scale, method and process
  including
• A review of the classification scale, categories
  and construct
• A review of examples of case narratives for each
  of the classification categories
• Independent classification of test event
  narratives by participants to evaluate the
  training

10
The FDA Audit Methods and ProcessReview
Assignments

• Each sampled event randomly assigned to three of
  four reviewers for independent and blinded review
• Each reviewed and rated 48 events (total of 192
  reviews)
• Reviewers blinded to treatment assignment,
  sponsor, diagnosis and to the Columbia ratings

11
The FDA Audit Methods and ProcessReview
Procedures/Instructions

• A memo outlining the procedures of the audit was
  prepared and provided to the audit team
• Reviewers were not allowed to discuss the events
  among themselves or with colleagues during the
  independent review
• Allowed reviewers to call Kelly Posner, Columbia
  University to obtain clarification on the
  classification scale but no discussion of the
  specifics of any case was allowed
• Required reviewers to record on rating form if
  they consulted with Columbia during the review
  process
• Modified and pre-coded rating form used

12
(No Transcript)
13
The FDA Audit Methods and Process

• Reviewers returned completed rating forms in
  sealed envelopes
• Ratings double key entered into an Excel database
  
• Discordant ratings identified and discussed
  during a consensus meeting facilitated
• by a board certified child and adolescent
  psychiatrist external to DNDP
• previously not involved in the review of these
  records or had knowledge of the Columbia ratings
  score
• Final consensus ratings entered into an Excel
  database and compared to the final Columbia
  ratings.
• Discordant ratings between FDA and Columbia
  discussed with Columbia to better understand the
  reason for the differences

14
Criteria for Evaluation of Concordant Ratings

• Columbia Definition of Concordance
• Categories 1, 2, 3, 6, and 10 required exact
  match
• Categories 4, 5 or 11 all describe self-injurious
  behavior with no suicidal intent and considered
  equivalent rating
• Categories 7, 8, 9 or 12 all describe other
  (non-suicidal and non-self-injurious behavior)
  and are considered equivalent rating

15
ResultsConcordance among FDA reviewers

• 47 of 64 initial ratings were concordant
• 17 of 64 initial ratings were discordant
• Event narratives discussed during the FDA
  reviewers consensus meeting
• Final consensus ratings reached for all 17 events

16
ResultsComparison of Final FDA and Columbia
Ratings

• Concordance on 57 of 64 final event ratings
• Percent agreement of 89 ( k0.84)
• Discordance analysis (n7)
• Severity Hierarchy
• (1 or 2) gt 6 gt 3 gt (4 or 5 or 11) gt 10
• Compared to Columbia, the FDA audit team
  classified
• six events with higher severity (3 of these
  events were classified as not enough
  information vs. other)
• one event with lower severity

17
Limitations

• Neither the quality of the narratives nor the
  clinical source material for the narratives were
  evaluated
• Validity of the Columbia classification method
  not assessed
• there is no gold standard

18
Strengths

• Achieved a high level of concordance between the
  two independent review teams despite
• Differences in expertise and experience
• Short FDA timeline for training and review
• Intentional over-sampling of difficult to
  classify events for review

19
Conclusion

• The Columbia Suicidality classification
  methodology is robust and reproducible.

20
Acknowledgments

• OCTAP/CDER Hari Sachs, ShaAvhree Buckman,
  Rosemary Addy, Kristin Phucas, Susan Cummins
• DNDP/CDER Robert Levin, Cara Alfaro, Thomas
  Laughren, Alice Hughes, Tarek Hammad
• DSI Robert Stasko
• OND Armando Oliva
• Columbia University Kelly Posner

21
Thank you!