1. The Lung Image Database Consortium (LIDC) Data Collection Process This presentation based on the RSNA 2004 InfoRAD theater presentation titled “The Lung Imaging Database Consortium (LIDC) : Creating a Publicly Available Database to Stimulate Research in CAD Methods for Lung Cancer” (9110 DS-i) November 29, 2004 Michael McNitt-Gray (UCLA), Anthony P. Reeves (Cornell), Roger Engelmann (U. Chicago), Peyton Bland (U. Michigan), Chris Piker (U. Iowa), John Freymann (NCI) and The Lung Image Database Consortium (LIDC)

2. Principal Goals To establish standard formats and processes for managing thoracic CT scans and related technical and clinical data for use in the development and testing of computer-aided diagnostic algorithms.

3. Principal Goals To establish standard formats and processes for managing thoracic CT scans and related technical and clinical data for use in the development and testing of computer-aided diagnostic algorithms. To develop an image database as a web- accessible international research resource for the development, training, and evaluation of computer- aided diagnostic (CAD) methods for lung cancer detection and diagnosis using helical CT.

4. The Database The database will contain:The database will contain: 1) A collection of CT scan images 2) Technical factors about the CT scan Non-patient information in DICOM headerNon-patient information in DICOM header 3) For Nodules > 3 mm diameter Radiologist drawn boundariesRadiologist drawn boundaries Description of characteristicsDescription of characteristics 4) For Nodules < 3 mm Radiologist marks centroid, no characteristics Radiologist marks centroid, no characteristics 5) Pathology results or diagnosis information whenever available 6) All in a searchable relational database

5. How to do this? The LIDC Data Collection Process For nodule detection, recent research has demonstrated that the results from a single reader are not sufficient For nodule detection, recent research has demonstrated that the results from a single reader are not sufficient

6. How to do this? The LIDC Data Collection Process At least two and perhaps four readers may be required. At least two and perhaps four readers may be required. Not practical to do joint reading sessions across five institutions Not practical to do joint reading sessions across five institutions LIDC Will NOT do a forced consensus read. We won’t force agreement on location of a nodule nor its boundary. LIDC Will NOT do a forced consensus read. We won’t force agreement on location of a nodule nor its boundary.

7. Truth – Detection LIDC – Initial Approach Multiple Reads with Multiple Readers Multiple Reads with Multiple Readers First Read – 4 readers, each reads independently (Blinded) First Read – 4 readers, each reads independently (Blinded) Compile 4 blinded reads and distribute to readers Compile 4 blinded reads and distribute to readers Second Read – Same 4 readers, this time unblinded to the results of the other readers from the first reading. Second Read – Same 4 readers, this time unblinded to the results of the other readers from the first reading. Still, no forced consensus on either location of nodules nor on their boundaries. Still, no forced consensus on either location of nodules nor on their boundaries.

8. Blinded Reads – Each Reader Reads Independently (Blinded to Results of Other Readers)

9. Reader 1 Blinded Read for Reader 1 – Marks Only One Nodule

10. Reader 2 Blinded Read for Reader 2 – Marks Two Nodules (Note: One nodule is same as Reader 1)

11. Reader 3 Blinded Read for Reader 3 – Marks Two Nodules (Note: Again, One nodule is same as for Reader 1)

12. Reader 4 Blinded Read for Reader 4 – Did Not Mark Any Nodules

13. 2 nd Round - UnBlinded Reads Readings in Which Readers Are Shown Results of Other Readers Each Reader Marks Nodules After Being Shown Results From Their Own and Other Readers’ Blinded Reads (Each Reader Decides to Include or Ignore).

14. Reader 1 Unblinded Read for Reader 1 – Now Marks Two Nodules (Originally only marked one)

15. Reader 2 Unblinded Read for Reader 2 – Still Marks Two Nodules (No Change)

16. Reader 3 Unblinded Read for Reader 3 – Now Marks Three Nodules (Originally only marked two)

17. Reader 4 Unblinded Read for Reader 4 – Now Marks Three Nodules (Originally did not mark any)

18. 4/4 Markings 2/4 Markings Results of Unblinded Reads from All Four Readers We will capture one aspect of reader variability in this way

19. Radiologist Review & Reconcile- V2 4 radiologist (blinded) – R1B, R2B, R3B, R4B Radiologist Review & Reconcile- V2 4 radiologist (blinded) – R1B, R2B, R3B, R4B Submit to Requesting Site; This site compiles markings and re-sends case Radiologist Review & Reconcile- V2 4 radiologist (blinded) – R1B, R2B, R3B, R4B Submit to Requesting Site; This site compiles markings and re-sends case 4 Radiologists see all (anonymized) markings Radiologist Review & Reconcile 4 Radiologists Perform Blinded Read – R1B, R2B, R3B, R4B Submit to Requesting Site; This site compiles markings and re-sends case 4 Radiologists see all (anonymized) markings 4 Radiologists Perform Unblinded Read (R1U, R2U, R3U, R4U) Database (will contain Blinded AND Unblinded reads) R1UR2UR3UR4U Nodules for each condition: (R1B, R2B, R3B, R4B, R1U, R2U, R3U, R4U) Location Outline (where appropriate) Label (where appropriate)

20. Case 5, Slice 19

21. Radiologist 1 - Method 1

22. Radiologist 1 - Method 2

23. Radiologist 1 - Method 3

24. Radiologist 2 - Method 1

25. Radiologist 2 - Method 3

26. Radiologist 3 - Method 1

27. Radiologist 3 - Method 2

28. Radiologist 3 - Method 3

29. Radiologist 4 - Method 1

30. Radiologist 4 - Method 2

31. Radiologist 4 - Method 3

32. Radiologist 5 - Method 1

33. Radiologist 5 - Method 3

34. For each voxel, sum the number of occurrences (across reader markings) that it was included as part of the nodule For each voxel, sum the number of occurrences (across reader markings) that it was included as part of the nodule Create a probabilistic map of nodule voxels Create a probabilistic map of nodule voxels Higher probability voxels are shown as brighter; lower probability are darker Higher probability voxels are shown as brighter; lower probability are darker Can use apply a threshold and show only voxels > some prob. Value if desired. Can use apply a threshold and show only voxels > some prob. Value if desired. How to Represent This Variability? Create a Probabilistic Description of Nodule Boundary

35. Probabilistic Description of Boundary

36. Apply Threshold if Desired

37. Challenge: Define the Boundary of a Nodule Do we need to have agreement between radiologists on boundaries? Do we need to have agreement between radiologists on boundaries? LIDC’s answer is no. LIDC’s answer is no. LIDC Approach will be to: LIDC Approach will be to: Construct a probabilistic description of boundaries to capture reader variability Construct a probabilistic description of boundaries to capture reader variability Use a threshold value (50% centile or 1% centile) to give fixed contours. Use a threshold value (50% centile or 1% centile) to give fixed contours.

38. Pathology Information In those cases in which pathology is available, we will extract from reports: In those cases in which pathology is available, we will extract from reports: Whether histology or cytology was performed Whether histology or cytology was performed If histology, try to establish the cell type according to WHO classifications If histology, try to establish the cell type according to WHO classifications If cytology, establish whether it was benign or malignant If cytology, establish whether it was benign or malignant

39. Pathology Information If no pathology, other diagnostic information may be substituted when available (such as 2 years Dx F/U with no change in radiographic appearance). If no pathology, other diagnostic information may be substituted when available (such as 2 years Dx F/U with no change in radiographic appearance). If neither is available, then case will be used for detection purposes only. If neither is available, then case will be used for detection purposes only.

40. Database Implementation How to capture and collect all of this data? 5 Phases of Data Collection 1. Initial Review review case for inclusion in database; review case for inclusion in database; anonymize case; anonymize case; Index case, e.g. Full Chest/Limited Chest, Image Quality. Index case, e.g. Full Chest/Limited Chest, Image Quality. 2. Blinded Read identifying and drawing nodules independently identifying and drawing nodules independently 3. Unblinded Read confirming using an overread, labeling nodules (characteristics) confirming using an overread, labeling nodules (characteristics) 4. Subject info demographics, smoking history, pathology. demographics, smoking history, pathology. 5. Export Data to NCI-hosted database (public)

41. Database Implementation How to capture and collect all of this data? We have developed an internal standard for representing a region of interest (ROI) that is 3-D based on xml. This is portable across software drawing tools. We are also using xml to capture radiologist interpretation of nodule characteristics (shape, subtlety, etc.) by using a limited set of descriptors

42. Database Implementation How to capture and collect all of this data? We have designed and tested a communication protocol to send image data and xml messages Read Request messages (with a code/mechanism to distinguish blinded from unblinded read request) Read Request messages (with a code/mechanism to distinguish blinded from unblinded read request) Read Response messages (with a code/mechanism to distinguish blinded from unblinded read response) Read Response messages (with a code/mechanism to distinguish blinded from unblinded read response)

43. Database Implementation How to capture and collect all of this data? Designed and implemented database for each host site for all case data. Designed and are implementing the central NCI hosted database.

44. Database Implementation Communication Model Each Site Plays Dual Roles Each Site Plays Dual Roles As a Requesting Site As a Requesting Site Identify Case and collect data Identify Case and collect data Phase 1- Initial Review Phase 1- Initial Review Manage it through blinded and unblinded read process Manage it through blinded and unblinded read process Create database entry for case Create database entry for case Phase 4 – Demographics, Pathology Phase 4 – Demographics, Pathology Phase 5 – Export to NCI Phase 5 – Export to NCI NOTE: Site does not READ/MARK its own cases NOTE: Site does not READ/MARK its own cases As a Servicing Site As a Servicing Site Perform blinded (Phase 2) and unblinded (Phase 3) reads Perform blinded (Phase 2) and unblinded (Phase 3) reads

45. A B,C,D,E Initial Review, Anonymize Send Image Data XML Reading Assignment message XML Reading Response Message, Compile Responses Nodule Marking Tools SSH or SCP for transfer Other Subject data fields Linked to case 1 5 11 6 3 2 4 LIDC Message System X 7 9 8 10 Requesting Site Servicing Site

46. Access to LIDC Database Cases Exported to NCI Cases Exported to NCI NCI hosts Database NCI hosts Database Publicly Available Publicly Available Query Based on Data Elements Collected Query Based on Data Elements Collected Imaging Data such as Slice Thickness, etc. Imaging Data such as Slice Thickness, etc. Pathology or F/U Data Pathology or F/U Data Other Fields Other Fields Obtain Obtain Image Data including DICOM headers Image Data including DICOM headers Serial Imaging when available Serial Imaging when available Radiologists’ Identification, Contours and Characterization of Nodules Radiologists’ Identification, Contours and Characterization of Nodules Diagnosis Data (Path, Radiographic F/U, etc) whenever available Diagnosis Data (Path, Radiographic F/U, etc) whenever available Case Demographics whenever available Case Demographics whenever available Currently Implementing MIRC model (see infoRAD exhibit for demo) Currently Implementing MIRC model (see infoRAD exhibit for demo)

47. Database Implementation TASKS COMPLETED (see reports on website): Specification of Inclusion Criteria: Specification of Inclusion Criteria: CT scanning technical parameters CT scanning technical parameters Patient inclusion criteria Patient inclusion criteria Process Model for Data collection Process Model for Data collection Determination of Spatial "truth" Using Blinded and Unblinded reads Determination of Spatial "truth" Using Blinded and Unblinded reads Development of Boundary Drawing Tools Development of Boundary Drawing Tools Development and implementation of xml standard for ROIs Development and implementation of xml standard for ROIs

48. Database Implementation TASKS COMPLETED Defined Common Data Elements for LIDC Defined Common Data Elements for LIDC Database design – tables and relationships between tables Database design – tables and relationships between tables Communication protocol Communication protocol Establishing Public Database and Access Mechanism at NCI Establishing Public Database and Access Mechanism at NCI

49. Other Products Publications/Presentations LIDC Overview manuscript LIDC Overview manuscript Radiology 2004 Sep;232(3):739-748. Radiology 2004 Sep;232(3):739-748. Assessment Methodologies manuscript Assessment Methodologies manuscript Academic Radiology April 2004 Academic Radiology April 2004 ( ( Acad Radiol 2004; 11:462–475) Special Session SPIE Medical Imaging Special Session SPIE Medical Imaging Sunday evening session at SPIE, 2005 Sunday evening session at SPIE, 2005

50. Summary LIDC mission – to create public database LIDC mission – to create public database Current understanding of problem dictated multiple readers Current understanding of problem dictated multiple readers Multi-Institutions dictated distributed, asynchronous reads Multi-Institutions dictated distributed, asynchronous reads

51. Summary LIDC developed: LIDC developed: Process Model for Blinded and Unblinded Reads w/Multiple Readers Process Model for Blinded and Unblinded Reads w/Multiple Readers Infrastructure to Communicate Radiologist Expert Information (Markings, Contours, Labelings) Infrastructure to Communicate Radiologist Expert Information (Markings, Contours, Labelings) Data Elements –image, meta data (DICOM), radiologist markings, contours and labels, pathology, demographics Data Elements –image, meta data (DICOM), radiologist markings, contours and labels, pathology, demographics Data Representation Scheme (xml) Data Representation Scheme (xml) Communication (messaging) protocol Communication (messaging) protocol Database Design Database Design Mechanism to handle reader disagreement/variability Mechanism to handle reader disagreement/variability