End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 The Lung Image Database Consortium (LIDC): Fundamental Issues for the Creation of a Resource for the Image Processing Research Community

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Exhibit Learning Objectives: Learn about the LIDC’s goals and methods for creating: Learn about the LIDC’s goals and methods for creating: a publicly available database, for the development, training, and evaluation of Computer-Aided Diagnosis (CAD) methods, for lung cancer detection and diagnosis using helical CT.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Learning Objectives: Learn about challenges in interpreting CT image data sets for the detection and diagnosis of lung cancer Learn about challenges in interpreting CT image data sets for the detection and diagnosis of lung cancer

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Learning Objectives: Learn about the intricacies of establishing spatial “truth” for lesion location and boundary. Learn about the intricacies of establishing spatial “truth” for lesion location and boundary.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 The Challenge: Lung Cancer Cancer of the lung and bronchus is the leading fatal malignancy in the United States. Cancer of the lung and bronchus is the leading fatal malignancy in the United States. Five-year survival is low, but treatment of early-stage disease improves chances of survival considerably. Five-year survival is low, but treatment of early-stage disease improves chances of survival considerably.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 The Challenge: Lung Cancer Given: Given: Promising results from recent studies involving the use of helical computed tomography (CT) for the early detection of lung cancer Promising results from recent studies involving the use of helical computed tomography (CT) for the early detection of lung cancer As well as rapid developments in Multi-detector CT (MDCT) technology which provide for the possibility of the detection of smaller lung nodules and offers a potentially effective tool for earlier detection. As well as rapid developments in Multi-detector CT (MDCT) technology which provide for the possibility of the detection of smaller lung nodules and offers a potentially effective tool for earlier detection. There has been an increased interest in computer-aided diagnosis (CAD) techniques applied to CT imaging for lung cancer to assist radiologists’ with their decision-making. There has been an increased interest in computer-aided diagnosis (CAD) techniques applied to CT imaging for lung cancer to assist radiologists’ with their decision-making.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 The NCI Response: Forming the LIDC To stimulate research in the area of CAD, the National Cancer Institute (NCI) formed a consortium of institutions to develop the standards and consensus necessary for constructing an image database resource of thoracic helical CT images.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Motivation The development of CAD methods by the imaging research community would be facilitated and enhanced through access to a repository of CT image data The development of CAD methods by the imaging research community would be facilitated and enhanced through access to a repository of CT image data

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Motivation The development of CAD methods by the imaging research community would be facilitated and enhanced through access to a repository of CT image data The development of CAD methods by the imaging research community would be facilitated and enhanced through access to a repository of CT image data (1) It would provide data to researchers without access to clinical images

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Motivation The development of CAD methods by the imaging research community would be facilitated and enhanced through access to a repository of CT image data The development of CAD methods by the imaging research community would be facilitated and enhanced through access to a repository of CT image data (1) It would provide data to researchers without access to clinical images (2) It would also allow for meaningful comparisons of different CAD methods

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 The LIDC This consortium - called the Lung Image Database Consortium (LIDC) - seeks to establish standard formats and processes by which to manage lung images and the related technical and clinical data that will be used by researchers to develop, train and evaluate CAD algorithms for lung cancer detection and diagnosis.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Member Institutions Five institutions were selected to form the Lung Image Database Consortium (LIDC) Five institutions were selected to form the Lung Image Database Consortium (LIDC)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Member Institutions Cornell University UCLA University of Chicago University of Iowa University of Michigan

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Steering Committee Cornell UniversityDavid Yankelevitz Anthony P. Reeves UCLAMichael F. McNitt-Gray Denise R. Aberle University of ChicagoSamuel G. Armato III Heber MacMahon University of IowaGeoffrey McLennan Eric A. Hoffman University of MichiganCharles R. Meyer Ella Kazerooni NCILaurence P. Clarke Barbara Y. Croft

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Contributing Participants Claudia Henschke, Cornell David Gur, U. of Pittsburgh Robert Wagner, FDA Nicholas Petrick, FDA Lori Dodd, NCI Ed Staab, NCI Daniel Sullivan, NCI Houston Baker, NCI Carey Floyd, Duke Aliya Husain, U. of Chicago Matthew Brown, UCLA Christopher Piker, U. of Iowa Peyton Bland, U. of Michigan Andinet Asmamaw, Cornell Richie Pais, UCLA Antoni Chan, Cornell Gary Laderach, U. of Michigan Junfeng Guo, U. of Iowa Charles Metz, U. of Chicago Roger Engelmann, U. of Chicago Adam Starkey, U. of Chicago Jim Sayre, UCLA Mike Fishbein, UCLA Andy Flint, U. of Michigan Barry DeYoung, U. of Iowa Brian Mullan, U. of Iowa Madeline Vazquez, Cornell

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Mission The mission of the LIDC is the sharing of lung images, especially low-dose helical CT scans of adults screened for lung cancer, and related technical and clinical data for the development and testing of computer-aided detection and diagnosis technology

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Principal Goals To establish standard formats and processes for managing thoracic CT scans and related technical and clinical data for use in the development and testing of computer-aided diagnostic algorithms.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Principal Goals To establish standard formats and processes for managing thoracic CT scans and related technical and clinical data for use in the development and testing of computer-aided diagnostic algorithms. To develop an image database as a web- accessible international research resource for the development, training, and evaluation of computer- aided diagnostic (CAD) methods for lung cancer detection and diagnosis using helical CT.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 The Database The database will contain:The database will contain: 1) a collection of CT scan images 2) a searchable relational database

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Fundamental Issues for the LIDC

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 LIDC Challenge #1 - Define a Nodule Though at first this seems trivial, the LIDC had significant discussion about what to include and what not to include as a nodule Though at first this seems trivial, the LIDC had significant discussion about what to include and what not to include as a nodule A Nodule is part of a spectrum of focal abnormalities. This spectrum includes scars, cancers, benign lesions, calcified lesions, etc.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 What is a Nodule? “nodule: any pulmonary or pleural lesion represented in a radiograph by a sharply defined, discrete, nearly circular opacity 2-30 mm in diameter” from the Fleischner Society's Glossary of Terms for Thoracic Radiology (AJR 1984)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 “nodule: round opacity, at least moderately well marginated and no greater than 3 cm in maximum diameter” from the Fleishner Society's Glossary of Terms for CT of the Lungs (Radiology 1996) What is a Nodule?

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Nodule

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Is this a Nodule?

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 NOTE: This is the slice which was shown earlier

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 What is a Nodule? Focal Abnormalities ()Nodules a spectrum of abnormalities Scar Spiculated Nodule Calcified Nodule

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 LIDC Challenge #1 - Define a Nodule LIDC Response is to develop a Nodule Visual Library using: LIDC Response is to develop a Nodule Visual Library using: Cases that ARE in Nodule portion of spectrum Cases that ARE in Nodule portion of spectrum Cases that are OUTSIDE Nodule portion of spectrum Cases that are OUTSIDE Nodule portion of spectrum Classification by Thoracic Radiologists Classification by Thoracic Radiologists In Development Now In Development Now Expected Completion Feb Expected Completion Feb 2004.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Truth Assessment Investigators will require “truth” information Investigators will require “truth” information

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Truth Assessment Investigators will require “truth” information Investigators will require “truth” information location of nodules location of nodules

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Truth Assessment Investigators will require “truth” information Investigators will require “truth” information location of nodules location of nodules spatial extent of nodules spatial extent of nodules

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Truth Assessment Investigators will require “truth” information Investigators will require “truth” information location of nodules location of nodules spatial extent of nodules spatial extent of nodules Spatial “Truth” will be estimated by “Radiologic Truth” Spatial “Truth” will be estimated by “Radiologic Truth”

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 LIDC Challenge #2 Define the Boundary of a Nodule Though it seems that the boundary of a nodule should be easy to define, we (and others) have found that there is considerable inter-reader variability in defining the boundary of a nodule. Though it seems that the boundary of a nodule should be easy to define, we (and others) have found that there is considerable inter-reader variability in defining the boundary of a nodule. This is difficult enough with a solid nodule, but even more difficult with spiculated nodules, ground glass nodules or non-solid nodules. This is difficult enough with a solid nodule, but even more difficult with spiculated nodules, ground glass nodules or non-solid nodules.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 SPICULATED NODULE Instructions to Thoracic Radiologists were “Draw the Boundary of the Nodule”

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 SPICULATED NODULE Expert Number 1 Contour

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 SPICULATED NODULE Expert Number 2 Contour

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 SPICULATED NODULE Comparison of Contours

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Five radiologists using 3 drawing methods: Five radiologists using 3 drawing methods: One manual 3-panel (3D) drawing method One manual 3-panel (3D) drawing method Two different semiautomatic 3D methods Two different semiautomatic 3D methods Reader and Method Variability in Drawing Boundaries

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Case 5, Slice 19

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 1 - Method 1

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 1 - Method 2

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 1 - Method 3

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 2 - Method 1

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 2 - Method 3

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 3 - Method 1

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 3 - Method 2

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 3 - Method 3

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 4 - Method 1

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 4 - Method 2

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 4 - Method 3

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 5 - Method 1

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Radiologist 5 - Method 3

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 For each voxel, sum the number of occurrences (across reader and method combinations) that it was included as part of the nodule For each voxel, sum the number of occurrences (across reader and method combinations) that it was included as part of the nodule Create a probabilistic map of nodule voxels Create a probabilistic map of nodule voxels Higher probability voxels are shown as brighter; lower probability are darker Higher probability voxels are shown as brighter; lower probability are darker Can use apply a threshold and show only voxels > some prob. Value if desired. Can use apply a threshold and show only voxels > some prob. Value if desired. Create a Probabilistic Description of Nodule Boundary

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Probabilistic Description of Boundary

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Apply Threshold if Desired

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 LIDC Challenge #2 Define the Boundary of a Nodule Do we need to reconcile these Boundaries? Do we need to reconcile these Boundaries? LIDC’s answer is no. LIDC’s answer is no. LIDC Approach will be to: LIDC Approach will be to: Come to a consistent definition of the desired boundary (include just solid portion? non-solid portion?) Come to a consistent definition of the desired boundary (include just solid portion? non-solid portion?) Assess reader variability of contours Assess reader variability of contours Construct a probabilistic description of boundaries to capture reader variability Construct a probabilistic description of boundaries to capture reader variability

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 LIDC Challenge #3- Data Collection Process Recent research has demonstrated that Single reads are not sufficient – At least two and perhaps four readers may be required. Recent research has demonstrated that Single reads are not sufficient – At least two and perhaps four readers may be required. Not practical to do joint readings across five institutions Not practical to do joint readings across five institutions LIDC Will NOT do a forced consensus read. LIDC Will NOT do a forced consensus read.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 LIDC Challenge #3- Data Collection Process Will do a Two-Staged Process: Will do a Two-Staged Process: Perform independent (Blinded) readings of cases by multiple radiologists Perform independent (Blinded) readings of cases by multiple radiologists Compile readings and redistribute composite readings Compile readings and redistribute composite readings Perform a Second, Unblinded read by same set of radiologists Perform a Second, Unblinded read by same set of radiologists Each reader can see readings of every other reader. Each reader can see readings of every other reader. No forced consensus No forced consensus Capture probabilistic detection (e.g. a nodule can be identified by 3 of 4 readers) and probabilistic contours. Capture probabilistic detection (e.g. a nodule can be identified by 3 of 4 readers) and probabilistic contours.

80 LIDC Process Model 2.4 October 2003 Overview Prerequisites Prerequisites major data collection steps, and major data collection steps, and data collected at each step. data collected at each step.

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Prerequisites Actions Data Collected IRB approvals Participants Scanned as part of Study/Clinical Program Non-Image Data Demographic Data Labeling Image Quality score Scan Classification Patient Classification Image Data/Dicom Data LIDC Activities CT Scan Criteria Apply Inclusion Criteria -Meets Scan Parameter Criteria [NOTE: All NLST & ELCAP eligible] Apply Inclusion Criteria: -IF Meets Scan Parameter Criteria [NOTE: All NLST & ELCAP eligible] - AND IF Meets Patient Inclusion Criteria Patient/Nodule Criteria Apply Inclusion Criteria: -IF Meets Scan Parameter Criteria [NOTE: All NLST & ELCAP eligible] - AND IF Meets Patient Inclusion Criteria -THEN Include in Db, Label Nodule Characteristics and Score Image Quality Labeling Vocabulary Image Quality Criteria

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Prerequisites Radiologist Review Process (described next) Identified lesions for each condition: Each reader, blinded and unblinded read Location Outline Label Definition of Nodules to be included in Db Agreement on Marking /Contouring process LIDC Activities Actions Data Collected Reader 1 Reader 2 Reader 3 Reader 4

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Blinded Reads – Each Reader Reads Independently (Blinded to Results of Other Readers)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 1 Blinded Read for Reader 1 – Marks Only One Nodule

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 2 Blinded Read for Reader 2 – Marks Two Nodules (Note: One nodule is same as Reader 1)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 3 Blinded Read for Reader 3 – Marks Two Nodules (Note: Again, One nodule is same as for Reader 1)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 4 Blinded Read for Reader 4 – Did Not Mark Any Nodules

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 UnBlinded Reads – Readings in Which Readers Are Shown Results of Other Readers Each Reader Marks Nodules After Being Shown Results From Other Readers’ Blinded Reads (Each Reader Decides to Include or Ignore).

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 1 Unblinded Read for Reader 1 – Now Marks Two Nodules (Originally only marked one)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 2 Unblinded Read for Reader 2 – Still Marks Two Nodules (No Change)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 3 Unblinded Read for Reader 3 – Now Marks Three Nodules (Originally only marked two)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Reader 4 Unblinded Read for Reader 4 – Now Marks Three Nodules (Originally did not mark any)

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 4 Markings 2 Markings Composite on Unblinded Reads for All Four Readers

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Database Implementation TASKS COMPLETED (see reports on website): Specification of Inclusion Criteria: Specification of Inclusion Criteria: CT scanning technical parameters CT scanning technical parameters Patient inclusion criteria Patient inclusion criteria Process Model for Data collection Process Model for Data collection Determination of Spatial "truth" Using Blinded and Unblinded reads Determination of Spatial "truth" Using Blinded and Unblinded reads Development of Boundary Drawing/Contouring Tools Development of Boundary Drawing/Contouring Tools

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Database Implementation TASKS ONGOING (expected completion date) Definition of Nodule - Nodule Visual Library (Feb 04) Definition of Nodule - Nodule Visual Library (Feb 04) Evaluation of Boundary Variability (Feb 04): Evaluation of Boundary Variability (Feb 04): Inter-Reader Variability Inter-Reader Variability Boundary Drawing Tool Variability Boundary Drawing Tool Variability

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Implementation Timeline TaskDate Expected Specify Complete Data Model Jan 04 Specify Complete Data Model Jan 04 Specify LIDC internal workflow Jan 04 Specify LIDC internal workflow Jan 04 Data passing, Performing reviews Data passing, Performing reviews Initial implementation, testing workflow Jan/Feb 04 Initial implementation, testing workflow Jan/Feb 04 Database Implementation- Start Jan 04 Database Implementation- Start Jan 04 Database Implementation- Completion Mar/Apr 04 Database Implementation- Completion Mar/Apr 04 Implement Public Interface to Database Apr/May 04 Implement Public Interface to Database Apr/May 04 PUBLIC ACCESS TO CASES – EXPECTED MAY/JUN 04 PUBLIC ACCESS TO CASES – EXPECTED MAY/JUN 04

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Publications/Presentations LIDC Overview manuscript LIDC Overview manuscript In Preparation, submission in 1 st Quarter 04 In Preparation, submission in 1 st Quarter 04 Assessment Methodologies manuscript Assessment Methodologies manuscript In Preparation, submission in 1 st Quarter 04 In Preparation, submission in 1 st Quarter 04 Special Session SPIE Medical Imaging Special Session SPIE Medical ImagingSPIE Medical ImagingSPIE Medical Imaging Sunday evening Feb 15, 2004 Sunday evening Feb 15, 2004

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 To learn more about the LIDC: Return for CME Category 1 Credit: Monday – Thursday Monday – Thursday 12:15 pm to 1:15 pm 12:15 pm to 1:15 pm At these times, LIDC members will be here to describe the efforts of the consortium, this exhibit and any other questions you might have

End/Return to Home End/Return to Home LIDC Education Exhibit - RSNA /99 Return to Exhibit Home Page Click Here Click Here