1. Slide - 1 Confidential

2. Slide - 2 Confidential

3. Slide - 3 Confidential Interalgorithm Study using CT Images of synthetic nodules……….

4. Slide - 4 Confidential  An inter-algorithm study, in the same way QIBA has been working on inter-reader,inter-scanner, and inter-site. We will also connect it to the analysis section of QIBA Profile.  The aim of the study is to estimate inter- and intra-algorithm variability by the volume estimation of synthetic nodules from CT scans of an anthropomorphic phantom (according to the work of the QIBA 1A Group (see Dr. Petrick‘s paper, SPIE 2011)  For the impelmentation of the obejctives a challenge could be organized (?) Objectives of the QIBA 3A Group

5. Slide - 5 Confidential Motivation for the Study Study Design Dataset Algorithms Analysis Protocol Analysis method Result evaluation Result presentation Result evaluation for the QIBA protocol Study Motivation

6. Slide - 6 Confidential Motivation

7. Slide - 7 Confidential  Changes in nodules volume is important for diagnosis, therapy planning, therapy response evaluation  Measuring volume changes requires high accuracy in measurement of absolute volume  Ground true has to be exact measured. This is not the case by data annotation (inter- and intra-observer variability)  Volumes of synthetic nodules are measured (high accuracy)  Therefore it make sense to use such data (as ground truth) in order to calculate accuracy measurement of algorithms  The study results could be combined with the QIBA 1A Group work. This combination will improve the QIBA volumetric CT Profile development.  The study has to be complementary to the study of the 1A Group:  1A study: Radiologists and synthetic nodules (inter- intra- observer variability)  3A Study: Algorithms and synthetic nodules: intra-algorithms variability)  Using of the same technological basis (scanner type, Philips (1A Group data, synthetic nodules)) to secure independence for inter-scanner variability (except BIAS between different serial numbers) Motivation

8. Slide - 8 Confidential  Describe the overall analysis procedure (use as example the Bio-change Challenge Protocol http://www.nist.gov/itl/iad/dmg/biochangechallenge.cfm: How to Participate in Biochange Challenge (Dr Fenimore)Bio-change Challenge Protocol http://www.nist.gov/itl/iad/dmg/biochangechallenge.cfm Download and read the Bio-change Challenge Protocol and email or fax Statement of Interest to NIST.Bio-change Challenge Protocol Download the Bio-changeChallengeSeries from the NIST FTP site and from the NBIA RIDER collection as described in the Protocol.Bio-changeChallengeSeries Run your change analysis algorithm or CAD tool in your lab on the validation data. Report your change results in one of the required formats and send a Participation Agreement signed by the your team leader to NIST by January 18, 2011. NIST will analyze the reported results, comparing them to the limited available ground truth as described in the Protocol. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without publicly identifying individual scores by Participant. Analysis procedure (analysis Protocol according Bio-change or Volcano Challenge?)

9. Slide - 9 Confidential Has to be discussed during the meeting March 17th 2011  The aim of the study is not a challenge to know:  Who is the best image analysis algorithm  The aim of the study is to gain knowledge for the QIBA profile (paragraph 9 concerns image analysis)  Avoid competition  Support Cooperation, conjoint approach OUR CONTEXT is QIBA

10. Slide - 10 Confidential  Phantom data as used for the 1A Group Study (see Dr. Patrick’s publication, SPIE 2011)  1C clinical data (Dr Fenimore/ Phantom –synthetic data)  Training dataset  Test dataset  Or only test dataset (analyze without trainings dataset) data pool specification (simple geometries, up to anatomical representations) (if clear than we could propose an experimental design)  Nodule classification concerning position. Shape and margins of the nodule … (according to the QIBA Profile)  Algorithm classification  Analysis protocol:  each algorithm is applied to the dataset separately  each algorithms is applied to the “training” data set  each algorithms is applied to the “test” data set  Statistical analysis of the analysis results of each algorithm separately Data/Nodules/Algorithms

11. Slide - 11 Confidential  Academia and non profit organizations  Industrial vendors (for example possible vendors, according to the Volcano 2009 challenge could be: Siemens, Phipps, MeVis, Kitware, Definiens, VIA CAD etc…  Description/Classification of the algorithms: according to the grade of user intervention is needed (for example Volcano’09, A. P. Reeves et al):  Totally automatic using seed points  Limited parameter adjustment (on less than 15% of the cases)  Moderate parameter adjustment (on less than 50% of the cases)  Extensive parameter adjustment (more than 50% of the cases)  Limited image/boundary modification (on less than 15% of the cases)  Moderate image/boundary modification (on less than 50% of the cases)  Extensive image/boundary modification (more than 15% of the cases) Or:  M. Gavrielide et al., Noncalcified Lung Nodules, Volumetric assessment with thoracic CT (RSNA 2009):  Semi automated  Manually derived Semi automated algorithms are typically initiated by defining a region of interest around a nodule or by a user-provided point inside the nodule area. Depending on the application, segmentation algorithms are then employed to delineate nodules from the surrounding lung parenchyma and neighboring structures such as attached vasculature and pleural surfaces. Manually derived methods require users to interactively delineate nodule boundaries, typically in a section-by-section fashion; this is followed by an application of three-dimensional software to merge the two-dimensional boundaries into a volume. The estimate of nodule volume is then based on the total number of voxels within the segmented region. The majority of volume measurement methods use voxel counting Algorithm vendors and Algorithm classification examples from the literature

12. Slide - 12 Confidential IEEE Transactions on medical imaging, vol. 28 MICCAI Algorithm Classification:  An automatic segmentation algorithm does not require any user intervention.  A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation.  An interactive algorithm requires manual editing of the final results. Abstract: In this paper, we present the organization of a competition of liver tumor segmentation techniques. The liver tumor segmentation competition is part of the workshop "3D Segmentation in the Clinic: A Grand Challenge II" at Medical Image Computing and Computer Assisted Intervention 2008 conference. The goal of this contest is to compare the he performance of different algorithms for segmenting liver tumor from contrast enhanced CT images. Several organizing topics are described, contrast including the motivation for organizing this competition, training and testing dataset evaluation methods Algorithm Classification according 3D Segmentation in the Clinic: A Grand Challenge II - Liver Tumor Segmentation, Xiang Deng and Guangwei Du, Center for Medical Imaging Validation, Siemens Corporate Technology (citation)

13. Slide - 13 Confidential MICCAI Liver Challenge proposed metrics (citation)

14. Slide - 14 Confidential

15. Slide - 15 Confidential  Sensitivity Sens = TP/TP+FN  Specificity Spec = TN/TN+FP  Accuracy = (TN +TP)/all Lucila Ohno-Machado An introduction to calibration and discrimination methods, HST951 Medical Decision Support Harvard Medical School, Massachusetts Institute of Technology

16. Slide - 16 Confidential  Box plot (for example: for each data set, nodule volume vs. method, relative Volume) Vol rel = 100%* (Vol est – Vol true )/ Vol true Vol rel = 100%* (Vol alg – Vol synthetic nodule )/ Vol synthetic nodule We propose: A simple but well defined and reliable approach Other

17. Slide - 17 Confidential We propose: keep it simple Algorithm Classification: An automatic segmentation algorithm does not require any user intervention. A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation. An interactive algorithm requires manual editing of the final results.

18. Slide - 18 Confidential Known: Vsn Measurement: Valg StdvValg (1) <- same Phantom, multiple measurements, same algorithm, multiple seed points (two groups of algorithms: semiautomatic, manually) StdvValg (2)<- different phantoms (according QIBA profile: nodule classification), same algorithm: StdV alg (nodule class) StdV alg (all nodule classes) Bias out of digitalization in the area of lesion margins (Voxels/Pixels on the lesion margins, mixed pixels) ( Group 1A, Group 1C, Group 1B, QIBA VIA Profile, Definitions and how should be calculated? ) Achievements: Transparent and Objective comparability between different algorithms One independent Error value for further investigations of clinical data Algorithm Evaluation Method

19. Slide - 19 Confidential Definition of the Study Data/Measurements/Metrics /  Which data:  1A (Phantom/synthetic data) ( defined experimental conditions, experimental design Phantom Data for dV?)  1B Clinical data according to that: do we have we to separate the study design as following?  Study design for Phantom/Synthetic data (focus)  Study design for Clinical data  Metrics

20. Slide - 20 Confidential Decision

21. Slide - 21 Confidential  Each participant has to be informed (only their own algorithm results)  Publication of the results (all participants)  Using the results for the QIBA protocol: knowledge exploitation for the QIBA protocol  Resulting recommendations for FDA (Is this possible?) Result information for the participants