Slide - 1 Confidential 3A Group Dr Maria Athelogou Grace Kim: Clarification in the study design: Since we have contours of lesions from many readers (for 1A, 1B, and 1C: will be), are we planning to use this as Gold standard? This means that we DO NOT require the radiologists’ visual readings to test inter- algorithm variation. Please confirm. This means in the study design that we will have a metric of overlap measures between a radiologist’s contour and result from an algorithm. - We have to define a metric for the overlap Answer Andy Buckler: discuss this during the next 3A call. My feeling is that we have two types of analyses: one being against true volume (in the phantom studies such as 1A and 1C directly, and indirectly by averaging readers results for the clinical data such as 1B), and the other type being MICCAI-type metrics such as volume overlap (and others from a variety of papers such as v8lbHcJ4rC9PiPtUQxIKSPSI&hl=en&ei=kzyrTcHzDITMgQfzypD0BQ&sa=X&oi=book_result&ct=result&resnum=4&ved=0CCAQ6AEwAw#v=onepag e&q=miccai%20metrics%20segmentation&f=false ). v8lbHcJ4rC9PiPtUQxIKSPSI&hl=en&ei=kzyrTcHzDITMgQfzypD0BQ&sa=X&oi=book_result&ct=result&resnum=4&ved=0CCAQ6AEwAw#v=onepag e&q=miccai%20metrics%20segmentation&f=false 3A group member contributions

Slide - 2 Confidential 3A Group Dr Maria Athelogou

Slide - 3 Confidential 3A Group Dr Maria Athelogou

Slide - 4 Confidential 3A Group Dr Maria Athelogou

Slide - 5 Confidential 3A Group Dr Maria Athelogou Keep it simple?

Slide - 6 Confidential 3A Group Dr Maria Athelogou

Slide - 7 Confidential 3A Group Dr Maria Athelogou

Slide - 8 Confidential 3A Group Dr Maria Athelogou

Slide - 9 Confidential 3A Group Dr Maria Athelogou Each algorithm needs a different computing time for the volume calculation. We need a appropriate metric for the time factor (performance).

Slide - 10 Confidential 3A Group Dr Maria Athelogou Hubert Beaumont

Slide - 11 Confidential 3A Group Dr Maria Athelogou Hubert Beaumont

Slide - 12 Confidential 3A Group Dr Maria Athelogou

Slide - 13 Confidential 3A Group Dr Maria Athelogou According to study design page 3 We need a exact Workflow for the measuring process (protocol)

Slide - 14 Confidential 3A Group Dr Maria Athelogou From last 3A call (April, 14th 2011) Regarding the definition of intra software variability, may I suggest instead to consider algorithm variability as a function of initial conditions? (e.g. From variation of the seed). It would give in the text: The aim of the study is twofold: To estimate inter-algorithm variability To estimate independently the variability of each algorithm regarding variation of initial condition (eg. the seed point) These estimations rely on the volume estimation of synthetic nodules from CT Asked: As all the evaluation is remotely processed via a batch processing, I cannot see how to interactively change contours afterward. How to check the result of a given segmentation and redraw part of the segmentation? If I have correctly understood, it would be quite tedious to process the first segmentation, download the result, check the contours, correct the segmentation, get the list of point, upload the list to the server and so on may be several time and this for a significant set of lesions. Hubert Beaumont

Slide - 15 Confidential 3A Group Dr Maria Athelogou Image Analysis algorithms and Batch Analysis Service conformity? May we give the users the possibility to download cases, if needed? Seed points could be predefined by QIBA Using predefined template for up- and download of control and result parameters. Clarify 3.3.1

Slide - 16 Confidential 3A Group Dr Maria Athelogou CTZ: Anthropomorphic phantoms have to be measured exact to guaranty connections to the SI Normals (page 2) CTZ : 2) Very clear algorithm classification page 2 3) make sure that digital phantom data are produced from physical objects instead of mathematical algorithms (page 7)

Slide - 17 Confidential 3A Group Dr Maria Athelogou David Gustafson

Slide - 18 Confidential 3A Group Dr Maria Athelogou David Gustafson

Slide - 19 Confidential 3A Group Dr Maria Athelogou David Gustafson

Slide - 20 Confidential 3A Group Dr Maria Athelogou David Gustafson

Slide - 21 Confidential 3A Group Dr Maria Athelogou David Gustafson

Slide - 22 Confidential 3A Group Dr Maria Athelogou David Gustafson

Slide - 23 Confidential 3A Group Dr Maria Athelogou McNitt-Gray, Michael A.B.: “ground truth.” Would it also be possible to use LIDC data, where ground truth is the aggregate of the radiologist readers on this data? If CoreLabs makes available the contours they generated from the 1B data (I understand from David Clunie they did record them, so in theory it is possible), then 3A might be able to use these as comparisons as “truth”. If not, then curating an evaluation set from LIDC, which would also have contours, would be possible as well. LIDC would have a larger set of lesions to draw from (hundreds, not 32). In both cases, first things to address would be: Format of contours – can you read in and decipher contour formats. LIDC used an xml format; CoreLabs presumably used some DICOM SR format (not sure) How to deal with four (LIDC) or five (QIBA 1B) contours so that you can use as a basis for comparison with algorithm. Do you want to compare actual contours? (voxel by voxel comparison of boundaries using something like STAPLE?) Or just compare outcome variable – volume, diameter, etc. And in each case, you will have 4 or 5 potential individual comparisons; or just take the average of the human contour results and compare algorithm to that. McNitt-Gray, Michael