QIBA CT Volumetrics Group 1B: (Patient Image Datasets)

Charge 1. To agree on questions to be answered by these Reference Datasets 2. To identify requirements for those datasets, based on questions to be answered 3. To identify existing datasets that can be leveraged to provide desired datasets

Questions (overview) 1. What level of accuracy and precision can be achieved in measuring tumor volumes in patient datasets? 2. What level of reproducibility in estimating change can be achieved when measuring tumors in phantom datasets? 3. What is the minimum detectable level of change that can be achieved when measuring tumors in patient datasets under a “No Change” condition? 4. What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? 5. What is the effect of slice thickness on estimating change in tumors using patient datasets?

Initially Agreed to First Pursue: 1. What level of accuracy and precision can be achieved in measuring tumor volumes in patient datasets? 2. What level of reproducibility in estimating change can be achieved when measuring tumors in phantom datasets? 3. What is the minimum detectable level of change that can be achieved when measuring tumors in patient datasets under a “No Change” condition? 4. What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? 5. What is the effect of slice thickness on estimating change in tumors using patient datasets?

Next Steps  Define requirements and Design Experiments for these two questions

Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes 1. Specific Aims (a) Investigate both bias and variance of both readers and algorithm-assisted readers in measuring volumes, diameters and bi-directional diameters of lesions (b) Investigate inter-observer variability in each task (c) Investigate Intra-observer variability in each task ??? (NOTE: here observer should be interpreted broadly – as reader measuring manually for diameters as well as algorithm-assisted reader measuring contours).

Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes 1. METHODS and MATERIALS  LIDC dataset  lesions with “known size” will be used (size based on contours from 4 LIDC readers)  Each lesion will have boundaries  Only a single time point is needed (no followup needed, no diagnosis needed)  Need to identify which nodules to use – criteria?

Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes 1. METHODS and MATERIALS  Like NIST Biochange 2008, lesions are identified and coordinates provided to readers.  Reader tasks Manually mark 2 Diameters (w/o LIDC marks) Manually mark 2 Diameters (w/o LIDC marks) Longest diameter and perpendicular diameterLongest diameter and perpendicular diameter Also semiautomated contour of lesion Also semiautomated contour of lesion From contour, determine volume, longest diameter and diameter perpendicular to longest diameterFrom contour, determine volume, longest diameter and diameter perpendicular to longest diameter Have readers perform some cases > 1 (intra reader variation) Have readers perform some cases > 1 (intra reader variation)

Data Collected Nodule # coords LIDC vol LIDC Longest Diam LIDC Perp Diam R1 vol R1 LD from cont our R1 PD from conto ur R1 LD R1 PD R2 vol

Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes 1. METHODS and MATERIALS  Analyses LIDC would be considered “truth” LIDC would be considered “truth” Estimate bias of each reader Estimate bias of each reader VolumeVolume Diam (manual and Assisted)Diam (manual and Assisted) Product of diameters (LD x PD) (Manual and assisted)Product of diameters (LD x PD) (Manual and assisted) Estimate inter-reader variability Estimate inter-reader variability Reader vs. “gold standard”Reader vs. “gold standard” Reader vs. readerReader vs. reader Intra-reader variability Intra-reader variability

Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes 1. Questions 1. How many cases? 2. How many readers? 3. Case composition (all spherical? Some spiculated? Range of sizes?) 4. Enough of each subgroup to perform a stat. sign analysis? 1.(reader bias on spherical nodules is XX; 2.(reader bias on spiculated nodules is YY) 5. Do we want to do both size and shape subgroup analyses?

Questions

Question 2 – What is the min detectable level of change in patient datasets under a “No Change” condition? 1. Specific Aims (a) For patient datasets acquired over a very short time interval (presumably the “no change” condition”) investigate variance of both readers and algorithm-assisted readers in measuring change in volume, diameter and bi-directional diameters of lesions (here, the expected value of the change should be zero) (b) Investigate several change metrics such as: (a)Absolute value of change (b)fractional change in volume/diameter (c)Categorical variables (progression, regression, etc.)?? (c) Investigate inter-observer variability in each task (d) Investigate Intra-observer variability in each task ??? (NOTE: again, observer should be interpreted broadly – as reader measuring manually for diameters as well as algorithm-assisted reader measuring contours).

Existing Resources  RIDER – MSK Coffee Break Experiment (No Change Condition) 32 NSCLC patients 32 NSCLC patients Imaged twice on the same scanner w/in 15 minutes Imaged twice on the same scanner w/in 15 minutes Thin section (1.25 mm) images Thin section (1.25 mm) images Manual linear measurements performed by 3 readers; volume obtained from algorithm. Manual linear measurements performed by 3 readers; volume obtained from algorithm.

Questions 1. METHODS and MATERIALS  MSK Coffee break experiment patient datasets Repeat scan of lesions over short time interval --- “No change” condition Repeat scan of lesions over short time interval --- “No change” condition True size is unknown True size is unknown

Questions 1. METHODS and MATERIALS  Like NIST Biochange 2008, lesions are identified and coordinates provided to readers.  Reader tasks For each lesion, manually mark 2 Diameters For each lesion, manually mark 2 Diameters Longest diameter and perpendicular diameterLongest diameter and perpendicular diameter Also each reader obtains a semiautomated contour of entire volume of lesion Also each reader obtains a semiautomated contour of entire volume of lesion From contour, determine volume, longest diameter and diameter perpendicular to longest diameterFrom contour, determine volume, longest diameter and diameter perpendicular to longest diameter Keep actual contour coordinates/mask (TBD)Keep actual contour coordinates/mask (TBD) Randomize order of cases so reader does not simultaneously review both scans of a patient (does not see both scans of same lesion); has to contour them independently. Randomize order of cases so reader does not simultaneously review both scans of a patient (does not see both scans of same lesion); has to contour them independently.

Data Collected QIBA Patient ID # Scan# Nodule # coords R1 vol R1 LD from contour R1 PD from contour R1 LD manual R1 PD manual R2 vol 111 (320,267, 52) (310, 248, 48) (221,335, 13) (215, 340, 15) 28

Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes 1. METHODS and MATERIALS  Analyses Estimate variance of different change metrics for Estimate variance of different change metrics for VolumeVolume Diam (manual and Assisted)Diam (manual and Assisted) Product of diameters (LD x PD) (Manual and assisted)Product of diameters (LD x PD) (Manual and assisted) Estimate inter-reader variability Estimate inter-reader variability Intra-reader variability ?? (can we do that with data we would have? Or would we have to specifically introduce the exact same cases multiple times?) Intra-reader variability ?? (can we do that with data we would have? Or would we have to specifically introduce the exact same cases multiple times?)

Questions to be Pursued in a Future Phase

Questions 2. What level of reproducibility in estimating change can be achieved in measuring tumors in phantom datasets? (a) RECIST change vs. volume change (b) Investigate just variance (not bias)? (c) inter-observer variability (d) Intra-observer variability (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? For this question:  Variety of lesions with known size  Compare “size” of different lesions somehow (TBD) Use different existing lesions and treat them as though they were the same lesion at different time points. Use different existing lesions and treat them as though they were the same lesion at different time points. Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment) Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment)

Questions 4. What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? (a) RECIST change vs. volume change (b) Investigate just variance? (c) inter-observer variability (d) Intra-observer variability (would be good to have more than 2 time points; may not exist in RIDER). (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? (f) Look at effects of lesion size for a specific (thin) slice thickness For this question:  Variety of lesions (true size may be unknown)  Patient datasets with same lesions at different time points.  Lesions may or may not have changed size – change is unknown.  RIDER datasets – unannotated as of yet. We have identified about 20 lesions of various sizes. Possible cases from RadPharm.

Questions 5. What is the effect of slice thickness on estimating change in tumors using patient datasets? (a) RECIST change vs. volume change (b) Investigate just variance? (c) inter-observer variability (d) Intra-observer variability (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? (f) Look at interactions between slice thickness and lesion size (and other lesion characteristics such as shape, margin, etc.) For this question:  Also Variety of lesions (true size may be unknown)  Patient datasets with same lesions at different time points, all done originally with thin slices; create a thick slice series (average together adjacent images).  Lesions may or may not have changed size – change is unknown.  Estimate change on thin and thick series and compare  Thin Slice RIDER datasets that can be fused together. Other cases from RadPharm?