1. QIBA CT Volumetrics Group 1B: (Patient Image Datasets) Teleconference Nov 5, 2008

2. Charge 1. To agree on questions to be answered by these Reference Datasets 2. To identify requirements for those datasets, based on questions to be answered 3. To identify existing datasets that can be leveraged to provide desired datasets

3. Questions (overview) 1. What level of accuracy and precision can be achieved in measuring tumor volumes in patient datasets? 2. What level of reproducibility in estimating change can be achieved when measuring tumors in phantom datasets? 3. What is the minimum detectable level of change that can be achieved when measuring tumors in patient datasets under a “No Change” condition? 4. What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? 5. What is the effect of slice thickness on estimating change in tumors using patient datasets?

4. Questions 1. What level of accuracy and precision can be achieved in measuring tumor volumes in patient datasets? (a) Investigate both bias and variance? (b) Comparison between volume and RECIST here? (c) inter-observer variability (d) Intra-observer variability (NOTE: here observer should be interpreted broadly – could be algorithm, could be human, could be combination). For this question:  lesions with “known size” will be used, where datasets with estimates of boundaries would be used.  Only a single time point is needed  LIDC datasets (thick and thin slices, lots of lesions)  Annotated RIDER datasets? Does this count as truth (currently these have single dimension measurements; could be extended to volumes via additional readers)

5. Questions 2. What level of reproducibility in estimating change can be achieved in measuring tumors in phantom datasets? (a) RECIST change vs. volume change (b) Investigate just variance (not bias)? (c) inter-observer variability (d) Intra-observer variability (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? For this question:  Variety of lesions with known size  Compare “size” of different lesions somehow (TBD) Use different existing lesions and treat them as though they were the same lesion at different time points. Use different existing lesions and treat them as though they were the same lesion at different time points. Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment) Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment)

6. Questions 3. What is the minimum detectable level of change that can be obtained in measuring tumors in patient datasets under a “No Change” condition? (a) RECIST change vs. volume change (b) Investigate just variance? (c) inter-observer variability (d) Intra-observer variability (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? For this question:  Coffee break experiment – lesions with “No change” condition  Variety of lesions (true size may be unknown)  Patient datasets with same lesions at different time points.  MSK Coffee Break experiment data  Extend their experiment with additional readers (RECIST only? Volumes?)

7. Questions 4. What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? (a) RECIST change vs. volume change (b) Investigate just variance? (c) inter-observer variability (d) Intra-observer variability (would be good to have more than 2 time points; may not exist in RIDER). (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? (f) Look at effects of lesion size for a specific (thin) slice thickness For this question:  Variety of lesions (true size may be unknown)  Patient datasets with same lesions at different time points.  Lesions may or may not have changed size – change is unknown.  RIDER datasets – unannotated as of yet. We have identified about 20 lesions of various sizes. Possible cases from RadPharm.

8. Questions 5. What is the effect of slice thickness on estimating change in tumors using patient datasets? (a) RECIST change vs. volume change (b) Investigate just variance? (c) inter-observer variability (d) Intra-observer variability (e) Change metric – absolute value? fractional change in volume/diameter? categorical variable? (f) Look at interactions between slice thickness and lesion size (and other lesion characteristics such as shape, margin, etc.) For this question:  Also Variety of lesions (true size may be unknown)  Patient datasets with same lesions at different time points, all done originally with thin slices; create a thick slice series (average together adjacent images).  Lesions may or may not have changed size – change is unknown.  Estimate change on thin and thick series and compare  Thin Slice RIDER datasets that can be fused together. Other cases from RadPharm?

9. Questions  NOTE: In many of these cases, we anticipate that “ground truth” or actual change in volume/size/density is unknown; we also anticipate that clinical outcome (progression or regression) will also be unknown

10. Existing Resources  RIDER thick slice annotated Annotated by two readers with single dimension (e.g. RECIST) Annotated by two readers with single dimension (e.g. RECIST) Thick sections (5mm) Thick sections (5mm) 23 cases, 99 series 23 cases, 99 series Used a subset of these cases for NIST Biochange 2008 Used a subset of these cases for NIST Biochange 2008  RIDER thin slice Thin section (1.25 mm) with >= 2 visits Thin section (1.25 mm) with >= 2 visits Looking at mix of lesions now (each patient will have multiple lesions, but may not want to use each one) Looking at mix of lesions now (each patient will have multiple lesions, but may not want to use each one) 7 lesions > 10 mm (from 2 different patients)7 lesions > 10 mm (from 2 different patients) 11 lesions < 10 mm (from 2 different patients)11 lesions < 10 mm (from 2 different patients) No annotations No annotations

11. Existing Resources  RIDER – MSK Coffee Break Experiment (No Change Condition) 32 NSCLC patients 32 NSCLC patients Imaged twice on the same scanner w/in 15 minutes Imaged twice on the same scanner w/in 15 minutes Thin section (1.25 mm) images Thin section (1.25 mm) images Manual linear measurements performed by 3 readers; volume obtained from algorithm. Manual linear measurements performed by 3 readers; volume obtained from algorithm.

12. Existing Resources  RadPharm Will search their own database for cases having thin slice and multiple time points Will search their own database for cases having thin slice and multiple time points Then search for good candidate lesions Then search for good candidate lesions

13. Next Steps  Next Conf Call Continue to investigate Inventory of Patient Datasets (RIDER thin slice, RadPharm, etc.) Continue to investigate Inventory of Patient Datasets (RIDER thin slice, RadPharm, etc.) Continue to Refine Questions and Experimental Design (similar to what 1A has done) Continue to Refine Questions and Experimental Design (similar to what 1A has done)