1 R2 ImageChecker CT CAD PMA: Clinical Results Nicholas Petrick, Ph.D. Office of Science and Technology Center for Devices and Radiological Health U.S. Food and Drug Administration

2 Outline Applicability of A z in analysis A z is same as area under the curve (AUC) Pool of CT cases for clinical study Defining actionable nodules by panel of experts Clinical studies Primary analysis: analysis using fixed expert panel Secondary analysis: analysis using random panels of experts Measurement of CAD standalone performance Algorithm’s performance with no reader involvement

3 Applicability of A z in analysis Average reader ROC Curves (pre/post CAD) Pre-CAD ROC Post-CAD ROC

4 Applicability of A z in analysis Pre and post-CAD curves do not cross No substantial pre/post-CAD crossing in either averaged or individual ROC curves A z is an appropriate performance measure A z used as figure of merit in all analysis

5 Pool of CT Cases Nodule cases Documented cancers Primary neoplasm or extrathoracic neoplasm with presumptive spread to lungs Cases were allowed to contain non-nodule, pathologic processes (e.g., pneumonia, emphysema, etc.) Non-nodule cases Normal cases No nodule deemed present by site P.I. Primarily relied upon original radiology report History of cancer, radiation therapy, or even previous thorocatomy allowed

6 Defining Actionable Nodules by Panel of Experts ‘Actionable’ nodules are objects of interest Panel of expert radiologists identify actionable nodules Nodules defined using a 2-pass process

7 Defining Actionable Nodules by Panel of Experts 1 st reading of CT cases Cases read independently & blinded by 3 expert radiologists Radiologist provided subject’s age, gender, and indication for exam Marked all findings deemed lung nodules Radiologist provided rating Intervention – Actionable, further workup advised Surveillance – Actionable, monitor with follow-up studies Probably Benign, calcified – no action required Probably Benign, non-calcified – no action required

8 Defining Actionable Nodules by Panel of Experts 2 nd pass Findings that lacked 100% consensus after 1 st pass were reviewed unblinded by all 3 radiologists 2/3 or 1/3 radiologists called the location a nodule are reevaluated Radiologists rated (or re-rated) the actionability of the nodule candidates Thresholds applied to all findings >4mm diameter > -100 HU maximum density Each lung quadrant categorized by the highest actionable finding within quadrant

9 Defining Actionable Nodules by Panel of Experts DispositionUnanimous Actionable 3/3 Majority Actionable 2/3 Minority Actionable 1/3 Sample Size experts per panel

10 Clinical Studies ROC Observer Study A z is test statistic Analysis of a 90 cases dataset (360 quadrants) Confidence intervals and significance testing ANOVA-after-jackknife Bootstrap analysis

11 Clinical Studies Analysis Flowchart Resampling Scheme Jackknife or Bootstrap Definition Of Nodules MRMC ROC Observer Study Pool of Cases Pool of Experts Pool of Readers Az Estimates

12 ANOVA-after-Jackknife Analysis Parametric analysis Leave-one case out (all 4 quadrants, quadrant-based analysis) Analysis assumes modality as a fixed effect and readers, cases and all interactions as random effects Example Set: [1 2 3], Partitions:[1 2], [1 3], [2 3]

13 Bootstrap Analysis Nonparametric analysis Randomly generated datasets, based on original data with replacement Example Set: [1 2 3], Partitions:[3 2 3], [3 1 2], [1 1 2], …

14 Clinical Studies Primary Analysis Resampling Scheme Jackknife or Bootstrap Definition Of Nodules MRMC ROC Observer Study Pool of Cases Pool of Experts Pool of Readers Az Estimates Fixed 3-member nodule definition panels (unanimous consensus) ANOVA-after-jackknife and Bootstrap analysis

15 Clinical Studies Primary Analysis Fixed 3-member nodule definition panels Variance Analysis Pre-CAD Az Post-CAD Az ΔAzp-value Lower C.L. Upper C.L. Jackknife Bootstrap <

16 Clinical Studies Primary Analysis Statistically significant improvement in A z pre- to post-CAD ΔA z ~0.025 ANOVA-after-jackknife and bootstrap analysis is consistent Analysis limited because it did not take into account any variation in the expert panel Variability of panel would add uncertainty to performance estimates How would performance change with a different panel makeup? Different number of panel members Different set of experts

17 Clinical Studies Secondary Analysis Resampling Scheme Bootstrap Definition Of Nodules MRMC ROC Observer Study Pool of Cases Pool of Experts Pool of Readers Az Estimates Random 3, 2, 1-member nodule definition panels (unanimous consensus) Only bootstrap analysis possible

18 Clinical Studies Secondary Analysis Bootstrap analysis Random 3-member nodule definition panels Random Panel Size Pre-CAD Az Post-CAD Az ΔAzp-value Lower C.L. Upper C.L. 3-members < members member <

19 Clinical Studies Secondary Analysis Sponsor's analysis takes into account random nature of expert panel for defining ‘actionable’ nodules Different number of panel members: 3, 2, 1-member panels Different panel makeup: bootstrap selection of panel All variations of panel makeup confirm a statistically significant improvement in A z from pre to post-CAD ΔA z ~0.02 Likely to be a more appropriate analysis for assessment of devices when only panel truth is available

20 CAD Standalone Performance Performance of the CAD algorithm alone Algorithm sensitivity and specificity (no reader involvement) Standalone CAD performance is important Radiologist needs this information to appropriately weight their confidence in the CAD markings Benchmark for future revisions to the algorithm What is an appropriate performance measure for this device?

21 CAD Standalone Performance Many of 142 findings (Fixed 3-member panel) did not meet criteria as a solid discrete, spherical density Second panel reevaluated nodules for appearance 5 independent radiologists 2 Categories Classic nodule: discrete solid, spherical or ovoid Non-classic: Not discrete Hyperdense Irregularly shaped Normal structure Not a nodule

22 CAD Standalone Performance No. Panelists defining as classic No. of Findings CAD TPF (%) CAD False Marker Rate TP Median Diameter (mm) <3/ ~3 per-case / / / All <3/ ~3 per-case ≥3/

23 CAD Standalone Performance Large variation in performance of the CAD based on physicians assessment of nodule appearance as “classic”

24 Summary A z appropriate test statistic for clinical analysis No substantial crossing of pre/post-CAD ROC curves Primary Analysis Nodule definition panel Fixed 3-member expert panel Shows statistically significant A z improvement in detection with CAD ANOVA-after-jackknife and bootstrap are comparable

25 Summary Secondary Analysis Nodule Definition panel Varied number of panel members Varied the panel makeup (bootstrap selection of panel members) Confirmed statistically significant A z improvement in detection with CAD Standalone performance Large variation in CAD performance based on reassessment of nodule appearance Necessary for appropriate utilization of the device by clinicians in the field and assessment of future algorithm revisions