1. Updates to and Cross-Reader Reliability Validation of the VASARI Brain Tumor Feature Set for Distributed Scoring of Low-Grade Brain Tumor Cases in Radiogenomic Research The TCGA Low-Grade Glioma Group, including, but not limited to Chen JY, Gutman DA, Yeom KW, Mukherjee S, Iv M, Wintermark M, Wolansky L, Griffith B, Holder CA, Hwang SN, Colen R, Kirby J, Jaffe CC, Fevrier-Sullivan B, and Flanders AE eEdE-205 Control: 1771

2. Disclosures None

3. VASARI Features Introduction VASARI terminology – Categorical descriptors of MR imaging features of human gliomas – Based on previous REMBRANDT project REpository for Molecular BRain Neoplasia DaTa – Bioinformatics framework leveraging data warehousing for clinical and functional genomic data from glioma clinical trials

4. Sept 2013 VASARI Feature Set Feature number/name Description Options F0 Image QA Quality assurance pre-check. Inventory of required image series including FLAIR/T2 and pre/post contrast T1WI.  Post biopsy (requires adjudication)  Post-op study (disqualified)  No contrast injected (disqualified)  No T2/FLAIR images (disqualified) F1 Tumor Center Location of lesion geographic center; the largest component of the tumor (either CET or nCET). (multiple selections acceptable – choose up to two)  Frontal lobe  Temporal lobe  Parietal lobe  Occipital lobe  Insula  Brainstem  Cerebellum  Lentiform nucleus  Caudate nucleus  Thalamus  Corpus callosum F34 Tumor OriginPropose where you think the tumor originated from using the following grades. Periventricular (most likely arising from periventricular/subventricular zone); Gyral (most likely arising from the cortex or adjacent gyral subcortical white matter zone); Deep white matter (most likely arising deep to the U fiber region but not in the periventricular zone). Use not applicable if none of these feature criteria apply. (multiple selections acceptable)  Periventricular  Gyral  Deep white matter  Other

5. Sept 2013 VASARI Feature Set F2 Side of Tumor Center Side of lesion center irrespective of whether the lesion crosses into the contralateral hemisphere.  Right  Center/Bilateral  Left F3 Eloquent Brain Does any component of the tumor (CET or nCET) involve eloquent cortex or the immediately adjacent subcortical white matter of eloquent cortex (motor,language, vision)? (multiple selections acceptable)  No Eloquent Brain  Speech motor  Speech receptive  Motor  Vision F22 nCET tumor Crosses Midline: nCET tumor crosses midline is defined by any nCET tumor that extends into the contralateral hemisphere through white matter commissures usually expected at the midline (exclusive of herniated ipsilateral tissue).  No nCET  No  Yes F13 Definition of the non-enhancing margin (e.g. Grade III) If most of the outside non-enhancing (nCET) margin of the tumor is well-defined (i.e. sharply marginated) and smooth (geographic), versus if the margin is poorly-defined (fluffy or indistinct).  Completely well-defined (100%)  Mostly well-defined (> two thirds)  Mixed (@ 50-50)  Mostly poorly-defined (> two thirds)  Completely ill-defined (100%)  No nCET

6. Sept 2013 VASARI Feature Set F6 Proportion nCETWhen scanning through the entire tumor volume, what proportion of the entire tumor is estimated to represent nonenhancing tumor or nCET (not edema)? Non-enhancing tumor (nCET) is defined as regions of T2W intermediate hyperintensity (less than the intensity of cerebrospinal fluid or vasogenic edema, with corresponding T1W hypointensity) that are associated with mass effect and architectural distortion, including blurring of the gray-white interface. (This may be difficult to discern from vasogenic edema – if uncertain error conservatively). (Assuming that the the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component.)  > Two Thirds  Between Two Thirds and One Third  Less than One Third  Minimal  None

7. Sept 2013 VASARI Feature Set F14 Proportion of Edema Visually, when scanning through the entire tumor volume, what proportion of the entire abnormality is estimated to represent vasogenic edema? Edema should be greater in signal than nCET and somewhat lower in signal than CSF on T2W or T2W FLAIR. Pseudopods are characteristic of edema extending up to the subcortical white matter but not infiltrating gray matter/cortex. (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component).  > Two Thirds  Between Two Thirds and One Third  Less than One Third  Minimal  None

8. Sept 2013 VASARI Feature Set F6 Proportion nCET When scanning through the entire tumor volume, what proportion of the entire tumor is estimated to represent nonenhancing tumor or nCET (not edema)? Non-enhancing tumor (nCET) is defined as regions of T2W intermediate hyperintensity (less than the intensity of cerebrospinal fluid or vasogenic edema, with corresponding T1W hypointensity) that are associated with mass effect and architectural distortion, including blurring of the gray-white interface. (This may be difficult to discern from vasogenic edema – if uncertain error conservatively). (Assuming that the the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component.)  > Two Thirds  Between Two Thirds and One Third  Less than One Third  Minimal  None F14 Proportion of Edema Visually, when scanning through the entire tumor volume, what proportion of the entire abnormality is estimated to represent vasogenic edema? Edema should be greater in signal than nCET and somewhat lower in signal than CSF on T2W or T2W FLAIR. Pseudopods are characteristic of edema extending up to the subcortical white matter but not infiltrating gray matter/cortex. (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component).  > Two Thirds  Between Two Thirds and One Third  Less than One Third  Minimal  None

9. Sept 2013 VASARI Feature Set F5 Proportion Enhancing: When scanning through the entire tumor volume, what proportion of the entire tumor is estimated to be enhancing. (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) an edema component.)  > Two Thirds  Between Two Thirds and One Third  Less than One Third  Minimal  None F7 Proportion Necrosis Visually, when scanning through the entire tumor volume, what proportion of the tumor is estimated to represent necrosis. Necrosis is defined as a region within the tumor that does not enhance, is hyperintense on T2W and proton density images, is hypointense on T1W images, and has an irregular border). (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component.)  > Two Thirds  Between Two Thirds and One Third  Less than One Third  Minimal  None F4 Enhancement Quality: Qualitative degree of contrast enhancement is defined as having all or portions of the tumor that demonstrate significantly higher signal on the postcontrast T1W images compared to precontrast T1W images. Mild = when barely discernible but unequivocal degree of enhancement is present relative to pre-contrast images. Marked = obvious tissue enhancement. (If it does not appear that contrast was administered, select No Contrast Injected.)  No Contrast Enhancement  Mild  Marked

10. Sept 2013 VASARI Feature Set F11 Thickness of enhancing margin If most of the enhancing rim is thin, regular, and measures < 3mm in thickness and has homogenous enhancement the grade is minimal. If most of the rim demonstrates nodular and/or thick enhancement measuring 3mm or more, the grade is thick/nodular. If there is only solid enhancement and no rim, the grade is solid. (If it does not appear that contrast was administered, select No Contrast Injected.)  No Contrast Enhancement  Minimal  Thick/nodular (=> 3mm)  Solid F12 Definition of the Enhancing margin Assess if most of the outside margin of the enhancement is well defined (i.e. sharply marginated) or poorly-defined (fluffy or indistinct). Are you able to easily trace the margin of enhancement?  No Contrast Enhancement  Completely well-defined (100%)  Mostly well-defined (> two thirds)  Mixed (@ 50-50)  Mostly poorly-defined (> two thirds)  Completely ill-defined (100%)

11. Sept 2013 VASARI Feature Set F23 Enhancing tumor Crosses Midline: Enhancing tissue crosses midline is defined by any CET that extends into the contralateral hemisphere through white matter commissures usually expected at the midline (exclusive of herniated ipsilateral tissue).  No Contrast Enhancement  No  Yes F31 HeterogeneityWhen assessing the complexity of the internal architecture of the tumor on FLAIR or T2WI overall grade the uniformity of the tumor matrix (exclusive of what appears to clearly be edema). Select completely homogeneous when the tumor matrix is completely uniform in consistency; mostly homogeneous when more than 2/3 of the tumor matrix is uniform; mixed when about half of the tumor is homogeneous; mostly heterogeneous when more than half of the tumor volume is non-uniform; completely heterogeneous when all of the tumor matrix is non-uniform. FLAIR should be used primarily. T2WI can be used when FLAIR images are not included.  Completely homogeneous (100%)  Mostly homogeneous (> two thirds)  Mixed (@ 50-50)  Mostly heterogeneous (> two thirds)  Completely heterogeneous (100%)  Indeterminate

12. Sept 2013 VASARI Feature Set F32 Shape Shape is defined as the overall contour of the abnormal tissue that you would characterize as a tumor boundary. Which descriptor best defines the shape of the entire mass: round, ovoid, lobulated or irregular. Consider the overall boundaries of the NCET and CET exclusive of any edema.  Round/circular/spherical  Ovoid  Lobulated  Irregular  None apply F8 Cyst(s)Cysts are well defined, rounded,often eccentric regions of very bright T2W signal and low T1W signal essentially matching CSF signal intensity, with very thin, regular, smooth, nonenhancing or regularly enhancing walls, possibly with thin, regular, internal septations. Differentiate from a necrotic enhancing tumor cavity with thick irregular walls and complex internal fluid.  Absent  Present

13. Sept 2013 VASARI Feature Set F9 Multifocal or Multicentric Multifocal is defined as having at least one region of tumor, either enhancing or nonenhancing, which is not contiguous with the dominant lesion and is outside the region of signal abnormality (edema) surrounding the dominant mass. This can be defined as those resulting from dissemination or growth by an established route, spread via commissural or other pathways, or via CSF channels or local metastases, whereas Multicentric are widely separated lesions in different lobes or different hemispheres that cannot be attributed to one of the previously mentioned pathways. Gliomatosis refers to generalized neoplastic transformation of the white matter of most of a hemisphere.  Focal  Multifocal or Multicentric  Gliomatosis Cerebri F10 T1/FLAIR RATIO T1/FLAIR ratio is a gross comparison in the overall lesion size between pre-contrast T1 and FLAIR (in the same plane). Select T1~FLAIR when pre-contrast T1 abnormality (exclusive of signal intensity) approximates size of FLAIR abnormality; Select T1<FLAIR when the size of T1 abnormality is moderately smaller than the surrounding FLAIR envelope; or select T1<<FLAIR when the size of the pre-contrast T1 abnormality is much smaller than size of FLAIR abnormality. (If no FLAIR images were provided select No FLAIR images).  T1~FLAIR  T1<FLAIR  T1<<FLAIR

14. Sept 2013 VASARI Feature Set F10 T1/FLAIR RATIO T1/FLAIR ratio is a gross comparison in the overall lesion size between pre-contrast T1 and FLAIR (in the same plane). Select T1~FLAIR when pre-contrast T1 abnormality (exclusive of signal intensity) approximates size of FLAIR abnormality; Select T1<FLAIR when the size of T1 abnormality is moderately smaller than the surrounding FLAIR envelope; or select T1<<FLAIR when the size of the pre-contrast T1 abnormality is much smaller than size of FLAIR abnormality. (If no FLAIR images were provided select No FLAIR images).  T1~FLAIR  T1<FLAIR  T1<<FLAIR F16 Hemorrhage: Intrinsic hemorrhage anywhere in the tumor matrix. Any intrinsic foci of low signal on T2WI (or gradient echo) or high signal on T1WI. Proportion is not a discriminating factor. Select cannot determine if findings are indistinct or may actually represent mineral instead of hemorrhage.  No  Yes  Indeterminate F17 Diffusion:Proportion of CET and nCET demonstrating ADC below or same as the ADC of normal-appearing brain. The remainder of the abnormality is assumed to demonstrate increased ADC relative to normal brain. (Based on ADC map only). Select indeterminate when findings are equivocal, (Select no ADC images if ADC images were not provided)  > 2/3  1/3 – 2/3  < 1/3  Minimal (< 5%)  None  Indeterminate  No ADC images

15. Sept 2013 VASARI Feature Set F18 Leptomeningeal invasion: Leptomeningeal invasion is defined by enhancement of the overlying pia/arachnoid in continuity with enhancing or non- enhancing tumor  Absent  Present F19 Ependymal invasion: Ependymal invasion is defined by tumor abutting any adjacent ependymal surface in continuity with enhancing or non- enhancing tumor matrix.  Absent  Present F20 Cortical involvement Cortical involvement is defined by non-enhancing or enhancing tumor that extends to the cortical mantle, or if the cortex is no longer distinguishable relative to tumor.  Absent  Present F21 Deep WM invasion Deep white matter invasion is defined by enhancing or nCET tumor extending into the internal capsule, corpus callosum or brainstem. (multiple choices allowed)  None  Corpus Callosum  Internal Capsule  Brainstem

16. Sept 2013 VASARI Feature Set F24 Satellites: A satellite lesion is defined by one or more areas of nCET or CET within the region of signal abnormality surrounding the dominant lesion but not contiguous in any part with the major tumor mass. This is in distinction from a multifocal lesion.  Absent  Present F25 Calvarial remodeling: Calvarial remodeling is defined as visible erosion/remodeling of inner table of skull (possibly a secondary sign of slow growth).  Absent  Present F29 & F30 Lesion SizeLesion size is defined as the largest perpendicular (x-y) cross- sectional diameter of entire T2 signal abnormality (longest dimension x perpendicular dimension) measured the single axial image that reveals the largest cross-sectional area of the lesion. (Measurement should incorporate all cardinal imaging features of CET, nCET, necrosis and edema.) (measured with ruler tool on workstation)

17. TCGA Glioma Subgroup Is a geographically distributed collaborative radiogenomic research group comprising multiple individuals from multiple institutions across the US The Cancer Imaging Archives data samples including imaging and clinical information submitted from participating institutions.

18. Reason to Update VASARI featureset was used for high-grade glioma work For the TCGA group each case was scored by multiple readers; not all readers scored all cases Interreader reliability analysis was performed for the case-reading rounds for the high-grade tumors

19. High-Grade Interreader Reliability Analysis Performed by group member Erich Huang Categorical features were analyzed with Krippendorff’s α – −1 ≤ α ≤ 1, with: α = 1 → perfect agreement α = 0 → any agreement solely attributed to chance α < 0 → disagreements are systematic – Extension of Fleiss’ κ to accommodate the study design where the imaging data for each patient are reviewed by a subset of a pool of readers, but the specific readers that review the imaging data differ for each patient – Can handle binary and multinomial codings, as well as ordinal

20. Example Features with High Agreement Were Not Changed Darker bars = more readers with same score; courtesy Erich Huang

21. Example Features with High Agreement Were Not Changed Darker bars = more readers with same score; courtesy Erich Huang

22. Sometimes Imbalanced Scores Can Lead to Low- Appearing α Despite Visually Apparent Agreement Darker bars = more readers with same score; courtesy Erich Huang

23. Reliability of Quantitative Features Quantitative features: used a reliability ratio ρ to assess the inter- reader reliability ρ = Between-Patient Measurement Variability / Total Measurement Variability Total Measurement Variability = Between-Patient Measurement Variability + Within-Patient Measurement Variability. – 0 ≤ ρ ≤ 1, with: ρ = 1 → no within-patient measurement variability ρ = 0 → all measurement variability attributed to within-patient measurement variability Analyzed square root of the length

24. Good Reliability of Major Axis Length

25. Good Reliability of Minor Axis Length

26. Updating VASARI for Low-Grade Gliomas Features with poor agreement evaluated for – Clarity of feature definition – Ease of using feature definition – Clarity of feature name – Scientific evidence for any biological relevance

27. Updating VASARI for Low-Grade Gliomas Domain experts within the TCGA Glioma group debated and discussed poorly performing features – 11 features subsequently modified or renamed 4 were essentially deleted and replaced – 1 feature was deleted – Some new features were created The complete, fully updated feature-set is to be made available in a subsequent TCGA publication

28. Iterative Testing and Updates of VASARI Features Testing of new/modified features was performed for validation of interreader reliability Volunteer readers for the TCIA low-grade glioma cases were all trained in multiple sessions by a single trainer on the updated VASARI feature set Each reader was then assigned a set of 5 test cases for training purposes

29. Iterative Testing and Updates of VASARI Features First round cases were scored using Annotation and Image Markup (AIM) implementation through ClearCanvas on Amazon’s cloud services Reader scores were compiled and visually compared for consistency – Categorical scores greater than 1 adjacent selection in distance were considered discrepant (e.g. mild versus severe) – Scores in adjacent categories were considered reasonable (e.g. minimal vs mild)

30. First Round Testing Areas of disagreement greater than 1 category distance apart (yellow) were highlighted Many areas of reader discrepancy discovered Images and VASARI definitions were analyzed to understand differences in scoring

31. First Round Scoring Differences Multiple etiologies for scoring differences – ADC maps sometimes difficult to find/hidden in various series on different patients – Different interpretation of feature definitions (definition clarification and training needed) – Interpretation based on feature name rather than definition (feature name clarification and training needed) – Classic interpretative challenges (e.g. edema/non- contrast-enhancing tumor)

32. Post-First Round VASARI Updates Multiple teleconferences for consensus among domain experts within the group for additional changes to VASARI feature set to account for the scoring differences Number of features added: 0 Number of features modified/clarified: 9 Number of features deleted: 2

33. Second Round Testing Though reader agreement was not perfect, it was greatly improved from the first round Majority of features had few discrepancies greater than 1 category apart

34. Second Round Scoring Differences Primary differences from – Interpretive error – Reasonable differences in opinion – User error (mistakenly clicking an incorrect choice)

35. Freezing the Feature Set As second round discrepancies appeared reasonable and marginal improvement in scoring reliability from additional feature modification was probably low, the feature set was frozen for use in the next TCGA project: low- grade gliomas

36. Conclusion VASARI feature set is a set of controlled categorical descriptors for MR imaging in human gliomas It has been tested and modified iteratively for best cross-reader reliability in distributed scoring of imaging datasets Its biological relevance in radiogenomic research is currently being tested against the TCGA dataset

37. Part of TCGA Glioma Phenotype Research Group Hwang, S.N. 1 Hwang, S.N. 1 Holder, C.A. 1 Holder, C.A. 1 Clifford, R.J. 2 Clifford, R.J. 2 Huang, E. 2 Huang, E. 2 Hammoud, D. 3 Hammoud, D. 3 Raghavan, P. 4 Raghavan, P. 4 Wintermark, M. 9 Wintermark, M. 9 Gutman, D.A 1 Gutman, D.A 1 Moreno, C. 1 Moreno, C. 1 Cooper, L. 1 Cooper, L. 1 Freymann, J. 5 Freymann, J. 5 Kirby, J. 5 Kirby, J. 5 Krishnan, A. 1 Krishnan, A. 1 Dehkharghani, S. 1 Dehkharghani, S. 1 Jaffe, C.C. 6 Jaffe, C.C. 6 Saltz, J.H. 1 Saltz, J.H. 1 Brat, D.J. 1 Brat, D.J. 1 Colen, R.R. 8 Colen, R.R. 8 Rubin, D.L. 9 Rubin, D.L. 9 Jain, R 10 Jain, R 10 Flanders, A.E. 7 Flanders, A.E. 7 Erickson, B.J. 11 Erickson, B.J. 11 Chen, J.Y. 12 Chen, J.Y. 12 Rao, A. 8 Rao, A. 8 Wolansky, L. 13 Wolansky, L. 13 1 Emory University 2 National Cancer Institute 3 National Institute of Health 4 University of Virginia 5 SAIC-Frederick, Inc. 6 Boston University 7 Thomas Jefferson University 8 MD Anderson Cancer Center 9 Stanford University 10 Henry Ford Hospital 11 Mayo Clinic 12 UCSD 13 Case Western Reserve U. Thanks and apologies to anyone who should be on this slide and isn’t