1 Preclinical-Bench Testing II Using Human Observers to Objectively Measure and Evaluate Imaging Performance of Digital WSI Systems Max Robinowitz, MD FDA Center for Devices and Radiological Health October 22, 2009

2 Object Related Diagnosis Please look at this picture  Identify all that you see What do you think they represent? What is your diagnosis?

3 From Kayser K et al Diagnostic Pathology 2006

4 How Can We Objectively Evaluate If Image Quality Presented to Pathologists is Fit-for-Purpose? How can we quantify the contribution of the image quality to a final tissue-based diagnosis? How can we quantify contribution added from the eye & brain of the pathologist? How can we quantify the ability of pathologists to search & detect different objects rendered by OM & WSI? How can we identify the contributions of the ergonomics and human factors of the different work stations?

5 Role of Preclinical-Bench Testing  Physical & biological phantoms may provide objective data for prediction of diagnostic performance of pathologists with the optical microscope & different WSI systems in the real world in a manner similar to x-ray phantoms  The general application of WSI to routine surgical pathology practice potentially involves all of clinical medicine  All clinically essential areas within the whole spectrum of surgical pathology may have to be tested

6 Human Observer Studies with Biologic & Human Histologic Specimens Phantoms Minimize the subjectivity of study pathologists by emphasizing each pathologist’s assessments of image quality and information content before the overlay of high-level intellectual diagnostic interpretation Targets can be constructed from biological materials of known size, number, fine morphological features & spatial distributions to quantify image quality---objectively

7 Controlling Subjectivity in Performance of Real World Tasks of Pathologists Pre-test tutorial to standardize criteria for the study pathologists. Pre-specify the features to observe, enumerate, using criteria that minimize requirements for higher-level interpretation by pathologists Use biological objects in study as the equivalent of a Snell eye chart to test eye glasses. Task: Recognition of shapes & sizes in images using description rather than diagnostic terms --- Have pathologist quantify the detailed features of the image

8 Diatoms = Algae with Known Size & 3D Textural Features Requiring 20 & 40X Objectives to View Details in Image 97 Diatoms ranging from about 2 to 100 micrometers in diameter. Note: fine 3D internal features that can be objectively measured & enumerated From Kemp K diatoms.uk.com

9 Objective “Open Book Exam” Study with Human Specimens Controlled, non-subjective evaluation of image quality rendered by OM versus the image quality rendered by WSI systems Reproducible, optimal preparations of normal and abnormal human specimens on TMA cores and whole slide

10 Critical objects & Features to evaluate Accuracy & precision of pathologists for identification of objects and features in targets Clinically significant objects, e.g., cancer microinvasion, nuclear chromatin structure, nucleoli, cytoplasmic vacuoles, inclusions, granules & striations, architectural growth patterns, etc.

11 Critical Variables to evaluate Effects of tissue preparation, slide thickness, different stains, different adjustments & settings of OM and WSI system Variability in planes of focus selected by WSI on different runs Variability in pathologists with different types of image features

12 Tasks for Study Pathologists That Can Use Cores or Whole Slide Specimens Task 1 Search & find/detect objects and textures to test image quality, focus and fidelity, dynamic range, color fidelity across whole slide Task 2 Enumerate number & measure objects Task 3 Classify detected objects

13 Task 4 for Study Pathologists is Limited to Whole Slide Specimens Task 4 Diagnose based on the enumeration and cumulative semi- quantitative scores from the objects and features from larger areas that exceed the size of TMA cores (mimics intended use) –Quantify with Scoring system that is a composite of multiple simple features— some require the optical resolution at 40X & z-axis focusing

14 Tissue Microarrays with Cores of Normal & Abnormal Features Selected by Experts TMA cores from ROIs of multiple specimens, in precise order of a grid on single slide, allows efficient viewing by study pathologists for only some of the fit-for-purpose testing TMA Paraffin block Individual core 9X9 grid

15 Tissue Microarray (TMA) Tissue Cores Diameter Sizes and Numbers/Slide Core sizes are 0.6, 1.0, 1.5 and 2.0 mm most common. Most popular is 1.0 mm. 0.6 mm = standard 40X objective field of view 1.0 mm = standard 20X objective field of view Larger cores can be made but even 1.5 and 2.0 are very hard to work with, but are good for normal tissues where you want to show a number of features. 0.6 = 500 cores/slide; 1.0 = 200 cores/slide 1.5 = 75 cores/slide; 2.0 = 35 cores/slide

16 Tissue Microarray (TMA) Tissue Cores Applications and Limitations Tissue microarrays are suitable to test pathologists’ ability for recognition of cellular objects (features) and some limited spatial tissue growth patterns TMAs are NOT suitable for testing the tasks of search & find/detection of ROI within the multiple different diagnostic areas on a whole slide specimen as required in the real-world of surgical pathology

17 Bench-Testing for Realistic Diagnostic Tasks: Combinations of Multiple Features 1.Pre-test tutorial with schematic with the features to be identified and compared with slide 2.Slide with multiple conditions to search, detect & describe according to their location on the whole slide and its image on WSI, e.g. upper right quadrant, lower right quadrant 3.Match & Describe each abnormality according to the schematics and pre-test “standardized terminology” III IV DCIS

18 Nottingham BrCa (Ordinal) Score Controlled Task for OM & WSI Tubule Formation –Majority of tumor (>75%) 1 –Moderate degree (10-75%) 2 –Little or none (<10%) 3 Mitotic Count –0-9 mitoses/10 hpf 1 – mitoses/10 hpf 2 –20 or > mitoses/10 hpf 3 Nuclear Pleomorphism –Small regular uniform cells 1 –Moderate nuclear size and variation 2 –Marked nuclear variation 3 Combined Histologic Grade Low grade (I) 3-5 Intermediate grade (II) 6-7 High grade (III) 8-9 This is just one of many clinically relevant examples

19 Heirarchy for Grading Dx Errors 10 levels of increasing complexity of diagnostic interpretation & final diagnosis nomenclature Mireskandari M, Kayser G, Hufnagl P, Schrader T, Kayser K Diagnostic Pathology 2004 Image Based High-Order Intellectual Overlays

20 FDA Desires Panel Input What roles can bench-testing of OM and WSI serve for guiding the designs for clinical studies in real world surgical pathology practices? What safety and effectiveness concerns can be resolved with pre-clinical studies using selected histopathology specimens, such as TMAs and whole slides? What are the limitations? What concerns require clinical trials with representatives of the intended pathologists who must examine whole slides in the real world setting of surgical pathology practice?