Knowledge Systems Lab JN 9/13/2015 An Advanced User Interface for Pattern Recognition in Medical Imagery: Interactive Learning, Contextual Zooming, and Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions

Knowledge Systems Lab JN 9/13/2015 Introduction Medical imagery… Consists of millions of images produced annually which doctors must gather and analyze Entails several modalities for each patient, such as MRI, CT, and PET Refine techniques for facilitating comprehension of this data

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions

Knowledge Systems Lab JN 9/13/2015 Techniques Common techniques for facilitating data comprehension: –Segmentation – Labeling of images –Magnification – Precision viewing –Exploration – Interacting intuitively with complex, 3D data

Knowledge Systems Lab JN 9/13/2015 Why Segmentation? Doctors and radiologists: –Spend several hours daily analyzing patient images (ie. MRI scans of the brain) –Search for patterns in images that are standard and well-known to doctors Why not have the doctor teach the computer to find these patterns in the images?

Knowledge Systems Lab JN 9/13/2015 Why Magnification? Doctors and radiologists: –Must be able to precisely view and select regions/pixels of the image to train the computer –Can easily lose where they are looking in the image when using magnification Why not use visualization techniques to preserve context while allowing precise selections?

Knowledge Systems Lab JN 9/13/2015 Why Exploration? Doctors and radiologists: –Need to intuitively interact with the system to maximize task performance –Need to perform this interaction while being unencumbered Why not use vision-based recognition to allow interaction with the data?

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions

Knowledge Systems Lab JN 9/13/2015 Problems & Solutions Problem #1: Segmentation Solution #1: Interactive Learning Problem #2: Magnification Solution #2: Contextual Zoom Problem #3: Exploration Solution #3: Gesture Recognition

Knowledge Systems Lab JN 9/13/2015 Platform Med-LIFE: –“L”earning of MRI image patterns –“I”mage “F”usion of multiple MRI images –“E”xploration of the fusion and learning results in an intuitive 3D environment Images used from “The Whole Brain Atlas” –

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions

Knowledge Systems Lab JN 9/13/2015 Simplified Fuzzy ARTMAP Simplified Fuzzy ARTMAP (SFAM) –An AI neural network (NN) system –Capable of online, incremental learning –Takes seconds for tasks that take backpropagation NNs days or weeks to perform

Knowledge Systems Lab JN 9/13/2015 Vector-based Learning Two “vectors” are sent to this system for learning: –Input feature vector provides the data from which SFAM can learn –‘Teacher’ signal indicates whether that vector is an example or counterexample

Knowledge Systems Lab JN 9/13/2015 Feature Vector Pixel values from images (16 for each slice)

Knowledge Systems Lab JN 9/13/2015 Learning Visualization Vector-based graphic visualization of learning Array of Pixel Values x y Category members Category member Category members

Knowledge Systems Lab JN 9/13/2015 T2 Learning Associations Full ResultsDetailed Results

Knowledge Systems Lab JN 9/13/2015 Varying Vigilance Only one tunable parameter – vigilance –Vigilance can be set from 0 to 1 and corresponds to the generality by which things are classified (ie. vig=0.3=>human, vig=0.6=>male, 0.9=>Joshua New)

Knowledge Systems Lab JN 9/13/2015 Input Order Dependence SFAM is sensitive to the order of the inputs x y Category members Category member Category members Vector 3 Vector 1 Vector 2

Knowledge Systems Lab JN 9/13/2015 Heterogeneous Network Voting scheme of 5 Heterogeneous SFAM networks to overcome vigilance and input order dependence –3 networks: random input order, set vigilance –2 networks: 3 rd network order, vigilance ± 10%

Knowledge Systems Lab JN 9/13/2015 Network Segmentation Results

Knowledge Systems Lab JN 9/13/2015 Segmentation Results Threshold results Overlay results Trans-slice results

Knowledge Systems Lab JN 9/13/2015 Segmentation Screenshot

Knowledge Systems Lab JN 9/13/2015 System Demonstration Interactive Learning

Knowledge Systems Lab JN 9/13/2015 Segmentation Solution Doctors and radiologists: –Spend several hours daily analyzing patient images (ie. MRI scans of the brain) –Search for patterns in images that are standard and well-known to doctors Solution: –Doctors and radiologists can teach the computer to recognize abnormal brain tissue –They can refine the learning systems results interactively

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions

Knowledge Systems Lab JN 9/13/2015 Zooming Approaches Inset Overlay Chip Window

Knowledge Systems Lab JN 9/13/2015 Research & Business Carpendale PhD Thesis –Elastic Presentation Space – rubber sheet images via mathematical constructs IDELIX ( –Pliable Display Technology – software development kit (SDK) product –Boeing: 20% increase in productivity

Knowledge Systems Lab JN 9/13/2015 Zoom Visualization Wireframe View Contextual Zoom

Knowledge Systems Lab JN 9/13/2015 System Demonstration Contextual Zoom

Knowledge Systems Lab JN 9/13/2015 System Comparison Previous System Zoom Overlay Contextual Zoom

Knowledge Systems Lab JN 9/13/2015 Magnification Solution Doctors and radiologists: –Must be able to precisely view and select regions/pixels of the image to train the computer –Can easily lose where they are looking in the image when using magnification Solution –They can precisely select targets/non-targets –They can zoom for precision while maintaining context of the entire image –The interface facilitates task performance through interactive display of segmentation results

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions

Knowledge Systems Lab JN 9/13/2015 Motivation Gesturing is a natural form of communication: –Gesture naturally while talking –Babies gesture before they can talk Interaction problems with the mouse: –Have to locate cursor –Hard for some to control (Parkinsons or people on a train) –Limited forms of input from the mouse

Knowledge Systems Lab JN 9/13/2015 Motivation Problems with the Virtual Reality Glove as a gesture recognition device: –Reliability –Always connected –Encumbrance

Knowledge Systems Lab JN 9/13/2015 System Diagram Standard Web Camera Rendering User Interface Display Hand Movement User Gesture Recognition System Image Capture Update Object Image Input

Knowledge Systems Lab JN 9/13/2015 System Performance System: OpenCV and IPL libraries (from Intel) Input: 640x480 video image Hand calibration measure Output: Rough estimate of centroid Refined estimate of centroid Number of fingers being held up Manipulation of 3D skull in QT interface in response to gesturing

Knowledge Systems Lab JN 9/13/2015 Calibration Measure Max hand size in x and y orientation (number of pixels in 640x480 image)

Knowledge Systems Lab JN 9/13/2015 Saturation Extraction Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline a) 0 th moment of an image: b) 1 st moment for x and y of an image, respectively: c) 2 nd moment for x and y of an image, respectively: d) Orientation of image major axis:

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) Total fingers is number of fingers minus 1 for the hand itself

Knowledge Systems Lab JN 9/13/2015 System Setup System Configuration System GUI Layout

Knowledge Systems Lab JN 9/13/2015 Interaction Mapping Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Demo

Knowledge Systems Lab JN 9/13/2015 Exploration Solution Doctors and radiologists: –Need to intuitively interact with the system to maximize task performance –Need to perform this interaction while being unencumbered Solution –Can use intuitive gesturing to interact with complex, 3D data –Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation

Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions and Future Work

Knowledge Systems Lab JN 9/13/2015 Interactive Learning Users can teach the computer to recognize abnormal brain tissue They can refine the learning systems results interactively They can save/load agents for background diagnosis on a database of medical images or to allow expert analysis in the absence of a well-paid expert

Knowledge Systems Lab JN 9/13/2015 Contextual Zoom They can zoom for precisely viewing and selecting targets/non-targets while maintaining context of the entire image The interface facilitates task performance through interactive and customizable display of segmentation results This system can be used with any 2D images and even with 3D datasets with some minor alterations

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Can use intuitive gesturing to interact with complex, 3D data Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation Easily replicated and distributable Mapping gestures to interaction is an independent stage

Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Dynamic Gesture Recognition Other interface applications include: graspable interfaces, 3D avatar / MoCap, multi-object manipulation in virtual environments, and augmented reality

Knowledge Systems Lab JN 9/13/2015 Platform Med-LIFE integration effort –Gesture Recognition has already been integrated into Med-LIFE’s Exploration tab –Contextual Zoom and Interactive learning have been combined, but not yet integrated into Med-LIFE’s learning tab Med-LIFE will function as a single application for medical image analysis