Download presentation
Presentation is loading. Please wait.
Published byGillian Pitts Modified over 9 years ago
1
Knowledge Systems Lab JN 9/13/2015 An Advanced User Interface for Pattern Recognition in Medical Imagery: Interactive Learning, Contextual Zooming, and Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University
2
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions
3
Knowledge Systems Lab JN 9/13/2015 Introduction Medical imagery… Consists of millions of images produced annually which doctors must gather and analyze Entails several modalities for each patient, such as MRI, CT, and PET Refine techniques for facilitating comprehension of this data
4
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions
5
Knowledge Systems Lab JN 9/13/2015 Techniques Common techniques for facilitating data comprehension: –Segmentation – Labeling of images –Magnification – Precision viewing –Exploration – Interacting intuitively with complex, 3D data
6
Knowledge Systems Lab JN 9/13/2015 Why Segmentation? Doctors and radiologists: –Spend several hours daily analyzing patient images (ie. MRI scans of the brain) –Search for patterns in images that are standard and well-known to doctors Why not have the doctor teach the computer to find these patterns in the images?
7
Knowledge Systems Lab JN 9/13/2015 Why Magnification? Doctors and radiologists: –Must be able to precisely view and select regions/pixels of the image to train the computer –Can easily lose where they are looking in the image when using magnification Why not use visualization techniques to preserve context while allowing precise selections?
8
Knowledge Systems Lab JN 9/13/2015 Why Exploration? Doctors and radiologists: –Need to intuitively interact with the system to maximize task performance –Need to perform this interaction while being unencumbered Why not use vision-based recognition to allow interaction with the data?
9
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions
10
Knowledge Systems Lab JN 9/13/2015 Problems & Solutions Problem #1: Segmentation Solution #1: Interactive Learning Problem #2: Magnification Solution #2: Contextual Zoom Problem #3: Exploration Solution #3: Gesture Recognition
11
Knowledge Systems Lab JN 9/13/2015 Platform Med-LIFE: –“L”earning of MRI image patterns –“I”mage “F”usion of multiple MRI images –“E”xploration of the fusion and learning results in an intuitive 3D environment Images used from “The Whole Brain Atlas” –http://www.med.harvard.edu/AANLIB/home.html
12
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions
13
Knowledge Systems Lab JN 9/13/2015 Simplified Fuzzy ARTMAP Simplified Fuzzy ARTMAP (SFAM) –An AI neural network (NN) system –Capable of online, incremental learning –Takes seconds for tasks that take backpropagation NNs days or weeks to perform
14
Knowledge Systems Lab JN 9/13/2015 Vector-based Learning Two “vectors” are sent to this system for learning: –Input feature vector provides the data from which SFAM can learn –‘Teacher’ signal indicates whether that vector is an example or counterexample
15
Knowledge Systems Lab JN 9/13/2015 Feature Vector Pixel values from images (16 for each slice)
16
Knowledge Systems Lab JN 9/13/2015 Learning Visualization Vector-based graphic visualization of learning Array of Pixel Values x y Category 1 - 2 members Category 2 - 1 member Category 4 - 3 members 0.30 0.45
17
Knowledge Systems Lab JN 9/13/2015 T2 Learning Associations Full ResultsDetailed Results
18
Knowledge Systems Lab JN 9/13/2015 Varying Vigilance Only one tunable parameter – vigilance –Vigilance can be set from 0 to 1 and corresponds to the generality by which things are classified (ie. vig=0.3=>human, vig=0.6=>male, 0.9=>Joshua New) 0.6750.750.825
19
Knowledge Systems Lab JN 9/13/2015 Input Order Dependence SFAM is sensitive to the order of the inputs x y Category 1 - 2 members Category 2 - 1 member Category 4 - 3 members Vector 3 Vector 1 Vector 2
20
Knowledge Systems Lab JN 9/13/2015 Heterogeneous Network Voting scheme of 5 Heterogeneous SFAM networks to overcome vigilance and input order dependence –3 networks: random input order, set vigilance –2 networks: 3 rd network order, vigilance ± 10%
21
Knowledge Systems Lab JN 9/13/2015 Network Segmentation Results
22
Knowledge Systems Lab JN 9/13/2015 Segmentation Results Threshold results Overlay results Trans-slice results
23
Knowledge Systems Lab JN 9/13/2015 Segmentation Screenshot
24
Knowledge Systems Lab JN 9/13/2015 System Demonstration Interactive Learning
25
Knowledge Systems Lab JN 9/13/2015 Segmentation Solution Doctors and radiologists: –Spend several hours daily analyzing patient images (ie. MRI scans of the brain) –Search for patterns in images that are standard and well-known to doctors Solution: –Doctors and radiologists can teach the computer to recognize abnormal brain tissue –They can refine the learning systems results interactively
26
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions
27
Knowledge Systems Lab JN 9/13/2015 Zooming Approaches Inset Overlay Chip Window
28
Knowledge Systems Lab JN 9/13/2015 Research & Business Carpendale PhD Thesis –Elastic Presentation Space – rubber sheet images via mathematical constructs IDELIX (www.idelix.com) –Pliable Display Technology – software development kit (SDK) product –Boeing: 20% increase in productivity
29
Knowledge Systems Lab JN 9/13/2015 Zoom Visualization Wireframe View Contextual Zoom
30
Knowledge Systems Lab JN 9/13/2015 System Demonstration Contextual Zoom
31
Knowledge Systems Lab JN 9/13/2015 System Comparison Previous System Zoom Overlay Contextual Zoom
32
Knowledge Systems Lab JN 9/13/2015 Magnification Solution Doctors and radiologists: –Must be able to precisely view and select regions/pixels of the image to train the computer –Can easily lose where they are looking in the image when using magnification Solution –They can precisely select targets/non-targets –They can zoom for precision while maintaining context of the entire image –The interface facilitates task performance through interactive display of segmentation results
33
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions
34
Knowledge Systems Lab JN 9/13/2015 Motivation Gesturing is a natural form of communication: –Gesture naturally while talking –Babies gesture before they can talk Interaction problems with the mouse: –Have to locate cursor –Hard for some to control (Parkinsons or people on a train) –Limited forms of input from the mouse
35
Knowledge Systems Lab JN 9/13/2015 Motivation Problems with the Virtual Reality Glove as a gesture recognition device: –Reliability –Always connected –Encumbrance
36
Knowledge Systems Lab JN 9/13/2015 System Diagram Standard Web Camera Rendering User Interface Display Hand Movement User Gesture Recognition System Image Capture Update Object Image Input
37
Knowledge Systems Lab JN 9/13/2015 System Performance System: OpenCV and IPL libraries (from Intel) Input: 640x480 video image Hand calibration measure Output: Rough estimate of centroid Refined estimate of centroid Number of fingers being held up Manipulation of 3D skull in QT interface in response to gesturing
38
Knowledge Systems Lab JN 9/13/2015 Calibration Measure Max hand size in x and y orientation (number of pixels in 640x480 image)
39
Knowledge Systems Lab JN 9/13/2015 Saturation Extraction Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation
40
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline
41
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline
42
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline a) 0 th moment of an image: b) 1 st moment for x and y of an image, respectively: c) 2 nd moment for x and y of an image, respectively: d) Orientation of image major axis:
43
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Pipeline The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) Total fingers is number of fingers minus 1 for the hand itself
44
Knowledge Systems Lab JN 9/13/2015 System Setup System Configuration System GUI Layout
45
Knowledge Systems Lab JN 9/13/2015 Interaction Mapping Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out
46
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Demo
47
Knowledge Systems Lab JN 9/13/2015 Exploration Solution Doctors and radiologists: –Need to intuitively interact with the system to maximize task performance –Need to perform this interaction while being unencumbered Solution –Can use intuitive gesturing to interact with complex, 3D data –Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation
48
Knowledge Systems Lab JN 9/13/2015 Outline Introduction Techniques: Segmentation, Magnification, Exploration Solutions: –Interactive Learning –Contextual Zooming –Gesture Recognition Conclusions and Future Work
49
Knowledge Systems Lab JN 9/13/2015 Interactive Learning Users can teach the computer to recognize abnormal brain tissue They can refine the learning systems results interactively They can save/load agents for background diagnosis on a database of medical images or to allow expert analysis in the absence of a well-paid expert
50
Knowledge Systems Lab JN 9/13/2015 Contextual Zoom They can zoom for precisely viewing and selecting targets/non-targets while maintaining context of the entire image The interface facilitates task performance through interactive and customizable display of segmentation results This system can be used with any 2D images and even with 3D datasets with some minor alterations
51
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Can use intuitive gesturing to interact with complex, 3D data Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation Easily replicated and distributable Mapping gestures to interaction is an independent stage
52
Knowledge Systems Lab JN 9/13/2015 Gesture Recognition Dynamic Gesture Recognition Other interface applications include: graspable interfaces, 3D avatar / MoCap, multi-object manipulation in virtual environments, and augmented reality
53
Knowledge Systems Lab JN 9/13/2015 Platform Med-LIFE integration effort –Gesture Recognition has already been integrated into Med-LIFE’s Exploration tab –Contextual Zoom and Interactive learning have been combined, but not yet integrated into Med-LIFE’s learning tab Med-LIFE will function as a single application for medical image analysis
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.