1 KMeD: A Knowledge-Based Multimedia Medical Database System Wesley W. Chu Computer Science Department University of California, Los Angeles

2 KMeD A Knowledge-Based Multimedia Medical Distributed Database System A Cooperative, Spatial, Evolutionary Medical Database System Knowledge-Based Image Retrieval with Spatial and Temporal Constructs Wesley W. ChuComputer Science Department Alfonso F. CardenasComputer Science Department Ricky K. TairaDepartment of Radiological Sciences October 1, 1991 to September 30, 1993 July 1, 1993 to June 30, 1997 May 1, 1997 to April 30, 2001

3 Research Team Students John David N. Dionisio Chih-Cheng Hsu David Johnson Christine Chih Collaborators Computer Science Department Alfonso F. Cardenas UCLA Medical School Denise Aberle, MD Robert Lufkin, MD Ricky K. Taira, MD

4 Significance Query multimedia data based on image content and spatial predicates Use domain knowledge to relax and interpret medical queries Present integrated view of multiple temporal and evolutionary data in a timeline metaphor

5 Overview Image retrieval by feature and content Query relaxation Spatial query answering Similarity query answering Visual query interface Timeline interface Sample cases

6 Image Retrieval by Content Features size, shape, texture, density, histology Spatial Relations angle of coverage, shortest distance, overlapping ratio, contact ratio, relative direction Evolution of Object Growth fusion, fission

7

8

9

10

11 Characteristics of Medical Queries Multimedia Temporal Evolutionary Spatial Imprecise

12 OO’ 01 Om OO 01 On Evolution: Object O evolves into a new object O’ Fusion: Object 01, …, Om fuse into a new object Fission: Object O splits into object 01, …, On

13 Case a: Case c: The object exists with its supertype or aggregated type. The life span of the object starts with and ends before its supertype or aggregated type. Case b: Case d: The life span of the object starts after and ends with its supertype or aggregated type. The life span of the object starts after and ends before its supertype or aggregated type.

14 Lesion Micro- Lesion Micro- Lesion

15

16

17

18

19 Query Modification Techniques Relaxation Generalization Specialization Association

20 Generalization and Specialization More Conceptual Query Specific Query Conceptual Query Specific Query Generalization Specialization Generalization Specialization

21 Type Abstraction Hierarchy Presents abstract view of Types Attribute values Image features Temporal and evolutionary behavior Spatial relationships among objects Provides multi-level knowledge representation

22 TAH Generation for Numerical Attribute Values Relaxation Error Difference between the exact value and the returned approximate value The expected error is weighted by the probability of occurrence of each value DISC (Distribution Sensitive Clustering) is based on the attribute values and frequency distribution of the data

23 TAH Generation for Numerical Attribute Values (cont.) Computation Complexity: O(n 2 ), where n is the number of distinct value in a cluster DISC performs better than Biggest Cap (value only) or Max Entropy (frequency only) methods MDISC is developed for multiple attribute TAHs. Computation Complexity: O(mn 2 ), where m is the number of attributes

24 Query Relaxation Relax Attribute Query Yes Display Query Modification Answers Database TAHs No

25 Cooperative Querying for Medical Applications Query Find the treatment used for the tumor similar-to (loc, size) X 1 on 12 year-old Korean males. Relaxed Query Find the treatment used for the tumor Class X on preteen Asians. Association The success rate, side effects, and cost of the treatment.

26 Type Abstraction Hierarchies for Medical Domain Age Preteens TeenAdult Ethnic Group Asian Korean Chinese Japanese Filipino AfricanEuropean Tumor (location, size) Class X [loc 1 loc 3 ] [s 1 s 3 ] Class Y [loc Y s Y ] X 1 [loc 1 s 1 ] X 2 [loc 2 s 2 ] X 3 [loc 3 s 3 ]

27 Knowledge-Based Image Model Representation Level (features and contents) Brain Tumor Lateral Ventricle TAH SR(t,b) TAH Tumor Size TAH SR(t,l) TAH Lateral Ventricle SR: Spatial Relation b: Brain t: Tumor l: Lateral Ventricle Knowledge Level Schema Level SR(t,b) SR(t,l)

28 Knowledge Based Query Processing Queries Query Analysis and Feature Selection Knowledge-Based Content Matching Via TAHs Query Relaxation Query Answers

29 User Model To customize query conditions and knowledge-based query processing User type Default Parameter Values Feature and Content Matching Policies Complete Match Partial Match

30 User Model (cont.) Relaxation Control Policies Relaxation Order Unrelaxable Object Preference List Measure for Ranking

31

32

33 Query Preprocessing Segment and label contours for objects of interest Determine relevant features and spatial relationships (e.g., location, containment, intersection) of the selected objects Organize the features and spatial relationships of objects into a feature database Classify the feature database into a Type Abstraction Hierarchy (TAH)

34 Similarity Query Answering Determine relevant features based on query input Select TAH based on these features Traverse through the TAH nodes to match all the images with similar features in the database Present the images and rank their similarity (e.g., by mean square error)

35 Spatial Query Answering Preprocessing Draw and label contours for objects of interest Determine relevant features and spatial relationships (e.g., location, containment, intersection) of the selected objects Organize the features and spatial relationships of objects into a feature database Classify the feature database into a type abstraction hierarchy (TAH)

36 Spatial Query Answering (cont.) Processing Select TAH based on t he query conditions and context Search nodes to match the query conditions Return images linked to the TAH node

37 Similarity Query Answering Preprocessing Select objects and specify features of interest in the image Create a feature database of the selected objects for all images Classify the feature databases as type abstraction hierarchies

38 Similarity Query Answering (cont.) Processing Determine relevant features based on query input Select TAH based on these features (interact with user to resolve ambiguity) Traverse through the TAH nodes to match all the images with similar features in the databases Present the images and rank their similarity (e.g., by mean square error)

39

40 Visual Query Language and Interface Point-click-drag interface Objects may be represented iconically Spatial relationships among objects are represented graphically

41 Visual Query Example Retrieve brain tumor cases where a tumor is located in the region as indicated in the picture

42

43

44

45

46

47

48

49

50

51

52 Implementation Sun Sparc 20 workstations (128 MB RAM, 24-bit frame buffer) Oracle Database Management System X/Motif Development Environment, C++ Mass Storage of Images (9 GB)

53

54

55

56

57 Conclusions Image retrieval by feature and content Matching and relaxation images based on features Processing of queries based on spatial relationships among objects Answering of imprecise queries Expression of queries via visual query language Integrated view of temporal multimedia data in a timeline metaphor

58

59

60 Semi-Automatic Segmentation of Lung Tumors classification seed estimation seed estimation adaptive fusion region growing region growing tumor segment tumor segment interesting area

61

62