Nov. 29, 2001 Ontology Based Recognition of Complex Objects --- Problems to be Solved Develop Base Object Recognition algorithms that identify non-decomposable objects in images. Assign classes to non-decomposable objects, also called base objects using a neural tree (or other) approach Identify low-level object sets/cliques that are suspected to be parts of instances of complex objects. Develop complex object recognition algorithms that assigns classes to complex object (mostly relying on ontological information) Develop an image indexing algorithm that assigns structural summaries to images relying on a domain ontology. Develop image search engines that matches structural image summaries against image queries / target images.

Example (First 5 Steps) X Step 1 Step 3+4 X X X Step 2 Step 5

Ontology Language Constructs Base Concepts (cannot be decomposed further) Complex concepts that are decomposed using base and complex concepts. Language constructs to define complex concepts include: S, S?(possibly S), S*, S+, S{n,},S{n,m}, S{n}; | (or operator); concatenation; permutations; ‘(‘ and ‘(‘ for operator precedence. Moreover, inheritance is supported through the use of a union/generalization operator.

Example Ontology ONT1 Base Concept A; Base Concept B; Base Concept C; Complex Concept D; signature=PERM(A?, B+, C{1,3}); Complex Concept E; signature=PERM(B|D)?, C+) Complex Concept F generalizes D, E; signature=A+B Complex Concept G signature= PERM(B, C{2,}) Complex Concept H generalizes D, G A B C D G E F H

Task3: Develop a higher order concept identification algorithm that identifies high order objects and assigns categories to them based on an ontology and relationships of the base concepts in the image Question1: What information will this algorithm use as its input? Structural relationships between observed parts and structural descriptions in the ontology. Importance of the presence of the low level concept for the higher level concept?!? Relationships of instances of lower level concepts in the image (measured by distance of object-centroids, percentage of object overlap, mean gray-level, variance in gray level) Other information extracted from the image --- what might this be?

Example Problem TASK3 using ONT1 C C C C B A C C B C B A A B C C B

IHLO Algorithm: Inferring Higher Level Objects Classify low level objects Identify cliques of objects that form higher level objects using centroid distance, percentage of overlap, mean and variance of gray levels and binary part-of-compatibility of low level objects. This is done in two steps: Compute connectivity matrix for all base objects in the image (contains number in [0,1] measuring degree of connectivity) Infer cliques from connectivity matrix (a simple version just takes the number in the matrix; a more complex version might use ontological information) Classify identified higher level objects based on ontological information relying on an evidence based algorithm. Improvements: 1) Interleave steps 2 and 3 2) Use Set-based Compatibility

Remarks IHLO Algorithm Step2 uses ontological information to exclude two objects “object1 and object2 are part of a higher level object”; in general, it relies more strongly on image specific data for the relationship of two objects in the image. Step3 tries to infer the higher level object by weighting the evidence and by assigning a number to each higher level object H that measures the our belief that the complex object is instance of concept H. Usually, the concept with the highest degree of belief will be used to label the complex object; however, if evidence is below a certain threshold the object will be classified as “unknown”. IHLO must allow for unclassified base objects (base objects whose class membership is unknown)

Important Concepts for the Algorithm Binary Part-of-Compatibility: A set of two objects belonging to concepts X and Y are compatible if there is a least one higher order concept H in the ontology that contains members of X and Y as parts. General Part-of-Compatibility: A set of objects (belonging to various concepts) is compatible if there is at least one higher concept in the ontology that instances can be decomposed of the observed set of parts. Partial levels of compatibility can be inferred by analyzing compatibility of subsets.

Questions1 The goals of objectives of the project sound very challenging but it also seems to be quite labor-intensive. What are your thoughts on how many people and what kind of people should work on the project? Time frame, languages used, who is currently working on the system; if I would find a student to work on the ontology part of the system would that be difficult? We need a benchmark. One interesting problem is to work on a matching algorithm, but in order to evaluate this structural matching algorithm we need a set of images that have been “either hand indexed or automatically indexed” using a given ontology. What relationships should be supported by the ontology language (the currently supported relationships include: is-a and part of)? Do we need object properties in addition to relationships in the ontology language?

Questions2 How does the neural tree learning algorithm exactly work. How is the training set up? How is the quality of the learnt classification algorithm accessed? How automated is the current neural tree learning algorithm? Are you thinking to have just one neural tree or many neural trees for a particular application domain. The concept learning algorithm assumes that we have example of images / sub-images that have been labeled by a human being. How is the labeling / annotation procedure currently performed? Is there a way to make it less labor-intensive? What kind of query languages do we need for searching image databases in the context of ontology-based indexing. How do solve the document/image (instance) vs. class problem? We have concepts in our ontology, but be have sets of related instances in a single image? How do we solve the “cars with at least 3 tires / 2 dancers in an image” problem. contd..

Ideas The ontology defines the ingredients / structural language that allow use to assign structural approximations to images. Structural approximation of an image: Set of instances of concepts + relationships between these instances + properties of those instances. Matching Problems: Match the structural description of an image against the structural description of a set of images. Match a query against a set of images Cluster images; define a new concept in each cluster for the ontology

Shared Ontologies “Ontologies are content theories about sorts of objects, properties of objects, and relationship between objects that are possible in a specified domain of knowledge” (Chandrasekaran) “We consider ontologies to be domain theories that specify a domain-specific vocabulary of entities, classes, properties, predicates, and functions, and a set of relationships that necessarily hold among those vocabulary items” (Fikes) “Shared ontologies form the basis for domain specific knowledge representation languages” (Chandrasekaran) “If we could develop ontologies that could be used as the basis of multiple systems, they would share a common terminology that would facilitate sharing and reuse” (W. Swartout) “Ontologies play an important role for the standardization of terminology in medicine (e.g. UMLS) and other domains” “Ontologies can serve as the glue between knowledge that is represented at different, usually heterogeneous information sources.” The last technology I like to introduce in today’s presentation are shared ontologies. Shared ontologies are important to standardize communication, and for gathering information from different information sources. Ontologies play an important role for agent-based systems. Ontologies basically describe...

What are Ontologies good for? As a shared conceptual model of a particular application domain that describes the semantics of the objects that are part of the domain, and captures knowledge that is inherent to the particular domain --- idea: knowledge base . Ontologies provide a vocabulary for representing knowledge about a domain and for describing specific situations in a domain (tool for defining and describing domain-specific vocabularies) --- idea: language for communication For data/knowledge translation and transformation (provide a solution to the translation problem between different terminologies); for fusion and refinement of existing knowledge --- idea: interoperation For matchmaking between users, agents, and information resources in agent-based systems --- idea: collaboration, brokering focus of next slides As reusable building blocks to build systems that solve particular problems in the application domain --- idea: model reuse Summary: “Ontologies can be used as building block components of knowledge bases, object schema for object-oriented systems, conceptual schema for data bases, structured glossaries for human collaborations, vocabularies for communication between agents, class definitions for conventional software system, etc.” (Fikes)

A “Traditional” Approach Service Provider Agents End User Agents A “Traditional” Approach Search Engine Specify keywords with respect to the documents they are looking for Abstract Clinical Trial Report Clinical Trial Report Summary Semantic Brokering Approach Service Provider Agents End User Agents Semantic Brokering Specify subset of ontology Subset of an Ontology Clinical Trial Report Summary := matchmaking

Example Semantic Brokering Data Analyst’s Information Requirement Patient Result Semantic Brokering: ((DataCollection1 nil ((missing slot weight) (contradictory (< age 15) (> age 40)) (DataCollection2 t) (DataCollection3 t ((> age 60)(> weight 300))) Age>40 weight Intensive-Care- Patient Hours-in-intensive-care Data Collection1 Data Collection2 Data Collection3 Patient Patient Patient Age<15 age Age>60 weight Weight>300 Intensive-Care- Patient Intensive-Care- Patient Intensive-Care- Patient Hours-in-intensive-care Hours-in-intensive-care Hours-in-intensive-care

Image Matching Algorithms What do Image Approximations contain? A set of higher order objects belonging to a category, each higher order object is described by a set of lower level objects it is decomposed of and a set of properties(all of which have been defined by an ontology). A set of intermediate objects that are part of an higher order object with their properties and parts A set of base objects with their properties

Object Similarity Object Similarity := Class Membership Similarity + Attribute Similarity + Attribute-value Similarity + Component Similarity + Link Similarity Simple Object Similarity := Class Membership Similarity + Attribute Similarity + Attribute-value Similarity + Component Similarity Class Membership Similarity := similarity of the set of classes the object belongs to Attribute Similarity := similarity of the bag (set?) of attributes the object has Attribute Value Similarity := weighted sum of Single-attribute value similarity Single Attribute Value Similarity := similarity of the attribute value or the set of attribute values for the particular attribute Component Similarity := similarity of the set of objects the higher level object is composed of (using simple object similarity) Link Similarity := weighted sum of Single-Link similarity Single-link Similarity := similarity of the set of objects the object is pointing to (using simple object similarity) for the particular link Problem: How can object-similarity efficiently computed in spite of the fact that its definition is highly recursive (perhaps we should initially exclude link similarity from the computations)? Work plan: Start of with single object similarity (ignoring link information) and add treatment of links on the top of the designed similarity framework.

Possible Example Application for Semantic Brokering Italian Image Collection Mars Crater Image Collection Structural summaries generated from Computer Science conference publications ??????