NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens

Knowledge Assisted Analysis

KAA for Images A set of regions is generated by an initial segmentation of images MPEG-7 Visual Descriptors (dominant color, texture, shape) are extracted Spatial relations (left-of, above-of, etc.) Regions Adjacency Graph as image representation

Regions Adjacency Graph A graph’s node represents a segment/region, where visual information (MPEG-7 descriptors, spatial relations, region mask, contour, etc.) is stored A graph’s edge represents link between two regions, holding the overall neighboring information

Region labeling For each region: Visual Descriptor matching with the instances of the concepts in the domain ontology Calculation of a combined distance from multiple descriptors Assignment of labels (concepts) along with a confidence of value -> fuzzy set of labels Hierarchical merging of regions based on the fuzzy set of labels

KAA architecture

Semantic based segmentation (1/2) Approach Graph-based representation of images Semantic vs. Syntactic: regions are assigned fuzzy set of labels instead of low-level features Modification of traditional segmentation algorithms to operate on labeled regions Simultaneous image segmentation and region labeling

Semantic based segmentation (2/2) Target: Solve over-segmentation problems Assign labels with confidence values to regions Link labels with concepts existing in ontologies

KAA example results

NEST …

Expert systems at UEP - history of the NEST May 2003: begining of implementation (P. Berka, V. Laš) DELPHI under Windows knowlege base represented in XML stand-alone + client/server (web) version knowledge base editor czech and english versions

Knowledge representation (1/4) Attributes and propositions binary True, False single nominal values of attribute multiple nominal values of attribute numeric fuzzy intervals sources and actions related to attributes attribute describes case or environment

Knowledge representation (2/4) Rules with priorities IF condition THEN conclusion AND action where condition is disjunctive form (disjunction of conjunctions) of literals (propositions or their negations), conclusion is a list of literals and action is a list of actions (external programms) compositional - each literal in conclusion has a weight apriori - compositional rules without condition logical - non-compositional rules without weights; only these rules can infer true or false

Knowledge representation (4/4) Integrity constraints ANT  SUC (degree) where ANT and SUC are DNF of literals and degree is a number expressing the importance of the integrity constraint used to check logical consistency of the consultation diagnosis(TBC)  not diagnosis(healthy) Contexts - disjunctive form of literals, that (iff having positive weight) determines the applicability of a rule or integrity constraint

Inference Inference in the network of rules as a combination of backward and forward chaining compositional inference for compositional and apriori rules (combining contributions of rules) non-compositional inference for logical rules (modus ponens + disjunction) Evaluation of integrity constraints IMPL(a,s) = max(0, min(1, a-s)) pro a > 0

Uncertainty processing (1/4) (Based on the algebraic theory of P. Hájek ) defined combination functions on [-1, 1]: NEG to compute the weight of negation, CONJ to compute the weight of conjunction, DISJ to compute the weight of disjunction, CTR to compute the contribution of a rule to the weight of conclusion, GLOB to combine contributions of more rules.

Uncertainty processing (3/4) NEG(w) = - w CONJ(w 1,w 2 ) = min(w 1,w 2 ) DISJ(w 1,w 2 ) = max(w 1,w 2 )

Modes of consultation dialogue mode - classical question/answer mode that selects current question using backward chaining dialogue/questionnaire mode – user can input some volunteer information (using questionnaire), during furthe consultation the system asks questions if needed questionnaire mode – after filling in the questionnaire the system directly inferrs the goals using forward chaining input answers form a file – answers can be changed using questionnaire

Types of answers binary attribute - weight single nominal attribute – value and weight multiple nominal attribute – list of values and their weights numeric attribute - value Questions not answered during consultation get the default answer „unknown“ [-1,1] or “irrelevant“ [0,0], Answers can be postponed - user can return to them after finishing the consultation

Consultation setup

Input from file (batch mode)

… for KAA

Basic idea (1/2) Expert system NEST (or it’s principles) can be used in KAA for: re-labeling a region if the original labeling has low confidence proposing to merge a region with it’s neighbors These two tasks can be solved separately, by two different knowledge bases (expert systems – ES).

Basic idea (2/2) Because NEST cannot express relations between objects, NEST will be used to process the image locally, i.e. to process one object in one step. So, NEST will be activated repeatedly for different regions in the image. This will require to determine some control mechanism that will decide: what region to take when to stop processing

Processed part of image – version 1 (Athens meeting)

NEST for re-labeling (1/5) Input: labels (and confidences) of the processed region labels (and confidences) of the neighboring region some global info?

NEST for re-labeling (2/5) So the input can be for example “sky(0.6), sea(0.8)…”. To be able to reason about the confidences, NEST has to turn them into (fuzzy) intervals like “very_low”, “low”, “medium”, “high”, “very_high” – this can be easily done when creating the knowledge base:

NEST for re-labeling (3/5) Output: (new) labels (and confidences) of the processed region Used knowledge: IF the labels have high confidence, THEN don’t change the labels ELSE change the labels according to the neighbors

NEST for re-labeling (4/5) Examples of rules: IF old_label(sky) THEN new_label(sky) (1) IF region_confidence_sky(very_low) AND region_confidence_sea(very_low) AND region_confidence_sand(very_low) AND west_confidence_sky(high) AND east_confidence_sky(high) THEN new_label(sky) (0.6),new_label(unidentified)(0.9)

NEST for re-labeling (5/5) General strategy: This module can be activated for each region once e.g. according to the confidence of labeling, starting with lowest confidence. The stopping criterion can have a form of a threshold of the confidence.

NEST for merging (1/3) Input: labels (and confidences) of the processed region labels (and confidences) of the neighboring region some global info? Output: recommendation for processed region: e.g. merge_west, merge_east, merge_north, merge_south, don’t_merge probably also labels (and confidences) for the merged region

NEST for merging (2/3) Used knowledge: The knowledge can be generalized as follows: IF the neighbors have same labels THEN merge ELSE don’t merge Example: THEN don’t_merge (0.5) {apriori rule that says, that we prefer not to merge} IF region_confidence_sky(high) AND west_confidence_sky(hig) THEN merge_west (1), merged_confidence(sky) (0.85)

NEST for merging (3/3) General strategy: This module can be activated for the regions repeatedly (performing a kind of bottom-up clustering of regions), starting e.g. with region with highest visual salience? The stopping criterion can have a form of a threshold of the salience (not to handle uninteresting regions)?

Processed part of image – version 2 (EKAW poster)

NEST for merging (1/3) Input: labels (and confidences) of the regions A and B labels (and confidences) of the neighboring regions some global info? Output: (dis)similarity between regions A and B

NEST for merging (2/3) Used knowledge: The knowledge can be generalized as follows: IF regions A and B have same labels THEN similar ELSE dissimilar Example: IF regionA_confidence_sky(high) AND regionB_confidence_sky(high) THEN similar (1)

NEST for merging (3/3) General strategy (Semantic Recursive Shortest Spanning Tree): 1. use NEST to evaluate dissimilarity of current neighbors (edges in ARG) 2. select neighbors with lowest dissimilarity 3. IF this value is below given threshold THEN a. merge neighbors A and B b. assign new label c. goto step 1.

Alternative approach Merge two adjacent regions if they have the same distribution (in quantitative sense) of classes: assign labels according to different combinations of majority classes - so for k classes (like sea, sky …) we will have 2 k -1 labels merge neighbor regions with the same label (Inspired by my algorithms for discretization and grouping)

NEST – to do Create knowledge base Implement convertors to transform data between KAA and NEST (xml -> xml) Build inference mechanism of NEST into KAA