Workshop on Semantic Knowledge in Computer Vision, ICCV 2005 Symbol Grounding for Semantic Image Interpretation: From Image Data to Semantics Céline Hudelot, Monique Thonnat and Nicolas Maillot INRIA Sophia Antipolis, FRANCE

16/10/2005 SKCV 2/24 Outline Introduction Symbol Grounding Problem Ontology-based Communication Learning Approach Knowledge-based Approach A Symbol Grounding Engine Conclusion

16/10/2005 SKCV 3/24 Introduction Problem: What does it means to perform semantic image interpretation ? What does it means to associate semantics to a particular image ?

16/10/2005 SKCV 4/24 Introduction Different interpretations are possible Image semantics is not inside the image Image interpretation depends on a priori knowledge and on the context A white object on a green background An insect An infection of white flies on a rose leaf

16/10/2005 SKCV 5/24 Introduction Three abstraction levels of data Vision [Marr,82], Cognitive Science [Gardenfors,2000] Semantic level Image level Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6)... APPLE FRUIT PEARORANGE PLATE_OF_FRUITS Composition link Specialization link STEM PEACH Visual level A circular shape, orange hue and regular granulated texture

16/10/2005 SKCV 6/24 Introduction Three sub-problems: Image processing : extraction of numerical image data Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6)... SEGMENTATION FEATURE EXTRACTION Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6)... Orange Fruit : Has for shape : circular Has for hue: orange Has for texture : granulated Symbol grounding : mapping between image data and high level representations of semantic concepts Symbol grounding Semantic interpretation : reasoning at the high level

16/10/2005 SKCV 7/24 The Symbol Grounding Problem Definition: Problem of the mapping between image data and semantic data Objective Propose generic tools to solve the symbol grounding problem as a problem as such Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6)... The Orange Fruit

16/10/2005 SKCV 8/24 The Symbol Grounding Problem Proposed Approach An independent intermediate level called visual level between the semantic level and the image level Two ontologies to make easier the communication between the different levels Visual concept ontology Image processing ontology A cognitive vision approach involving a priori knowledge and machine learning

16/10/2005 SKCV 9/24 The Symbol Grounding Problem Proposed Approach Semantic level Image level Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6)... Visual level A circular shape, orange hue and regular granulated texture Orange Visual concept ontology Image processing ontology Symbol grounding problem : matching image data with combination of visual concepts

16/10/2005 SKCV 10/24 The Symbol Grounding Problem Proposed Approach Build the correspondence links between images features and visual concepts Learning approach : the correspondence links are learned from images samples A priori knowledge based approach: links are built explicitly and stored in a knowledge base A symbol grounding engine uses these links to perform the matching

16/10/2005 SKCV 11/24 Ontology Based Communication A visual concept ontology [Maillot et al. 03] Experts often use and share a generic visual vocabulary to describe their domain A hierarchy of three kinds of 2D visual concepts Spatial Concepts (64 concepts) Shape, Size: circular, elongated,… Spatial Structure : network of, ring of,… Binary spatial relations : near of, connected to, left of Color Concepts (37 concepts) : red, light, vivid (ISCC-NBS lexicon) Texture Concepts (14 concepts) : granulated, regular (cognitive studies [Bhushan,97]) Application independent A basis for further extensions

16/10/2005 SKCV 12/24 Ontology Based Communication Why a Visual Concept Ontology ? To guide and constrain the semantic knowledge acquisition Reduce the semantic gap : a shared representation of image content at an intermediate level Communication between the semantic level and the visual level Domain Expert Images Samples Visual Concept Ontology Knowledge Acquisition Knowledge Base Manually Segmented and Annotated Images

16/10/2005 SKCV 13/24 Ontology Based Communication An Image Processing Ontology Domain of discourse of image processing: set of generic terms to describe images and image processing results “Images have an ontological description of their own” Hierarchical set of : Image entity concepts : region, edge, graph …(11 concepts) Image feature concepts : eccentricity, RGB values, … (167 concepts) Image processing functionalities :object_extraction, feature_extraction,… (5 generic functionalities) Communication level between the image level and the visual level Not complete, a basis for further extensions

16/10/2005 SKCV 14/24 Supervised Learning Approach Goal: Training a set of detectors (e.g. Multi Layer perceptrons, SVM) to the detection of visual concepts Each visual concept C is associated to a set of image features F C Only visual concepts used during the semantic knowledge acquisition phase are learned Positive and negative samples of each visual concept Visual Concept Detectors Feature Selection Training Feature Extraction

16/10/2005 SKCV 15/24 Supervised Learning Approach Example : learning of the visual concept granulated texture Granulated Texture Detector LDA NN Positive and negative samples of visual concept Granulated Texture Gabor Filter circular shape orange hue granulated texture Manually segmented and annotated images

16/10/2005 SKCV 16/24 Supervised Learning Approach  Reduce the learning problem by addressing it at an intermediate level of semantics  Automatic building of the symbol grounding link between visual concepts C and image features F  Does not learn spatial structure and spatial relations  Dependent on the learning base : a large amount of image samples is needed

16/10/2005 SKCV 17/24 A Priori Knowledge Base Approach Explicit representation with frames: Visual concepts (symbolic data): description of visual concepts C and of their grounding link with image features F Image data concepts (image data): primitives (ridge, region, edge), features (area, eccentricity) organized in feature sets Spatial relations : topology (RCC8), distance and orientation Explicit representation with rules: Object extraction criteria: to constrain image processing Spatial deduction criteria: to infer spatial relations

16/10/2005 SKCV 18/24 A Priori Knowledge Based Approach Visual concept : simple examples VisualConcept{ name Circular_Surface Super Concept Elliptical_Surface Grounding Link Symbol name eccentricity Comment ratio of the length of the longest chord to the longest chord perpendicular to it Linguistic-values [ high very_high] FuzzySet Fhigh ={0.57, 0.62, 0.76, 0.84} Fvery_high ={0.76, 0.84, 1, 1} Domain [0 1] Symbol name compactness Linguistic-values [ high very_high] …} VisualConcept{ name Orange Super Concept Hue Grounding Link Float name H_value Domain [ ] Float name L_value Domain [ ] }

16/10/2005 SKCV 19/24 A Priori Knowledge Base Approach Explicit representation of spatial relations [Le Ber, 98] : distance, orientation, topology (Binary, 2D) Spatial Relation{ name Externally_Connected Super Relation Discrete Inverse Externally_Connected Complement None Symmetry true Conditions Intersection(Interior(O1), Interior(O2))=Ø Intersection(Boundary(O1),Boundary(O2))!=Ø Objects_In_Relation VisualObject name O1 VisualObject name O2} Spatial Relation{ name Near_of Super Relation DistanceRelation Inverse Near_Of Complement Far_From Symmetry true Float name distance_threshold Conditions Distance(O1,O2) < distance_threshold Objects_In_Relation VisualObject name O1 VisualObject name O2}

16/10/2005 SKCV 20/24 A Priori Knowledge Base Approach Object extraction criteria: how to constrain image processing (using visual concepts and spatial relations) Example Spatial deduction criteria : how to infer spatial relations from other ones Example: Rule { Let c a visual content context and O a visual object If O.geometry is a Open Curve and O.width is {Thin, Very Thin} then c.ImageEntityType:=Curvilinear Structure } Rule { Let O1, O2, O3 three visual objects If NTTP(O1, O2) is true and Left_Of(O2,O3) is true then Left_Of(O1,O3) is true} O2 O1 O3

16/10/2005 SKCV 21/24 A Priori Knowledge Base Approach  Reduce the learning problem by addressing it at an intermediate level of semantics  No need of image samples  Spatial relations are explicit  Manual building of the symbol grounding links between visual concepts C and image features F  Difficult to express some criteria for texture

16/10/2005 SKCV 22/24 Symbol Grounding Engine Symbol Grounding (Symbols, Image) Image processing request building using object extraction criteria Primitive selection (region, ridge,…) Feature extraction Matching between image processing results (image features F) and symbolic data (visual concepts C) Fuzzy Matching using explicit knowledge (Frames) OR, Matching using the detectors obtained during the learning Spatial Reasoning for multiple objects management using spatial deduction criteria and spatial relations

16/10/2005 SKCV 23/24 Conclusion The two methods have been tested on real world applications A priori knowledge based approach : Automatic early diagnosis of rose disease [Hudelot et al 03] Supervised learning approach : Application on aircraft/cars retrieval [Maillot et al 05] Two complementary methods The symbol grounding link is difficult to build explicitly by a human expert in vision (e.g. texture concepts) A large amount of data (image examples) is not available for all the applications

16/10/2005 SKCV 24/24 Conclusion Original Symbol Grounding Approach: Ontology-based Approach Visual concept ontology and Image processing ontology Independence between application domain semantics and image processing library Symbol grounding link Either learned from samples or a priori knowledge Future works Learning for spatial relations Extension of the visual concept ontology Temporal concepts

16/10/2005 SKCV 25/24 Symbol Grounding for Semantic Image Interpretation: From Image Data to Semantics Any Questions??

16/10/2005 SKCV 26/24 The Symbol Grounding Problem Related Works Knowledge based Vision: Not often considered as a problem as such Encapsulated in the semantic level Intermediate Symbolic Representation [Brolio,89] VISIONS system [Hanson,78] Database management technology Conceptual Spaces [Chella, 1997] Conceptual space = metric space which dimensions are entity qualities Natural concepts = convex regions in the conceptual space

16/10/2005 SKCV 27/24 The Symbol Grounding Problem Related Works Artificial intelligence : the Symbol grounding problem [Harnad, 90] Robotics community: the Anchoring problem « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99] Image retrieval community : the semantic gap Use of ontological engineering: object ontology [Mezaris, 04], visual ontology [Mao,98], ontology for language based querying [Town, 04]

16/10/2005 SKCV 28/24 Ontology Based Communication Ontology : set of concepts and relations useful to describe a domain “A formal, explicit specification of a shared conceptualization” [Gruber, 93] Conceptualization : abstract relevant model of a phenomenon Explicit : the meaning of the concepts is defined explicitly Formal : machine readable Shared : consensual knowledge accepted by a group