Towards Cognitive Vision: Knowledge and Reasoning for Image Analysis and Understanding Monique THONNAT Orion team INRIA Sophia Antipolis FRANCE

16/08/2004 Bonn 2 Introduction Past Research Object Categorisation Program Supervision Cognitive Vision Platform Application Conclusion Overview

16/08/2004 Bonn 3 Introduction: the Problem Problem 1: What does it mean to perform image interpretation ? semantic image interpretation (i.e. object classification) What does it mean to associate semantics to a particular image ?

16/08/2004 Bonn 4 Different interpretations of this image are possible: A light object on a dark background An astronomical object NGC4473 galaxy Introduction: the Problem Image semantics is not inside the image Image interpretation depends on a priori knowledge

16/08/2004 Bonn 5 Introduction: the Problem Problem 2: Is it possible to build fully automatic image interpretation systems? Given an image and a set of vision programs: which programs have to be applied on a particular image ?

16/08/2004 Bonn 6 Introduction: the Problem Is there a unique or an optimal solution ? In this image: shall we extract the biggest object in the center? shall we detect all the objects in the image? If solution depends on the objective low level processing related to high level interpretation?

16/08/2004 Bonn 7 Introduction: the Approach Proposed approach: Knowledge-based Vision formalize the a priori knowledge for image interpretation in knowledge bases explicit the reasoning (how to use a priori knowledge) define sub-problems and build solutions independent of any application (object categorization, program supervision, video understanding) test the solutions on difficult real cases

16/08/2004 Bonn 8 Introduction Past Research Object Categorisation Program Supervision Cognitive Vision Platform Application Conclusion Overview

16/08/2004 Bonn 9 Object Categorisation Goal: object categorization = find the class of an object (versus object identification = find a particular individual) Focus: complex natural objects with existing taxonomy Method: a priori knowledge of hierarchy of classes

16/08/2004 Bonn 10 Object Categorisation: Method Image Parameters Class + Description Classification Object description Image processing Knowledge-based system CLASSIC engine

16/08/2004 Bonn 11 Object Categorisation: Knowledge Knowledge formalization: Class: symbolic description of a class implemented by frames Tree: predefined hierarchy of classes implemented by a tree of frames Rules: inference of symbolic terms from numerical parameters implemented by inference rules

16/08/2004 Bonn 12 Object Categorisation: Reasoning CLASSIC: Categorization engine Algorithm: Depth-first tree traversal from root (general class) to leaves (specific classes) 3 recursive phases: data abstraction (activation of inference rules) object - class matching classification refinement (tree traversal)

16/08/2004 Bonn 13 Object Categorisation: Applications Galaxy classification: Fish sorting Zooplankton classification: Foraminifers classification: Pollen recognition:

16/08/2004 Bonn 14 Object Categorisation: Applications 35 parameters Class=E5 SYGAL Knowledge based system Galaxy description Built with CLASSIC engine

16/08/2004 Bonn 15 Object Categorisation: Applications Tree of galaxy classes (ref. De Vaucouleurs)

16/08/2004 Bonn 16 Object Categorisation: Conclusion + 1 unique engine CLASSIC for several applications + 3 PhD thesis (M-H Gandelin, S Liu-Yu, J-C Ossola) + 1 European project ASTHMA - how to perform object segmentation? --> program supervision - only one single object --> future research on cognitive vision - mapping between domain objects and vision features --> future research on ontology and learning

16/08/2004 Bonn 17 Program Supervision Program supervision definition : automate the configuration and execution of programs (versus fixed procedure) selection, scheduling, execution, and control of results Focus: mainly image processing libraries of programs Method: knowledge-based approach: explicit formalization of expertise on how to use programs Planning techniques (HTN)

16/08/2004 Bonn 18 Program Supervision: Reasoning Reasoning: Program selection in a library of programs Selected programs execution Evaluation and adjustment if needed Program Supervision Engine Library of programs Program Utilisation KB PlanningExecution EvaluationRepair results plan (part of) judgements Actions to correct correct incorrect Request + data

16/08/2004 Bonn 19 Program Supervision: Knowledge Operator: a means of achieving a specific goal primitive operator: a particular program composite operator: a particular combination of programs sequential decomposition (THEN) parallel decomposition (AND) specialization (OR) Criteria: decision rules attached to an operator choice rules parameter initialization rules evaluation rules repair rules

16/08/2004 Bonn 20 Program Supervision: Applications These techniques have been used for very different applications: Road obstacle detection: off-line processing of stereovision programs real-time reconfiguration of programs for Vehicle Driving Assistance Medical imaging: segmentation of 3D MRI brain images Astronomical imagery: morphological description of galaxies

16/08/2004 Bonn 21 Program Supervision : Conclusion + 4 more or less sophisticated engines: Ocapi, Planete (real-time) Pegase (repair), MedIA (planning) + 7 PhD thesis (V. Clement, J. Van den Elst, J-C Ossola, R. Vincent, M. Crubezy, M. Marcos, J-C Noel) + 1 European project PROMETHEUS - communication between program supervision and image interpretation --> future research on cognitive vision - program supervision engines handling data flow (videos) and temporal constraints (real-time) --> future research on video understanding

16/08/2004 Bonn 22 Past Research : Where We Are + Formalization of knowledge & reasoning Feasibility proven on difficult cases Reusable engines & knowledge languages - No general solution only partial ones for 3 sub-problems object categorization, program supervision, video understanding Knowledge bases still difficult to build

16/08/2004 Bonn 23 Introduction The Problem The Approach Past Research Object Categorisation Program Supervision Cognitive Vision Platform Application Conclusion Overview

16/08/2004 Bonn 24 Where we go: No general solution for image interpretation --> Conception of a cognitive vision platform (Celine Hudelot PhD) Knowledge bases still difficult to build --> Design of ontology driven knowledge bases i.e. visual concept ontology, video event ontology (Nicolas Maillot & T. Van Vu PhDs) --> Design of learning techniques to complement knowledge bases (N. Maillot & V. Martin PhDs) Cognitive Vision: Introduction

16/08/2004 Bonn 25 Cognitive Vision Platform Cooperation of three knowledge based systems: Separate the problem of image interpretation into tractable tasks For each task: Formalize the different types of knowledge involved in the image interpretation problem Explicit the reasoning ( use of a priori knowledge)

16/08/2004 Bonn 26 Cognitive Vision Platform A platform with 3 dedicated tasks Semantic data interpretation Application expert knowledge (domain taxonomy and terminology) Ontological engineering to facilitate knowledge acquisition Data management Matching between numerical image data and symbols Scene analysis using spatial reasoning Image processing numerical object description program supervision techniques : to automate the management of an image processing library

16/08/2004 Bonn 27 Cognitive Vision Platform Data Management Knowledge Base of Visual Concepts and Data Data Management Engine Interpretation Knowledge Base of Application Domain and Visual Concepts Interpretation Engine Program Supervision Library of vision programs Knowledge Base of Program Utilization Program Supervision Engine Current Image Interpretation Object Hypotheses Image Processing Request Numerical data Image description Visual Concept Ontology Use of visual concept ontology in a cognitive vision platform

16/08/2004 Bonn 28 Cognitive Vision: Interpretation Goal: Find the semantic class of physical objects observed on images How: Perform the interpretation in the same way experts do: Use of domain terminology and taxonomy Top down strategy Physical object hypothesis propagation

16/08/2004 Bonn 29 Cognitive Vision: Interpretation Knowledge Acquisition: Ontological engineering contribution Ontology : set of concepts and relations useful to describe a domain [maillot03]: contribution of a visual concept ontology for the task of object description: spatio-temporal concepts color concepts texture concepts Domain concepts are described by visual concepts

16/08/2004 Bonn 30 Cognitive Vision: Interpretation Texture Regular TextureIrregular Texture Smooth Texture Weaved Texture Oriented Texture Periodic Texture Granulated Texture Marbled Texture Veined Texture 3D Texture Visual concept ontology content: some texture concepts

16/08/2004 Bonn 31 Cognitive Vision: Interpretation Knowledge Formalization Domain concept tree : specialization relations Sub-part tree linked to domain concept Reflects the domain taxonomy Class: a domain concept (plant leaf, pollen grain) described by visual concepts (green color and oval shape or pink and circular) Representation by frames with slots

16/08/2004 Bonn 32 Cognitive Vision: Interpretation Subpart Tree Poaceae : Circular Shape Granulated Texture Pink Color Poaceae Pore Cytoplasm Pore: Subpart of Poaceae Elliptic Shape Small Size Domain knowledge described using visual concept ontology

16/08/2004 Bonn 33 Cognitive Vision: Interpretation Knowledge Formalization Domain Concept name White_Fly sub-class of Insects List of components Domain Concept Fly_Body Domain Concept 2 Fly_Antenna Domain Concept name Fly_Body sub-part of White_Fly List of attributes ST_VisualConcept Shape[oval] Elongation [important] Color_VisualConcept Hue [white] Domain Concept name Fly_Antenna sub-part of White_Fly List of attributes ST_VisualConcept Shape [line] Thickness [thin] Color_VisualConcept Hue [white] Spatial_Relation Connected [Fly_Body] Spatial_Relation Right_of [Fly_Body] Sub-part

16/08/2004 Bonn 34 Cognitive Vision: Interpretation Reasoning Depth-first domain concept tree traversal Physical object hypotheses by building symbol grounding requests (visual object instance finding) Matching between visual object instances and predefined classes Classification refinement

16/08/2004 Bonn 35 Cognitive Vision: Data Management Goal: Matching between symbols and sensor data How: Data management, spatial reasoning, top down and bottom up strategies Symbol grounding or Anchoring: Anchoring = « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99]

16/08/2004 Bonn 36 Cognitive Vision: Data Management Reasoning Image processing request building according to visual object hypotheses (Object extraction criteria) Matching between image processing results and symbolic data (Verification and data management criteria) Instantiation and sending of visual objects to the Interpretation task Spatial Reasoning: multiple objects ( scene analysis criteria )

16/08/2004 Bonn 37 Cognitive Vision: Data Management Knowledge Formalization: Declarative knowledge: Data concepts : primitives (ridge, region, edge), descriptors (compacity, area, perimeter) Spatial relations : topology ( RCC8 ), distance and orientation Inferential knowledge : Object extraction criteria : to build image processing requests Object verification criteria : to diagnose the image processing results Scene analysis criteria: to manage multi-object hypotheses

16/08/2004 Bonn 38 Cognitive Vision: Image Processing Goal : Object extraction and numerical description How: Automate the configuration and execution of a library of programs for a given objective Use of program supervision techniques

16/08/2004 Bonn 39 Cognitive Vision: Image Processing Knowledge formalization: Declarative knowledge: Goals: image processing functionality (thresholding, edge extraction,…) Operators: knowledge to solve a given problem: primitive: particular program composite: particular combination of programs Requests: instantiations of goals on particular data, under particular constraints

16/08/2004 Bonn 40 Cognitive Vision: Image Processing Knowledge Formalization Primitive Operator Name Ridge_Linking Functionality Ridge Point Linking Input data in image_sequence Output data out image Parameters Thresh_first_dev Thresh_second_dev nb_sigma Preconditions in.coding.format = Calling syntax: PointLinking -s in out Composite Operator Name Ridge_Extraction Functionality Extraction of ridge linear features Input data in image Output data out image out_data ridge_feature_fil Parameters sigma nb_sigma Sub-components MulRidPointDetection, RidgeLinking Decomposition DO MulRidPointDetection THEN Ridge_Linking

16/08/2004 Bonn 41 Application on plant disease diagnosis Why Image Understanding ? Plant disease diagnosis = visual observation which aims at inferring disease presence by the observation of signs and symptoms TO BE ABLE TO REASON : signs and symptoms interpretation in terms of diseases TO BE ABLE TO SEE : Focusing on relevant criteria Star shape network of white and thin filaments (5-10 μ) Presence of elliptical white blobs in the centre of the network Climatic Context: High humidity, Temperature : 25 °C Early powdery mildew infection in propitious conditions Early diagnosis: Microscopic image (x64) of rose leaf part

16/08/2004 Bonn 42 Application on Plant Disease Diagnosis: Rose Diseases Powdery mildew : State of infection : early Vegetal support : red leaf Powdery mildew : State of infection : very early Vegetal support : green leaf Two white flies close to their eggs Need of domain knowledge Intelligent management of image processing programs Complexity and variability of object appearance Variability of contexts Scene knowledge and spatial reasoning Multiple objects and various object types

16/08/2004 Bonn 43 Leaf Healthy Non Healthy Insects Virus Fungi White fly Penicillium Powdery mildew Germinated tubes Filamentous Aphid Vegetal tissue Veins red green Subpart Subclass Acarid Ungerminated Pellets Application on plant disease diagnosis Domain knowledge base : the concept tree Mycelium: Part of : Fungi network of at least 2 connected Hyphae nb_hyphae = {unknown} Hyphae: Part of : Mycelium Geometry: line Thickness: thin, very thin Straightness:=almost straight Luminosity=bright...

16/08/2004 Bonn 44 Input : User Request Fungi infection? Image + Context Variety : Leonidas Leaf : young Season: summer Temp: 24° C Humidity: Application : early detection of plant diseases Interpretation Domain concept tree traversal to build visual object hypotheses Leaf Scene VegetalPart Disease Insects Virus Fungi White fly Penicillium Powdery mildew Dispersed Clump Aphid Subpart Subclass Acarid Very Early Pellet 1 Data Management Symbolic request to image processing request 3 Goal: segmentation Contraints:Image entity = ridge Object.width = [1..3] Object.intensity > 150 Input Data: Image : input image Mask : area of interest Image Processing Request 4 Image Processing: request solving by program supervision techniques 5 Image Data Ridge 1 numerical descriptors Ridge 3 + Numerical descriptors Ridge 2 + Numerical Descriptors 6 Interpretation Domain concept tree traversal to build visual object hypotheses Visual Object Hypothesis 2 Group of : Geometry: star shape network of { Geometry: line Thickness : thin width [7..10  m] very thin width [5..7  m] Straightness : almost straight Lightness: bright} Spatial Relation: Connected} 1

16/08/2004 Bonn 45 Application : early detection of plant diseases Image Data Ridge 1 numerical descriptors Ridge 3 + Numerical descriptors Ridge 2 + Numerical Descriptors Data Management Visual object hypothesis verification and instantiation 7 Interpretation : classification Matching between visual object instances and domain concepts 9 Interpretation : diagnosis Post classification rules activation 11 Visual Object Instance Network of lines Line 1 Line 2 Line 3 Line 5Line 4 EC Line line1 Thickness:=thin (0.8) Straightness:= straight (0.5) Lightness:=bright (0.7) Connected (line2) Connected (line4) + link to image data 8 Diagnosis Early powdery mildew infection on young leaf 12 Recognised domain concept 10 Freely dispersed mycelium

16/08/2004 Bonn 46 Conclusion A generic platform for automatic recognition of natural objects a formalism and an ontology for knowledge base building 3 dedicated reusable engines semantic interpretation image/symbol matching and spatial reasoning management of a generic image processing library Future works machine learning for image/symbol matching (Nicolas Maillot) and for image segmentation (Vincent Martin)

16/08/2004 Bonn 47 Conclusion Next Lectures Program Supervision (in more details) Video Understanding (temporal scenario recognition)

16/08/2004 Bonn 48 Thank you for your attention!