1. A Cognitive Vision Platform for Semantic Image Understanding Monique THONNAT and Celine HUDELOT Orion team INRIA Sophia Antipolis FRANCE

2. M. Thonnat 2 Introduction Cognitive Vision Platform Application to Plant Disease Recognition Conclusion Overview

3. M. Thonnat 3 Introduction: the Problem Problem: What does it mean to perform image understanding ? semantic image understanding (e.g. object classification) What does it mean to associate semantics to a particular image ?

4. M. Thonnat 4 Different interpretations of this image are possible: A light object on a dark background An astronomical object NGC4473 galaxy Introduction Image semantics is not inside the image Image interpretation depends on a priori knowledge

5. M. Thonnat 5 Introduction Focus: complex natural objects with existing taxonomy Proposed approach: Knowledge-based Vision formalize the a priori knowledge for image interpretation in knowledge bases explicit the reasoning (how to use a priori knowledge) for each subtask of image interpretation propose a platform reusable for different applications

6. M. Thonnat 6 Cognitive Vision Platform A platform performing 3 subtasks Semantic data interpretation Application expert knowledge (domain taxonomy and terminology) Visual data management Matching between numerical image data and symbols Scene analysis using spatial reasoning Image processing numerical object description program supervision techniques : to automate the management of an image processing library

7. M. Thonnat 7 Cognitive Vision Platform For each task: An application independent engine A conceptual model for the knowledge Two ontologies for the interoperability between the different components: Visual Concept Ontology: spatial, color and texture concepts Image Processing Ontology: image data and image processing functionality concepts

8. M. Thonnat 8 Cognitive Vision Platform

9. M. Thonnat 9 Cognitive Vision: Semantic Interpretation Goal: Find the semantic class of physical objects or situations observed on images How: Perform the interpretation in the same way experts do: Use a priori knowledge of application domain terminology and taxonomy Top down strategy

10. M. Thonnat 10 Cognitive Vision: Interpretation Knowledge Formalization: Declarative knowledge: Domain class: application concept (plant leaf, pollen grain) described by visual concepts (green color and oval shape or pink and circular) and subparts Domain class tree : hierarchy of domain classes Context: explicit representation of current domain context and acquisition context Domain request: request of an end user Representation by frames with slots

11. M. Thonnat 11 Cognitive Vision: Interpretation Knowledge Formalization Domain Class name White_Fly SuperClass Insect SubPart Description Domain Class Fly_Body Domain Class 2 Fly_Antenna Domain Class name Fly_Body Visual Description ST_VisualConcept Shape[oval] Elongation [important] Color_VisualConcept Hue [white] Domain Class name Fly_Antenna Visual Description ST_VisualConcept Shape [line] Thickness [thin] Color_VisualConcept Hue [white] Spatial_Relation Connected [Fly_Body] Spatial_Relation Right_of [Fly_Body] Sub-part

12. M. Thonnat 12 Cognitive Vision: Interpretation Knowledge Formalization: Inferential knowledge: Context criteria: describe decisions during the semantic interpretation Initialization interpretation criteria: information on how to initialize the problem using the context Post interpretation criteria: information to refine the interpretation results according to the context Implemented by rules Exemple of post interpretation criteria If Powdery Mildew detected and temperature < 25 C and Humidity > 80% then Alert “treatment is needed”

13. M. Thonnat 13 Cognitive Vision: Interpretation Reasoning Depth-first domain class tree traversal Visual object hypothesis propagation by building visual data management requests (visual object instance finding) Matching between visual object instances and predefined domain classes Classification refinement

14. M. Thonnat 14 Cognitive Vision: Visual Data Management Goal: Matching between symbols and sensor data How: Data management, spatial reasoning, top down and bottom up strategies Symbol grounding or Anchoring: Anchoring = « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99]

15. M. Thonnat 15 Cognitive Vision: Visual Data Management Knowledge Formalization: Declarative knowledge: Visual concepts (symbolic data): description of visual concepts and of their grounding relation with image descriptors Image data concepts (sensor data): primitives (ridge, region, edge), descriptors (area, eccentricity) Spatial relations : topology ( RCC8 ), distance and orientation Visual data management requests : express the visual data management problem Represented by frames with slots

16. M. Thonnat 16 Cognitive Vision: Data Management Knowledge Formalization: Inferential knowledge : Object extraction criteria : how to constrain image processing requests (using visual concepts and spatial relations) Spatial deduction criteria : how to infer spatial relations from another ones to diagnose the image processing results Visual evaluation criteria: how to diagnose image processing results Implemented by rules

17. M. Thonnat 17 Cognitive Vision: Data Management Knowledge Formalization: Example of object extraction criteria Let c a visual content context and O a visual object If O.geometry is an Open Curve and O.thickness is {Thin, Very Thin} then c.ImageEntityType:=Curvilinear Structure “Ridge or Valley” Example of spatial deduction criteria Let O1, O2, O3 three visual objects If NTTP(O1, O2) is true and Left_Of(O2,O3) is true then Left_Of(O1,O3) is true Example of visual evaluation criteria If mode is interactive then assess_data_by _user [correct under_segmentation over_segmentation noisy]

18. M. Thonnat 18 Cognitive Vision: Visual Data Management Reasoning Image processing request building according to visual object hypotheses (Object extraction criteria) Matching between image processing results and symbolic data Instantiation and sending of visual objects to the Interpretation task Spatial Reasoning: multiple objects ( spatial deduction criteria )

19. M. Thonnat 19 Cognitive Vision: Image Processing Goal : Object extraction and numerical description How: Use of program supervision techniques: Dynamic configuration and execution of a library of image processing programs (versus fixed procedure) Explicit formalization of expertise on how to use programs

20. M. Thonnat 20 Cognitive Vision: Image Processing Knowledge formalization: Declarative knowledge: Goals: image processing functionality (thresholding, edge extraction,…) Operators: knowledge to solve a given problem: primitive: particular program composite: particular combination of programs Program supervision requests: instantiations of goals on particular data, under particular constraints

21. M. Thonnat 21 Cognitive Vision: Image Processing Knowledge Formalization Primitive Operator Name Recursive_Gaussian_Derivation Input data Image name input Output data Image name mfxx Image name mfyy Image name mfxy Parameters sigma default 1.0 Preconditions valid input Postconditions valid mfxx, valid mfxy Initialization Criteria Rule init-sigma Let c a visual content context If true Then sigma := c.objectwidth/sqrt(3) Calling syntax: Gaussian -sigma input mfxx mfyy mfxy Composite Operator Name Ridge_Extraction Functionality Object Extraction Input data image input_image Output data image segmented_image Preconditions valid input Postconditions valid mfxx Body “sequential decomposition” Recursive_Gaussian_derivation – Steger_Detector- Ridge-Filtering Distribution Ridge_Extraction.input_image / Recursive_Gaussian_Derivation.input … Flow Recursive_Gaussian_Derivation.mfxx / Steger_detector.mfxx

22. M. Thonnat 22 Cognitive Vision: Image Processing Reasoning: Planning techniques (HTN) Program selection in a library of programs Selected programs execution Evaluation and adjustment if needed Program Supervision Engine Library of programs Program Utilisation KB PlanningExecution EvaluationRepair results plan (part of) judgements Actions to correct 1 2 3 4 56 7 correct incorrect Request + data

23. M. Thonnat 23 Cognitive Vision Platform

24. M. Thonnat 24 Application on plant disease diagnosis Why Image Understanding ? Plant disease diagnosis = visual observation which aims at inferring disease presence by the observation of signs and symptoms TO BE ABLE TO REASON : signs and symptoms interpretation in terms of diseases TO BE ABLE TO SEE : Focusing on relevant criteria Star shape network of white and thin filaments (5-10 μ) Presence of elliptical white blobs in the centre of the network Climatic Context: High humidity, Temperature : 25 °C Early powdery mildew infection in propitious conditions Early diagnosis: Microscopic image (x64) of rose leaf part

25. M. Thonnat 25 Application on Plant Disease Diagnosis: Rose Diseases Powdery mildew : State of infection : early Vegetal support : red leaf Powdery mildew : State of infection : very early Vegetal support : green leaf Two white flies close to their eggs Need of domain knowledge Intelligent management of image processing programs Complexity and variability of object appearance Variability of contexts Scene knowledge and spatial reasoning Multiple objects and various object types

26. M. Thonnat 26 Leaf Healthy Non Healthy Insects Virus Fungi White fly Penicillium Powdery mildew Germinated tubes Filamentous Aphid Vegetal tissue Veins red green Subpart Subclass Acarid Ungerminated Pellets Application on plant disease diagnosis Domain knowledge base : the class tree Mycelium: Part of : Fungi network of at least 2 connected Hyphae nb_hyphae = {unknown} Hyphae: Part of : Mycelium Geometry: line Thickness: thin, very thin Straightness:=almost straight Luminosity=bright...

27. M. Thonnat 27 Input : User Request Fungi infection? Image + Context Variety : Leonidas Leaf : young Season: summer Temp: 24° C Humidity: 80... Application : early detection of plant diseases Interpretation Domain concept tree traversal to build visual object hypotheses Leaf Scene VegetalPart Disease Insects Virus Fungi White fly Penicillium Powdery mildew Dispersed Clump Aphid Subpart Subclass Acarid Very Early Pellet 1 Data Management Symbolic request to image processing request 3 Goal: segmentation Contraints:Image entity = ridge Object.width = [1..3] Object.intensity > 150 Input Data: Image : input image Mask : area of interest Image Processing Request 4 Image Processing: request solving by program supervision techniques 5 Image Data Ridge 1 numerical descriptors Ridge 3 + Numerical descriptors Ridge 2 + Numerical Descriptors 6 Interpretation Domain concept tree traversal to build visual object hypotheses Visual Object Hypothesis 2 Group of : Geometry: star shape network of { Geometry: line Thickness : thin width [7..10  m] very thin width [5..7  m] Straightness : almost straight Lightness: bright} Spatial Relation: Connected} 1

28. M. Thonnat 28 Application : early detection of plant diseases Image Data Ridge 1 numerical descriptors Ridge 3 + Numerical descriptors Ridge 2 + Numerical Descriptors Data Management Visual object hypothesis verification and instantiation 7 Interpretation : classification Matching between visual object instances and domain concepts 9 Interpretation : diagnosis Post classification rules activation 11 Visual Object Instance Network of lines Line 1 Line 2 Line 3 Line 5Line 4 EC Line line1 Thickness:=thin (0.8) Straightness:= straight (0.5) Lightness:=bright (0.7) Connected (line2) Connected (line4) + link to image data 8 Diagnosis Early powdery mildew infection on young leaf 12 Recognised domain concept 10 Freely dispersed mycelium

29. M. Thonnat 29 Conclusion A platform for automatic recognition of natural objects Ontology-based formalism for knowledge acquisition 3 dedicated reusable engines for semantic interpretation visual data management program supervision for image processing Future works integrate results on : machine learning for visual concept detection (Nicolas Maillot) machine learning for image segmentation (Vincent Martin)