1. A practical approach to the development of ontology-based information fusion systems Juan Gómez-Romero, Miguel A. Serrano, Jesús García, Miguel Á. Patricio, José M. Molina Applied Artificial Intelligence Group University Carlos III of Madrid, Spain

2. Summary Ontologies –What are ontologies? –Why should we care? –How can they be exploited? –Are there any successful experience? –How can we contribute?

3. Outline Motivation Knowledge representation and reasoning with ontologies Ontologies for HLIF in the maritime domain Proposed architecture Implementation Discussion, conclusions and future work

4. Outline > Motivation Knowledge representation and reasoning with ontologies Ontologies for HLIF in the maritime domain Proposed architecture Implementation Discussion, conclusions and future work

5. 1. Motivation Information Fusion –"theories and methods to effectively combine data from multiple sensors and related information to achieve more specific inferences that could be achieved by using a single, independent sensor." (Llinas and Hall, 2009)

6. 1. Motivation Low-level data fusion –To process multi-sensor signals to estimate objects properties Tracking: data acquisition, collection, spatial and temporal alignment Video-based tracking: estimate the kinetics of scene objects in a video sequence –Surveillance

7. 1. Motivation High-level information fusion –To obtain a symbolic description of the qualitative relations between the objects in the scenario Actions, intentions, threats, etc. > Situation assessment

8. 1. Motivation High-level information fusion –Understand the scene, evaluate threats, support decision making –Purely numerical techniques are insufficient Cognitive abilities Complex and unpredictable world behavior

9. 1. Motivation High-level information fusion –Flexible and dynamic situation models –Context exploitation –Symbolic formalisms to represent and reason with high-level information: abstract scene objects and relations

10. Outline Motivation > Knowledge representation and reasoning with ontologies Ontologies for HLIF in the maritime domain Proposed architecture Implementation Discussion, conclusions and future work

11. 2. Knowledge representation and reasoning with ontologies Ontologies –Knowledge model that describes the objects in a domain by means of a language that can be automatically processed Description Logics (DLs) representation Proposed to be the language for metadata representation in the Semantic Web

12. 2. Knowledge representation and reasoning with ontologies Advantages –Interpretability High level symbolic description –Interoperability Agreed representation of fusion information –Scalability Promote extension and reuse –Formal Reasoning with logic-based formalisms –Tools OWL standard, reasoning engines, programming interfaces, …

13. 2. Knowledge representation and reasoning with ontologies Representation primitives –Concepts Vessel, HarborZone –Relations hasFlag, insideOf –Instances vessel_1, nearShoreZone –Axioms Vessel and (hasFlag some (Flag and (belongsTo some AlliedCountry)) subclassOf FriendVessel transitive(insideOf) (vessel_1, nearShoreZone: insideOf)

14. 2. Knowledge representation and reasoning with ontologies

15. 2. Knowledge representation and reasoning with ontologies Reasoning –Satisfiability / consistency An axiom is satisfiable if it is not a contradiction to the remaining axioms –Subsumption A (super-) concept includes a (sub-) concept –Equivalence Two concepts include the same instances –Disjointness Two concepts do not have any common instance –Instance checking An instance belongs to a class

16. Outline Motivation Knowledge representation and reasoning with ontologies > Ontologies for HLIF in the maritime domain Proposed architecture Implementation Discussion, conclusions and future work

17. 3. Ontologies for HLIF in the Maritime Domain Situation and threat assessment in the harbor surveillance scenario –Detected and estimated vessel information from VTS Position, AIS identification –Context knowledge Restrictions to the fusion process –Is the situation plausible? Enrich available information –Link to external information sources –Normalcy models Harbor navigation restrictions Expert knowledge about threats

18. 3. Ontologies for HLIF in the Maritime Domain Knowledge representation –Terminological knowledge base to describe harbor elements, actors and context Concepts, relations, axioms (GCIs) –Geometrical elements of harbor –Vessel classification –Rules of operation –Assertional knowledge base to represent current instances of the harbor entities and relevant contextual information Instance axioms –Static –Dynamic

19. 3. Ontologies for HLIF in the Maritime Domain Terminological Knowledge Base: Harbor areas

20. 3. Ontologies for HLIF in the Maritime Domain Qualitative spatial knowledge management –Zone boundaries and vessel position are represented according to their relative situation –RCC (Region Connection Calculus) Logic theory for qualitative spatial knowledge representation and reasoning –disconnected, externally connected, partial overlap, tangential inner part, etc. Cannot be fully represented with ontologies Supported by reasoning engines

21. 3. Ontologies for HLIF in the Maritime Domain Assertional knowledge base (factual) –Individuals representing current instantiation of the terminological model Scene interpretation is a model-construction procedure Instances representing more abstract entities are inferred from instances representing concrete measures > From sensor-based data to situation assessment

22. 3. Ontologies for HLIF in the Maritime Domain Individual: vessel1 Types: PowerDrivenVessel Facts: inside_of middle_harbour, has_property vessel_size Individual: vessel1_size Types: Size Facts: size vessel_size_value Individual: vessel_size_value Types: AbsoluteFloatValue Facts: val 25.0f

23. 3. Ontologies for HLIF in the Maritime Domain Rules of operation Any vessel inside an area with restricted speed moving at a speed under the speed limit is normal –Classes describing the normal behavior –Instance checking can be used to classify a vessel as threatening or non-threatening according to the “normal behavior” classes

24. 3. Ontologies for HLIF in the Maritime Domain

25. Rules of operation –Vessels that do not accomplish any normalcy rule are not classified as non-threatening It is easy to describe normal scenarios according to harbor rules Better supported by ontologies that abnormalcy models –Open World Assumption: the knowledge in an ontology is incomplete »Default reasoning is not performed Any entity not classified as normal remains in an unknown state

26. 3. Ontologies for HLIF in the Maritime Domain Individual: vessel1 Types: PowerDrivenVessel Facts: inside_of middle_harbour, has_property vessel_size has_property vessel_speed Individual: vessel1_speed Types: Speed Facts: speed vessel_speed_value Individual: vessel_speed_value Types: AbsoluteFloatValue Facts: val 4.0f Individual: vessel1 Types: PowerDrivenVessel, NoSpeedViolation, SafeVessel Facts: inside_of middle_harbour, has_property vessel_size has_property vessel_speed Individual: vessel1_speed Types: Speed Facts: speed vessel_speed_value Individual: vessel_speed_value Types: AbsoluteFloatValue Facts: val 4.0f reasoner

27. 3. Ontologies for HLIF in the Maritime Domain Individual: vessel1 Types: PowerDrivenVessel Facts: inside_of middle_harbour, has_property vessel_size has_property vessel_speed Individual: vessel1_speed Types: Speed Facts: speed vessel_speed_value Individual: vessel_speed_value Types: AbsoluteFloatValue Facts: val 6.0f Individual: vessel1 Types: PowerDrivenVessel, owl:Thing Facts: inside_of middle_harbour, has_property vessel_size has_property vessel_speed Individual: vessel1_speed Types: Speed Facts: speed vessel_speed_value Individual: vessel_speed_value Types: AbsoluteFloatValue Facts: val 6.0f reasoner Trigger abductive reasoning

28. 3. Ontologies for HLIF in the Maritime Domain Abductive reasoning –Takes a set of facts as inputs and finds a suitable hypothesis that explains them See whether inconsistency is result of low-quality observations, or this vessel exhibits a possible threatening behavior –Increase threat level –Not directly supported by ontologies Monotonic formalisms –do not allow adding or retracting knowledge while reasoning –Reasoning engines include extensions to allow abductive rules (not uncertain)

29. 3. Ontologies for HLIF in the Maritime Domain Reasoning under uncertainty –Additional reasoning layer BAS (Belief Argumentation System) –Combination of symbolic logic and belief theory »Compute beliefs supporting or rejecting a hypothesis (e.g., vessel features or spatio- temporal relations with other vessels and zones) Probabilistic ontologies –Reduction to Bayesian inference

30. Outline Motivation Knowledge representation and reasoning with ontologies Ontologies for HLIF in the maritime domain > Proposed architecture Implementation Discussion, conclusions and future work

31. 4. Architecture

32. Outline Motivation Knowledge representation and reasoning with ontologies Ontologies for HLIF in the maritime domain Proposed architecture > Implementation Discussion, conclusions and future work

33. 5. Implementation Two layers –General tracking layer Numerical measures (Desnsity functions, movement vectors) –Ontology-based contextual layer Tracking representation Contextual data Symbolic reasoning

34. 5. Implementation Tracking layer –Four modules that run in sequence –Each module has a set of interchangable algorithms –Input: Frames –Output: Track and features

35. 5. Implementation Association module –Fuzzy Region Assignment (FRA) Low level granularity, image segmentation operations Medium level granularity, smoothness criteria on target features High level granularity, constraints on tracking continuity

36. 5. Implementation –Fuzzy Region Assignment (FRA) Bayesian formulation to determine when a blob is related with a track Heuristic function to update the track situation and dimensions Fuzzy rules derived from experimentation to define the final group

37. 5. Implementation Context layer –Set of ontologies organized according to the JDL model Tracking entities (TREN) Scene object (SCOB) e.g. Restricted area, Person, Vessel Activities (ACTV) e.g. Threatening behaviours Situation assessment (IMPC) e.g Emergency –Inputs managed through the OWL 2 API Context knowledge given by users or previous executions Sensor data (Video tracking) –Ontologies are instanced in the RACER reasoner Abductive nRQL rules and independent RCC implementation –Scene interpretation and recommendations as output

38. 5. Implementation Context layer and mass storage –Timestamps / snapshots allow temporal dimension “A vessel stopped in a restricted area during the last ten intervals” –Delays in the overall system Query search through a larger number of axioms –Compromise between data storage and query performance –Temporal window

39. 5. Implementation Scalability - Dynamic RCC –Aims Representation and reasoning with qualitative spatial properties Efficient update of the spatial properties of the objects –Architecture Knowledge base (SCOB spatial features) Optimized geometric model: Geometric model (JTS with OpenGIS) and a data structure RCC implementation to store the qualitative spatial relationships (RACER substrate)

40. 5. Implementation What is the problem? –A full topological analysis has a quadratic complexity –Choose only candidate geometries that could modify the spatial relations How? –Querying the auxiliary data structure Advantages –Topological relations of a geometry is obtained by checking only a few geometries

41. 5. Implementation Video examples –Scene annotation –Object identification –Tracking enhancement –Scene recognition http://www.giaa.inf.uc3m.es/miembros/jgomez/et/

42. Outline Motivation Knowledge representation and reasoning with ontologies Ontologies for HLIF in the maritime domain Proposed architecture Implementation > Discussion, conclusions and future work

43. 6. Discussion, conclusions and future work Ontologies for high-level fusion –Cognitive model Symbolic representation of the world –Formal knowledge model Representation Reasoning Ontologies in the maritime domain –Heterogeneous information Vessel Traffic Systems, AIS Security protocols Harbor areas and navigation restrictions

44. 6. Discussion, conclusions and future work Ontological model –Categorization of interesting entities Vessels (Temporal properties) Harbor areas (Spatial features) –Normalcy models Normal categories of behaviors –Abduction and uncertainty management Extended rule-based reasoning Belief-based argumentation (BAS), Bayesian networks

45. 6. Discussion, conclusions and future work Architecture and implementation –Low-level fusion layer Tracker –High-level fusion layer Cognitive scene model –Reasoner –Spatial module Video-surveillance applications

46. 6. Discussion, conclusions and future work Future work –Practical implementation in real domains (harbor!) Multiple information sources –Acquisition –Integration Expert knowledge –Acquisition –Representation Real-time demands High reliability –Specific features Uncertain and imprecise knowledge Interfacing with human users

47. Questions, comments? Juan Gómez-Romero, Miguel A. Serrano, Jesús García, Miguel Á. Patricio, José M. Molina Applied Artificial Intelligence Group University Carlos III of Madrid, Spain