2. Leveraging Semantic Web techniques to gain situational awareness Can Semantic Web techniques empower perception and comprehension in Cyber Situational Awareness? Talk at Cyber Situational Awareness Workshop, Fairfax, VA Nov 14-15, 2007. Amit Sheth LexisNexis Ohio Eminent Scholar Kno.e.sis Center Wright State University http://knoesis.wright.edu Thanks: Cory Henson and Sensor Data Management team (M. Perry, S. Sahoo)

3. Outline 1.Situational Awareness (SA) 2.SA within the Semantic Web Situation Awareness (SAW) Ontology Sensor Web Enablement Provenance Context Spatial-Temporal-Thematic Analysis

4. Situation Awareness “Situation awareness is the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.” (1988, Mica Endsley). http://en.wikipedia.org/wiki/Situation_awareness

5. JDL: Data Fusion Model A. Steinberg, et al., Rethinking the JDL Data Fusion Levels

6. JDL: Data Fusion Model M. Kokar, Ontology Based High Level Fusion and Situation Awareness: Methods and Tools Level 0: Signal/Feature Assessment: pixels, signals Level 1: Entity Assessment: object-event identification and tracking Level 2: Situation Assessment: relational analysis of objects-events Level 3: Impact Assessment: threat intent estimation and consequence prediction Level 4: Performance Assessment: resource management, adaptive search and processing

7. Endsley’s Model M. Kokar, et al., Ontology-based Situation Awareness

8. Perception involves monitoring and simple recognition produces Level 1 SA, an awareness of multiple situational elements (objects, events, people, systems, environmental factors) and their current states (locations, conditions, modes, actions). Comprehension involves pattern recognition, interpretation and evaluation produces Level 2 SA, an understanding of the overall meaning of the perceived elements - how they fit together as a whole, what kind of situation it is, what it means in terms of one's mission goals. Projection involves anticipation and mental simulation produces Level 3 SA, an awareness of the likely evolution of the situation, its possible/probable future states and events. This is the highest level of SA. http://en.wikipedia.org/wiki/Situation_awareness Endsley’s Model

9. Collect Relevant Data Provenance Relate Situation Entities Semantic Analysis thematic Spatio-Temporal trust M. Kokar, et al., Ontology-based Situation Awareness* (Modified Figure) Endsley’s Model w/ Semantics Identify Situation Entities

10. Situation Awareness Data Pyramid Sensor Data (World) Entity Metadata (Perception) Relationship Metadata (Comprehension) Expressiveness Data Information Semantics/Understanding /Insight Data Pyramid

11. Situation Awareness Situation Awareness Components Physical World: Sensor Data Perception: Entity Metadata Comprehension: Relationship Metadata Semantic Analysis How is the data represented? Sensor Web Enablement What are the antecedents of the event? Provenance Analysis Where did the event occur? Spatial Analysis When did the event occur? Temporal Analysis What is the significance of the event? Thematic Analysis

12. Sensor Web Enablement

13. OGC Mission To lead in the development, promotion and harmonization of open spatial standards Open Geospatial Consortium Consortium of 330+ companies, government agencies, and academic institutes Open Standards development by consensus process Interoperability Programs provide end-to-end implementation and testing before spec approval Standard encodings, e.g. –GeographyML, SensorML, Observations & Measurements, TransducerML, etc. Standard Web Service interfaces, e.g. –Web Map Service –Web Feature Service –Web Coverage Service –Catalog Service –Sensor Web Enablement Services (Sensor Observation Service, Sensor Alert Service, Sensor Process Service, etc.) http://www.opengeospatial.org/projects/groups/sensorweb

14. Network Services Vast set of users and applications Constellations of heterogeneous sensors Weather Chemical Detectors Biological Detectors Sea State Surveillance Airborne Satellite Distributed self-describing sensors and related services Link sensors to network and network- centric services Common XML encodings, information models, and metadata for sensors and observations Access observation data for value added processing and decision support applications Users on exploitation workstations, web browsers, and mobile devices Sensor Web Enablement http://www.opengeospatial.org/projects/groups/sensorweb

15. GeographyML (GML) TransducerML (TML) Observations & Measurements (O&M) Information Model for Observations and Sensing Sensor and Processing Description Language Multiplexed, Real Time Streaming Protocol Common Model for Geography Systems and Features SensorML (SML) Sam Bacharach, “GML by OGC to AIXM 5 UGM,” OGC, Feb. 27, 2007. SWE Languages and Encodings

16. Catalog Service SOSSASSPS Clients Access Sensor Description and Data Command and Task Sensor Systems Dispatch Sensor Alerts to registered Users Discover Services, Sensors, Providers, Data Accessible from various types of clients from PDAs and Cell Phones to high end Workstations Sam Bacharach, “GML by OGC to AIXM 5 UGM,” OGC, Feb. 27, 2007. SWE Components – Web Services

17. 17 Semantic Sensor ML – Adding Ontological Metadata Person Company Coordinates Coordinate System Spatial Ontology Mike Botts, "SensorML and Sensor Web Enablement," Earth System Science Center, UAB Huntsville Domain Ontology Event Situation Situation Awareness Ontology Time Units Timezone Temporal Ontology

18. Situation Awareness Ontology

19. Ontology What is an Ontology? “Ontology is about the exact description of things and their relationships.” World Wide Web Consortium (W3C)

20. Core SAW Ontology Provides a framework from which to build ontologies for arbitrary situations Represents objects and relationships as well as their evolution over time Captures sufficient information about a situation to support high-level reasoning Economical design permits its implementation in a working system Situation Awareness Ontology C. Matheus, et al., A Core Ontology for Situation Awareness

21. Situation Awareness Ontology C. Matheus, et al., An Application of Semantic Technologies to Situation Awareness

22. Provenance Context

23. Provenance What is Provenance? The recording of details in a data process workflow Trace back to where the particular data entity originated The phenomena captured by the sensor The sensor characteristics associated with data What processing was done on data Enables effective interpretation of object or event - Trust Evaluate whether particular data entity is relevant in current situation based on its provenance Enhanced situation comparison through use of provenance

24. Semantic Provenance Context for Situation Awareness Use of provenance associated with data to evaluate if situation awareness is correct Situation analysis utilizes Provenance Context Provenance Context components Type of Data Type of Process Type of Agent (e.g., Sensor) Provenance Context components defined in Ontology Well defined formal semantics Machine processable Scalable to large data sets

25. Spatial, Temporal, Thematic Analysis

26. North Korea detonates nuclear device on October 9, 2006 near Kilchu, North Korea Thematic Dimension: What Temporal Dimension: When Spatial Dimension: Where Three Dimensions of Information

27. Where we are, where we need to go Semantic Analytics Searching, analyzing and visualizing semantically meaningful connections between named entities Significant progress with thematic data Semantic associations (Rho-Operator) Subgraph discovery Query languages (SPARQ2L, SPARQLeR) Data stores (Brahms) Spatial and Temporal data is critical in many analytical domains Need to support spatial and temporal data and relationships

28. Current Research Towards STT Relationship Analysis Modeling Spatial and Temporal data using SW standards (RDF(S)) 1 –Upper-level ontology integrating thematic and spatial dimensions –Use Temporal RDF 3 to encode temporal properties of relationships –Demonstrate expressiveness with various query operators built upon thematic contexts Graph Pattern queries over spatial and temporal RDF data 2 –Extended ORDBMS to store and query spatial and temporal RDF –User-defined functions for graph pattern queries involving spatial variables and spatial and temporal predicates –Implementation of temporal RDFS inferencing 1.Matthew Perry, Farshad Hakimpour, Amit Sheth. "Analyzing Theme, Space and Time: An Ontology-based Approach", Fourteenth International Symposium on Advances in Geographic Information Systems (ACM-GIS '06), Arlington, VA, November 10 - 11, 2006 2.Matthew Perry, Amit Sheth, Farshad Hakimpour, Prateek Jain. “Supporting Complex Thematic, Spatial and Temporal Queries over Semantic Web Data", Second International Conference on Geospatial Semantics (GeoS ‘07), Mexico City, MX, November 29 – 30, 2007 3.Claudio Gutiérrez, Carlos A. Hurtado, Alejandro A. Vaisman. “Temporal RDF”, ESWC 2005: 93-107

29. Upper-level Ontology modeling Theme and Space Occurrent Continuant Named_Place Spatial_Occurrent Dynamic_Entity Spatial_Region Occurrent: Events – happen and then don’t exist Continuant: Concrete and Abstract Entities – persist over time Named_Place: Those entities with static spatial behavior (e.g. building) Dynamic_Entity: Those entities with dynamic spatial behavior (e.g. person) Spatial_Occurrent: Events with concrete spatial locations (e.g. a speech) Spatial_Region: Records exact spatial location (geometry objects, coordinate system info) occurred_at located_at occurred_at: Links Spatial_Occurents to their geographic locations located_at: Links Named_Places to their geographic locations rdfs:subClassOf property

30. Occurrent Continuant Named_Place Spatial_Occurrent Dynamic_Entity Person City Politician Soldier Military_Unit Battle Vehicle Bombing Speech Military_Event assigned_to on_crew_of used_in gives participates_in trains_at Spatial_Region located_atoccurred_at Upper-level Ontology Domain Ontology rdfs:subClassOf used for integration rdfs:subClassOf relationship type dynamic entities get spatial properties indirectly through relationships with spatial entities

31. Sample STT Query Scenario (Biochemical Threat Detection): Analysts must examine soldiers’ symptoms to detect possible biochemical attack Query specifies (1)a relationship between a soldier, a chemical agent and a battle location (graph pattern 1) (2)a relationship between members of an enemy organization and their known locations (graph pattern 2) (3)a spatial filtering condition based on the proximity of the soldier and the enemy group in this context (spatial Constraint)

32. Using SW to enable perception and comprehension Perception Leveraging current research in sensor data representation found in the Sensor Web Enablement metadata languages Using SWE languages to model sensors, processes, and data Comprehension Extending the Sensor Web Enablement languages with semantic metadata to provide the ability to model relationships between entities Semantic relationships provide “meaning” to objects and events within a situation Using Situational Awareness Ontology to model situations and provide a framework for Semantic Analysis Provenance Context provides a historical record of relevant objects and events within a situation Spatial, Temporal and Thematic analysis provides the “where”, “when”, and “what” of objects and events within a situation Utilizing Semantic Web technologies to enable perception and comprehension within Situational Awareness

33. C. Matheus, M. Kokar and K. Baclawski, A Core Ontology for Situation Awareness, Sixth International Conference on Information Fusion, pp.545-552, Cairns, Australia, July 2003 C. Matheus, M. Kokar, K. Baclawski and J. Letkowski, An Application of Semantic Web Technologies to Situation Awareness, 4 th International Semantic Web Conference, ISWC 2005, Galway, Ireland, November, 2005 M. Kokar, C. Matheus and K. Baclawski, Ontology-based situation awareness, Informat. Fusion, 2007, doi:10.1016/j.inffus.2007.01.004 M. Kokar, Ontology Based High Level Fusion and Situation Awareness: Methods and Tools, Presentation, Quebec, 2007 A. Steinberg and C. Bowman, Rethinking the JDL data fusion levels, National Symposium on Sensor and Data Fusion, 2004 Wikipedia, Situation Awareness, http://en.wikipedia.org/wiki/Situation_awarenesshttp://en.wikipedia.org/wiki/Situation_awareness Open Geospatial Consortium, Sensor Web Enablement WG, http://www.opengeospatial.org/projects/groups/sensorweb Sam Bacharach, “GML by OGC to AIXM 5 UGM,” OGC, Feb. 27, 2007. References