An Ontology framework for Knowledge-Assisted Semantic Video Analysis and Annotation Centre for Research and Technology Hellas/ Informatics and Telematics Institute S.Dasiopoulou, V.K.Papastathis, Y.Kompatsiaris, M.G.Strintzis

Informatics and Telematics Institute Centre for Research and Technology Hellas 2 Importance of Extracting Multimedia Semantics Vast amount of multimedia resources  sophisticated content management Inherent multimedia content ambiguity QBE, keyword-based: inadequate Manual annotation: impractical, expensive Emergent applications: filtering, summarization, semantic transcoding, personalized content delivery

Informatics and Telematics Institute Centre for Research and Technology Hellas 3 Image/Video analysis and retrieval Semantic gap: automatically extracted features vs. human interpretation Mapping low-level features (e.g color) to high-level (perceptual) concepts: 1. Learning approaches 2. Knowledge-based approaches

Informatics and Telematics Institute Centre for Research and Technology Hellas 4 Learning approaches - SoA Support Vector Machines (SVMs) Neural networks (+fuzzy logic) Probabilistic methods: HMMs, BBNs Case-based reasoning Applications: face detection, human recognition, sports domain, visual surveillance

Informatics and Telematics Institute Centre for Research and Technology Hellas 5 Knowledge Assisted approaches - SoA Modeling of a priori domain specific knowledge+rules Ontology modeling [Yoshitaka94], [Benitez00], [Hunter04], [Jaimes03], [Meghini97], [Troncy03], [Hyvonen02] etc.

Informatics and Telematics Institute Centre for Research and Technology Hellas 6 System overview Two ontologies: 1. multimedia analysis ontology 2. domain-specific ontology Ontologies: shared models of formal conceptualizations, inference enabling Logic rules: algorithm determination, object detection auxiliary, event definition/detection

Informatics and Telematics Institute Centre for Research and Technology Hellas 7 Proposed Approach Objectives Capturing video semantics: 1. Object detection 2. Event detection 3. Content semantic description/annotation Required Knowledge: 1. Object modeling (visual+spatial) 2. Event modeling (spatial+temporal) 3. Appropriate processing algorithms domain specific

Informatics and Telematics Institute Centre for Research and Technology Hellas 8 System Architecture

Informatics and Telematics Institute Centre for Research and Technology Hellas 9 Current state Simple, RDFS (OntoEdit) domain ontologies 1. simple spatial relations (position constraints) 2. events are not included Rules associated only with algorithmic issues (F- Logic rules, OntoBroker) Unifying model: domain knowledge and analysis process are both ontological concepts Tested on Formula One and Football domains

Informatics and Telematics Institute Centre for Research and Technology Hellas 10 Multimedia analysis ontology

Informatics and Telematics Institute Centre for Research and Technology Hellas 11 Formula One ontology

Informatics and Telematics Institute Centre for Research and Technology Hellas 12 Analysis Process Initial color clustering Generation of color-homogeneous, connected regions mask Color model selection using EMD Generation of motion-homogeneous regions mask Additional constraints (size, motion, spatial relations)

Informatics and Telematics Institute Centre for Research and Technology Hellas 13 Rules construction Rules to determine whether an algorithm comprises a detection step of an object + execution order Rules to determine each algorithm’s input parameters Rules to determine which objects have to be detected first, to proceed with a particular object’s detection

Informatics and Telematics Institute Centre for Research and Technology Hellas 14 Experimental results I

Informatics and Telematics Institute Centre for Research and Technology Hellas 15 Experimental results II

Informatics and Telematics Institute Centre for Research and Technology Hellas 16 Future Work – Considerations Acquisition of more accurate/detailed models Definition/exploitation of more complex spatial relations Inclusion of temporal relations Allen’s interval algebra (spatial+temporal) Event detection