Informatics and Telematics Institute Centre for Research and Technology Hellas ITI-CERTH Amsterdam, Multimedia Semantics XG, July 2006 Vasileios Papastathis, Research Fellow Centre for Research and Technology Hellas (CERTH) Informatics and Telematics Institute (ITI) ( Director Prof. Michael Strintzis Multimedia Knowledge Group ( Head Dr. Yiannis Kompatsiaris Thessaloniki, GREECE

Informatics and Telematics Institute Centre for Research and Technology Hellas 2 Presentation Overview ITI Multimedia Knowledge Group Towards a Common Multimedia Ontology Framework IST BOEMIE Project

Informatics and Telematics Institute Centre for Research and Technology Hellas 3 ITI Multimedia Knowledge Group Semantic Multimedia Analysis Multimedia Indexing and Retrieval Multimedia and the Semantic Web Knowledge Structures, Languages and Tools for Multimedia Reasoning and Personalization for Multimedia Applications Ontology structures in OWL for knowledge-assisted multimedia analysis and reasoning Rules in SWRL including multimedia features for analysis and reasoning MPEG-7 Visual part representation in RDF Graphical tool for ontology-based annotation of images (M- Ontomat)

Informatics and Telematics Institute Centre for Research and Technology Hellas 4 R&D Projects “aceMedia: Integrating knowledge, semantics and content for user centred intelligent media services”, IP “KnowledgeWeb: Realizing the Semantic Web”, funded by the DG XIII, NoE “MESH: Multimedia Semantic Syndication for Enhanced News Services”, IST – IP, “K-Space: Knowledge Space of Semantic Inference for Automatic Annotation and Retrieval of Multimedia Content, IST - NoE, “X-Media: Knowledge Sharing and Reuse Across Media”, IST – IP, “BOEMIE: Bootstrapping Ontology Evolution with Multimedia Information Extraction”, IST – STREP, “PATExpert: Advanced Patent Document Processing Techniques”, IST-STREP, (collaboration with AUTH)

Informatics and Telematics Institute Centre for Research and Technology Hellas Towards a common Multimedia Ontology Framework Requirements and Harmonization approaches

Informatics and Telematics Institute Centre for Research and Technology Hellas 6 Multimedia Ontology Framework Framework for collecting existing multimedia knowledge representation Define approach for harmonization Initiated by aceMedia in EWIMT 2005 but not directly connected to aceMedia work /multimedia_ontology/index.html

Informatics and Telematics Institute Centre for Research and Technology Hellas 7 List of contributors 1. National Technical University of Athens 2. Istituto per la Matematica Applicata e le Tecnologie Informatiche, Consiglio Nazionale delle Ricerche, 3. University Rennes 4. Centre for Digital Music, Queen Mary, University of London 5. Lehrgebiet Multimedia und Internetanwendungen, Informatikzentrum 6. University of Southampton, UK 7. Institute of Informatics & Telecommunications, N.C.S.R. "Demokritos" 8. Institute of Information Systems & Information Management, JOANNEUM RESEARCH Forschungsgesellschaft mbH

Informatics and Telematics Institute Centre for Research and Technology Hellas 8 List of contributors (2) 9. ISWeb, University of Koblenz 10. Freie Universitaet,Netzbasierte Informationssysteme, Institut fuer Informatik 11. 'Signals and Images' Laboratory, Institute of Information Science and Technologies "A. Faedo" (ISTI–CNR), Italian National Research Council (CNR) 12. Image Bioinformatics Research Group, Department of Zoology, University of Oxford 13. Technical University of Crete, 14. CWI Amsterdam, Maryland Information and Network Dynamics Lab, National Technical University of Athens 15. University of Twente

Informatics and Telematics Institute Centre for Research and Technology Hellas 9 Requirements for MM ontologies Scope and Usage Media Description-Description of the information object Representation of Content Structure Representation of Content Description Knowledge Representation Formalism Reasoning Support Following an established methodology

Informatics and Telematics Institute Centre for Research and Technology Hellas 10 Requirements (2) Scope and Usage Annotation Analysis Retrieval Reasoning Personalized filtering Meta-Modeling Media description Image Video 3D Graphics Audio AudioVisual Multimedia presentation

Informatics and Telematics Institute Centre for Research and Technology Hellas 11 Requirements (3) Representation of content structure Multimedia structure representation Spatiotemporal relation representation Localization Representation of content description Low-level description Multiple audiovisual descriptors Alignment to existing metadata standards Basic and complex data types

Informatics and Telematics Institute Centre for Research and Technology Hellas 12 Requirements (4) Knowledge representation formalism High complexity of multimedia modeling Compliance to W3C standards Inferencing support OWL Reasoning support DL-based reasoning Rules Uncertainty Fuzziness Established methodology OntoKnowledge OntoClean

Informatics and Telematics Institute Centre for Research and Technology Hellas 13 Harmonization approaches (1) Modularization Clear, independent ontology modules Examples: Visual Descriptors Ontology (VDO), Multimedia Structure Ontology (MSO), Spatio-Temporal Ontology (STO) Ontology cross-domain linking Alignment techniques for multimedia ontologies linking with domain-specific ontologies Merging techniques for multimedia ontology coordination Compliance to standards Semantic Web, W3C, MPEG-7, etc

Informatics and Telematics Institute Centre for Research and Technology Hellas 14 Harmonization approaches (2) Harmonization of core ontologies Core ontologies cannot be simply mapped but need to be “harmonized” Need to apply ontological modeling principles during harmonization Development of a common upper ontology Starting point for new ontology construction Bridge between existing ontologies Act as the ground basis for the establishment of an integrated ontology framework Back up the common upper ontology

Informatics and Telematics Institute Centre for Research and Technology Hellas BOEMIE Multimedia annotation tool and A proposed knowledge framework

Informatics and Telematics Institute Centre for Research and Technology Hellas 16 M-OntoMat Annotizer Mid-Level Concepts Selection of regions for merging Semantic labeling Automatic segmentation Region merging functionality Descriptor extraction

Informatics and Telematics Institute Centre for Research and Technology Hellas 17 XML output

Informatics and Telematics Institute Centre for Research and Technology Hellas 18 Knowledge framework Core Ontology-Holistic Representation Domain Ontology Spatiotemporal Constraints Modality Elements Multimedia Content Ontology Classifiers/ Algorithms Features/ Multimedia Descriptor Ontology

Informatics and Telematics Institute Centre for Research and Technology Hellas 19 Athletics_Event High_ Jump_ Event Pole_Vault _Event Long_Jump_ Event Jumping_Event Horizontal _Bar Pole Sandpit Foam_Mats Foam Mats Mats Still region Multimedia object VideoImage Video segment Element Textual Element Visual Element Horizontal_Bar_ve Sandpit_ve Foam_Mats_ve is_Right is_Below is_On HasVisualElement Still_region_Inst1 Foam_Mats_ve_Inst1 Still_region_Inst2 Horizontal_Bar_ve_Inst1 depicts Horizontal_Bar_Inst_1Foam_Mats_Inst_1 HasVisualElement depicts Image_Inst1 Athletics_Event_Inst_1 Pole_Vaulter_Failed_Landing High_Jumper_Failed_Landing High_Jumper_Successful _Landing Pole_Vaulter_Successful_Landing Pole_Vaulter_Jump_Off High_Jumper_Approach High_Jumper_Jump_Off Pole_Vaulter_Swing_and_Row An indicative example