Informatics and Telematics Institute - CERTH 1 BOEMIE: Bootstrapping Ontology Evolution with Multimedia Information Extraction Vasileios Papastathis Centre for Research and Technology Hellas (CERTH) Informatics and Telematics Institute Multimedia Knowledge Group 3 rd Know-How Transfer Event Thessaloniki, 8 March 2007

data Centre for Research and Technology Hellas ( CERTH ) 2 Presentation Overview  A short presentation of CERTH-ITI and the Multimedia Knowledge Group (MKG)  BOEMIE project: An FP6 success story  FP7 – Challenge 4: “Digital Libraries and Content”

data Centre for Research and Technology Hellas ( CERTH ) 3 History-Scope  Founded in 1998 as a non-profit organisation under the auspices of the General Secretariat of Research and Technology of the Greek Ministry of Development  Since March 2000, it is part of the Centre for Research and Technology – Hellas as one of its four constituent institutes  Set-up to constitute a major research and development centre, with continuous interaction with the academic community, the National and European Informatics and Telematics Industry, the international scientific community and the Public Sector  Play a key role in the development of the Greek Information Society as a National Center of Excellence in Informatics and Telematics  Set-up spin-off companies aiming at the commercial exploitation of ITI ’ s research results

data Centre for Research and Technology Hellas ( CERTH ) 4 Structure and Organization  Virtual Reality Research Unit  Advanced e-Services for the Knowledge Society Research Unit  Telecommunications and Telematics Research Unit  Intelligent Systems and Software Engineering Research Unit  Business Information Systems Research Unit

data Centre for Research and Technology Hellas ( CERTH ) 5 Structure and Organization Multimedia Knowledge Group  Semantic Multimedia Analysis  Multimedia Indexing and Retrieval  Multimedia and the Semantic Web  Knowledge Structures, Languages and Tools for Multimedia  Reasoning and Personalization for Multimedia Applications  MPEG-7 and MPEG-21 Standards

data Centre for Research and Technology Hellas ( CERTH ) 6 Personnel  13 Professors  7 Researchers Grade C and D  5 Post-Doctoral Researchers  20 PhD Candidates –(Postgraduate Research Fellows)  45 Research Assistants –(all University Graduates, MSc)  2 Technicians (University Graduates)  3 Administration Staff  20 Undergraduate Students

data Centre for Research and Technology Hellas ( CERTH ) 7 Publications and Projects  Since 2000 –140 publications in peer-reviewed international journals –46 book chapters –370 publications in international and national conferences –More than 450 citations (not by authors or coauthors)  70 R&D projects funded by European Commission Programmes (8.8 MEuro)  31 R&D projects funded by National Programmes (2.2 MEuro)  50 Industrial Contracts and Subcontracts (2.45 MEuro)  27 5th FP European projects –Coordinator in 4 EC IST projects (SCHEMA NoE, LAURA, EU- PUBLI.COM, KOD: Knowledge on Demand) –Financial coordinator of 2 EC IST projects (INTERVUSE, P2People)

data Centre for Research and Technology Hellas ( CERTH ) 8 Funding ( )

data Centre for Research and Technology Hellas ( CERTH ) 9 FP6 R&D Projects  “aceMedia: Integrating knowledge, semantics and content for user centred intelligent media services”, IP  “KnowledgeWeb: Realizing the Semantic Web”, funded by the DG XIII, NoE  “MESH: Multimedia Semantic Syndication for Enhanced News Services”, IST – IP,  “X-Media: Knowledge Sharing and Reuse Across Media”, IST – IP,  “BOEMIE: Bootstrapping Ontology Evolution with Multimedia Information Extraction”, IST-STREP,  “K-Space: Knowledge Space of Semantic Inference for Automatic Annotation and Retrieval of Multimedia Content”, 6th FP IST NoE,  Since 2000 –106 publications in peer-reviewed international journals –46 book chapters –342 (326+16) publications in international and national conferences –More than 450 citations (not by authors or coauthors)  40 R&D projects funded by European Commission Programmes (8.8 MEuro)  31 R&D projects funded by National Programmes (2.2 MEuro)  50 Industrial Contracts and Subcontracts (2.45 MEuro)  27 5th FP European projects –Coordinator in 4 EC IST projects (SCHEMA NoE, LAURA, EU- PUBLI.COM, KOD: Knowledge on Demand) –Financial coordinator of 2 EC IST projects (INTERVUSE, P2People)

Informatics and Telematics Institute - CERTH 10 BOEMIE: BOEMIE: Bootstrapping Ontology Evolution with Multimedia Information Extraction

data Centre for Research and Technology Hellas ( CERTH ) 11  Aims and Objectives: –An RTD project designed to gain knowledge or improve existing products, processes or services –A demonstration project designed to prove the viability of new technologies, but which cannot be commercialized directly  Number of participants: –Minimum of 3 partners from three different Member States  Duration: –Typically between 2 to 3 years  Projects Management: –Require overall management and coordination of the consortium Specific Targeted Research Projects (STREP)

data Centre for Research and Technology Hellas ( CERTH ) 12 The facts  Specific Targeted Research Projects (STREP), IST – 2004 – “Semantic-based Knowledge and Content Systems”  Start: March 1, 2006  End: February 28, 2009  Budget: Euro  EU Funding: Euro  More than 30 people already active in the project  Project portal:

data Centre for Research and Technology Hellas ( CERTH ) 13 Consortium  Inst. of Informatics & Telecommunications, NCSR “Demokritos”, Greece (Coordinator)  Fraunhofer Institute for Media Communication (NetMedia), Germany  Dip. di Informatica e Comunicazione, University of Milano, Italy  Centre for Research and Technology Hellas (CERTH) - Informatics & Telematics Institute (ITI), Greece  Hamburg University of Technology, Germany  TeleAtlas SA, the Netherlands

data Centre for Research and Technology Hellas ( CERTH ) 14 Vision  Pave the way towards automation of the knowledge acquisition from multimedia content.  Break new ground by introducing and implementing the concept of evolving multimedia ontologies.  Make domain-specific semantic webs feasible with limited human effort.

data Centre for Research and Technology Hellas ( CERTH ) 15 Objectives  Providing technology to represent and evolve domain-specific multimedia ontologies.  Moving from low-level, general-purpose, single- modality feature extraction towards semantic, multimedia analysis.  Robust and scalable ontology-driven multimedia content extraction through ontology evolution.

data Centre for Research and Technology Hellas ( CERTH ) 16 Approach  Driven by domain-specific multimedia ontologies, BOEMIE information extraction systems will be able to identify high-level semantic features in image, video, audio and text and fuse these features for optimal extraction.  The ontologies will be continuously populated and enriched using the extracted semantic content.  This is a bootstrapping process, since the enriched ontologies will in turn be used to drive the multimedia information extraction system.

data Centre for Research and Technology Hellas ( CERTH ) 17 Approach EVOLVED ONTOLOGY EVOLUTION PROCESS INITIAL ONTOLOGY POPULATION & ENRICHMENT COORDINATION O O L L R R M INTERMEDIATE ONTOLOGY SEMANTICS EXTRACTION OTHER ONTOLOGIES EVENTS DATABASE MAPS DATABASE MAP ANNOTATION INTERFACE INFORMATION EXTRACTION RESULTS MULTIMEDIA Content ONTOLOGY EVOLUTION TOOLKIT L LEARNING TOOLS R REASONING ENGINE M MATCHING TOOLS O ONTOLOGY MANAGEMENT TOOL SEMANTICS EXTRACTION TOOLKIT T TEXT EXTRACTION TOOLS A AUDIO EXTRACTION TOOLS F INFORMATION FUSION TOOLS V VISUAL EXTRACTION TOOLS VTA F

data Centre for Research and Technology Hellas ( CERTH ) 18  No single modality is powerful enough to support robust and large-scale extraction.  Emphasis on fusion of multiple modalities, using reasoning and uncertainty handling.  Contribution to the state-of-the-art in visual content analysis, due to its richness and the difficulty of extracting semantics.  Non-visual content will provide supportive evidence, to improve precision. Semantics extraction: Objectives

data Centre for Research and Technology Hellas ( CERTH ) 19  A multimedia ontology describes the structure of multimedia content and visual characteristics of content objects in terms of low-level features.  One or more domain ontologies, e.g. about athletics.  A geographic ontology, e.g. about landmarks.  An event ontology, e.g. about athletic events.  Potential contribution: –Uncertainty in concept descriptions –Spatial and temporal relations Multimedia semantic model: Objectives

data Centre for Research and Technology Hellas ( CERTH ) 20  Ontology population and enrichment, i.e. addition of concepts, relations, properties and instances.  Coordination of homogeneous ontologies (same domain) and heterogeneous ontologies (e.g. domain and multimedia ontologies).  Potential contribution: –Ontology population from multimedia content. –Coordination of different types of reasoning for enrichment and coordination. –Matching, coordination and versioning of the integrated semantic model. Ontology evolution: Objectives

data Centre for Research and Technology Hellas ( CERTH ) 21 7 th FP

data Centre for Research and Technology Hellas ( CERTH ) 22 Challenge 4 “Digital Libraries and Content” Make content and knowledge abundant, accessible, interactive and usable over time by humans and machines alike. –Content must be made available through digital libraries and its long term usability, accessibility and preservation must be ensured –Effective technologies need to be developed for intelligent content creation and management, and for supporting the capture of knowledge and its sharing and reuse –Individuals, organisations and communities must find new ways to acquire and exploit knowledge, and thereby learn Political framework: « i Digital Libraries »

data Centre for Research and Technology Hellas ( CERTH ) 23 Intelligent Content & Semantics Make digital resources that embody creativity and semantics easier and more cost effective to produce, organize, search, personalise, distribute and use across the value chain. –CREATORS: Design more communicative and participative forms of content (media professionals, enterprise designers, talented amateurs). –PUBLISHERS: Increase productivity in creative industries, enterprises and professional sectors (e.g. health, law, etc.). –SCIENTISTS: Automate link between data analysis, theory and experimental validation. –ORGANISATIONS & COMMUNITIES: Automate collection and distribution of digital content and machine-tractable knowledge, and their sharing in collaborative environments.

data Centre for Research and Technology Hellas ( CERTH ) 24 Target socio-economic sectors  key features – ICT based, high growth & innovation potential – pronounced international character – sophisticated users – very large data volumes – well defined flows & protocols  obvious candidates (in addition to ICT!) – creative industries (film, TV, games, advertising …) – enterprises in information bound industries utilities eg energy manufacturing & process industries construction & engineering, financial services … – eScience eg life sciences

data Centre for Research and Technology Hellas ( CERTH ) 25 In NO intend to support research into:  basic research with no identifiable by-products within 10 years  domain specific applications - not portable/replicable in other socio-economic sectors  developments addressing immediate commercial imperatives (e.g. content protection & monetisation)  issues covered by other Challenges and Objectives eg media networking, peer to peer, technology enabled learning …  topics well covered by on-going FP6 projects & networks (see our website) Do NOT do

data Centre for Research and Technology Hellas ( CERTH ) 26 Schedule of 1st call (provisional)  51 Meuro in total of which: –46 Meuro for IP & STR projects – 5 Meuro for NoEs & CSA’s  first call expected to close late April (?)  evaluation/selection mid-May – late Jun (?)  negotiations until Nov  contract awarding in Dec  projects due to start Q highly demanding process …

data Centre for Research and Technology Hellas ( CERTH ) 27 CERTH-ITI in FP7  Continue research in Multimedia and Knowledge Technologies  Expand to new areas and applications (Health, Industry, Cognition, Robotics, Environment, Security, Surveillance, …)  Challenges in IST –Networked media –Cognitive systems, interaction, robotics –Digital libraries and technology-enhanced learning –Intelligent content and semantics –Personal health systems for monitoring and point-of-care diagnostics –Advanced ICT for risk assessment and patient safety

data Centre for Research and Technology Hellas ( CERTH ) 28 Thank you! Mr. Vasileios Papastathis Multimedia Knowledge Group