D2 Conceration, Vilamoura, April 16th Video & Image Indexing and Retrieval in the Large Scale V T A L A S Progress Nozha Boujemaa, Scientific Coordinator

D2 Conceration, Vilamoura, April 16th Video & Image Indexing and Retrieval in the Large Scale Consortium Instrument: Integrated Project Start-End Date: 1 January December 2009 Project co-ordinator: ERCIM Scientific coordinator: INRIA Consortium: 12 partners from 7 countries ERCIM (FR), EADS (FR), CWI (NL), Fraunhofer (GE), INRIA (FR), ROBOTIKER (ES), Univ. of Sunderland (UK), CERTH-ITI (GR), Codeworks (UK), INA (FR), Belga (BE), IRT (GE) + AFP

D2 Conceration, Vilamoura, April 16th Video & Image Indexing and Retrieval in the Large Scale Objectives VITALAS is a use-case driven project that aims at providing a pre-industrial prototype system dedicated to intelligent access services to multimedia professional archives that would provide the consumer with new technological functionalities. VITALAS will provide advanced solution for indexing, searching and accessing large scale of non previously (or partly) annotated content. Cross-media (audio/speech, video, image, text) Content enrichment / indexing Interactive retrieval methods (query refinement with log files, RFB, context adaption) Vitalas system to be deployed by INA, Belga and AFP (V1 results)

D2 Conceration, Vilamoura, April 16th Video & Image Indexing and Retrieval in the Large Scale First year achievements User issues - WP1: Description of users requirements, Definition of VITALAS Use Cases, Elaboration and availability of test corpora Enabling technologies – WP2: Beyond state-of-the-art mono-media content description techniques (New local visual features [Rebai2007] [Joly2007] ; Well-trained models for word- and syllable-based recognition of German data) New probabilistic similarity search structure, 5 to 50 times faster than SoA techniques on large datasets. Integrated modules as web services +

D2 Conceration, Vilamoura, April 16th Video & Image Indexing and Retrieval in the Large Scale First year achievements Cross-Media indexing – WP3: Integration (web services) of partners know-how cross- media indexing tool (text and visual features; combination using SVM) Research activities on cross-media concept vocabularies: Concept dictionaries for cross-media indexing has been derived Cross-Media Retrieval –WP4: Query expressions to represent VITALAS use-cases Search log analysis of Belga image collection to refine query expression Coordination of and participation in INEX multimedia track

VITALAS Review, Paris, March 14th Vitalas System - V1 Architecture validation Services interfaces Exchange formats Hosting structure (ESB, Orchestrator) Evaluate & Validate a development and integration methodology of heterogeneous components Origin (Offline part : IRT, UoS, CERTH-ITI, INRIA, EADS Online part : INRIA, INA, EADS, Robotiker) Form (COTS, Open Source) Programming languages (Java, C++, …) Produce a first complete and running application Currently running, and being improved WebLab open source platform available on the web

VITALAS Review, Paris, March 14th Demonstration use case Cross media search : text + image + automatic metadata generation Dataset : images from Belga Offline part : Indexation Process –Image normalization and metadata extraction : IRT –Text concept extraction : UoS –Image concept extraction : CERTH-ITI –Text and Annotation Indexation : EADS –XML Repository : EADS –Image Indexation : INRIA

VITALAS Review, Paris, March 14th Online part : –Search engine GUI : Robotiker –XML Repository : EADS –Text or annotation retrieval : EADS –Image Search : INRIA –Cluster Map Visualisation : INA –Orchestration and Cross-medis Search : EADS Demonstration use case

VITALAS Review, Paris, March 14th Exploration Map X Files Subtask technical achievements Interactive exploration from visual similarities Task: exploring images with horses

VITALAS Review, Paris, March 14th QUERY: Unwanted results.... Just because KIA is part of the title/caption Remember: Belga Use-Case => Public Relations Manager of KIA: … I want to be alerted whenever the KIA logo is used…

VITALAS Review, Paris, March 14th MISSED RESULT Traditional Jouranlistic annotation no longer meets user requirements But extra-, post-editing of more conceptual metadata is expensive/time-intensive… Remember: Belga Use-Case => Public Relations Manager of KIA: … I want to be alerted whenever the KIA logo is used…

VITALAS Review, Paris, March 14th Whats next? Vitalas System V1+ & V2 V1+: (M18) (image and text) Large scale indexing and retrieval Full integration of Map Visualization tools V2: (M24) (audio and Video) Integration of Audio modules (speech to text) Integration of Video Mining and Search tools

VITALAS Review, Paris, March 14th

VITALAS Review, Paris, March 14th

VITALAS Review, Paris, March 14th Logical architecture SubmitQuery ESB ImageSearch INRIA::Maestro > TextAndAnnotationSearch > EADS::Lucene XMLRepository > EADS::WebLabRepo Robotiker::SearchGUI > Tomcat > EADS::FusedSearch INA:ClusterGUI >

VITALAS Review, Paris, March 14th Physical architecture Text / Annotation Search Image Search GUI ESB Fused Search Repository Image Server

VITALAS Review, Paris, March 14th Sample Process Fused search

D2 Conceration, Vilamoura, April 16th Video & Image Indexing and Retrieval in the Large Scale « Integrated Demo » « EADS, INRIA, ROBOTIKER, INA, UoS, CERTH-ITI, IRT »