LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Cross-Media Indexing in the Reveal-This System Murat Yakici, Fabio Crestani Dept. Computer & Information Sciences University of Strathclyde, Glasgow, UK

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Overview Application Scenario Reveal-This Project Cross-Media Indexing –Process Model –Information Model –Indexing Model Evaluation Future work

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Application Scenario Persistency MPEG-7 Coded … How about the Armani jacket that I saw on Fashion TV

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May The Reveal-This (R-T) Project Aims at: Developing content programming technology able to: 1.capture 2.semantically index 3.categorise 4.cross-link … Multiplatform, multimedia and multilingual digital content…

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May The Reveal-This Project (2) … as well as provide the system user with: 1.semantic search 2.retrieval 3.summarisation 4.translation… functionalities Clearly an ambitious project!

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Digital Content in R-T Digital Content is –Distributed over different platforms –Repurposed and delivered to diverse devices –Can be in a range of media types –Rapidly consumed (on demand provision) Key issues –Managing meta-data –Managing data

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Reveal-This Architecture

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Cross-Media Indexing Component (CMIC) Addresses the 2 nd scientific objective and in part also the 4 th Part of Cross-Media Indexing and Analysis Subsystem Media meta-data integration and indexing Service

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May CMIC Overview Builds relationships among concepts extracted from different processors –such as video, speech and text analysis How do we do it?

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May CMIC Overview (2) Two interpretations of “media” 1.Different sources on same topic 2.Single source but different type on the same topic

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May CMIC Features Transform source XML (feature) streams into an internal cross-media representation (MPEG-7) in order to make comparisons in the same similarity space Capture relations across media Cross-link media (within document as cross- media) Augment digital objects with semantic information (MPEG-7) Store meta-data and relations Online indexing and retrieval Support to various languages English, Greek, French Enable push and pull of events

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May CMIC Process Model

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May CMIC Information Model Structures and patterns for representing digital content at any level –What is the unit of information processing across R-T system? –Are there any common patterns out there? –What are the emerging standards? We adoped MPEG-7 as the information model

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May MPEG-7 Overview Standard is finalised “How to describe content” Provides diverse and large set of description elements Tools for –Multiplexing of descriptions –Synchronization of descriptions with content

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May MPEG-7 How to describe content –A set of description schemes (DS) and descriptors (D) –A language to specify description schemes (Description Definition Language - DDL) –A scheme for binary coding the descriptions Consider DSs as a library of descriptions –Feel free to pick and use appropriate subset of relevant DSs depending on your requirements.

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Subtitles Face 1 Face 2 Speaker 2 Speaker 1 Relevant Segment And Others… Studio Setting Transitions Zoom in Closed captions Noise Music Transcription … Task of Cross-Media Indexing

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Subtitles Face 1 Face 2 Speaker 2 Speaker 1 Relevant Segment Semantic Indexing Person 2 Sea Person 1 Boat Sailing

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Challenge Signal and Semantic Gap problem Task entails dealing with –uncertainty –imprecision –inconsistency from each single media analysis module

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Indexing Models supported by CMIC –Tf = term frequency –Tf*Idf = gives less importance to a term, if it appears in high number of stories –Modal Tf = the term frequency provided from each processing module (such as image analysis, text processing etc.) is incorporated to the previous approaches –Dempster-Shafer Multi-Evidence Approach

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Dempster-Shafer (1) Dempster-Shafer combines two or more bodies of evidence defined with in the same frame of discernment into one body of evidence. Every modality individually gives a support for a single story, a term’s existence in one modality is counted as an evidence to support the topical similarity hypothesis. Each processing module is treated as a probability function also called as Source of Evidence or Base Probability Assignment (BPA).

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Dempster-Shafer (2) Combination

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Initial experience Accuracy of processing units – In general Granularity of Topics and Categories –Topics and Categories are not describing on the same level –Terms in categories/Topics do not appear in text (Inconsistencies between values received) Confidence scores & Ranking Faces are detected but… Indexing in other languages (Greek and French) Indexing Model Indexing time depends on the indexing model

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Evaluation Task oriented user experiment. –The users are given a test video sequence and then asked to detect-recognize faces, transcribe text and describe certain aspects. Their descriptions are regarded as confidence level 1. –The tasks are introduced. The users try to find out information needed to accomplish each individual task and find most relevant segments –Compare this with CMIC’s output.

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Evaluation (2) Data annotation by experts: relevance assessments: A three-hour multimedia test collection Covering politics, travel and news In English, French and Greek languages

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Towards complex models After we finish testing the simpler indexing models we will explore more complex models: –Bayesian Networks –Kernel Canonical Correlation Analysis –Gaussian Mixture Models

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Future work Research –Add and improve indexing models –Benchmark performance with current version on updated data sets –Evaluation with annotated data set Software engineering tasks –Add management and administration services –Use in push and pull services –Adapt to user profile –Integrate with other services (summarisation, etc.)

LREC – Workshop on Crossing media for Improved Information Access, Genova, Italy, 23 May Thank you Questions?