Internet Streaming Media Metadata Interchange with MPEG-7 Eric Rehm CTO, singingfish.com Thomson multimedia 4 May 2001, Hong Kong

Overview Brief look at Singingfish Indexing Internet streaming media Automating metadata delivery and processing Case Study: Using XSL to transform MSNBC schema to MPEG-7

singingfish.com Wholly-owned subsidiary of Thomson Multimedia B2B Streaming Media Search Service Pay per query business model Over 15 M streams indexed Live with customers since Jan 2000 –InfoSpace: Metacrawler, Dogpile –Inside Internet AG: Swiss-Search, Austria-Search Involved with MPEG-7 standards development since Sept 1999

Service Model

Indexing Streaming Media High quality metadata improves relevancy of multimedia search results Crawl….or…work directly with multimedia “Content Producers” to acquire quality metadata Solution: Implement FTP push/pull of metadata –Automated processing upon FTP close –Support bulk or incremental operations: add, update, delete, reset –Future: SOAP or other W3C XML protocol

Design Content Producer Program Metadata Engine

Development Goals Single metadata schema interface to a database –Control development costs –Partition engineering and content development Adapt to any “content partner” metadata –XML, CSV, Excel, Virage VDF, …. –Transform “content partner” metadata to MPEG-7 via: Custom applications (CSV, Excel)  MPEG-7 Proprietary XML schemas  XSL  MPEG-7

Case Study Create XSL transformation From: – MSNBC "Partner XML Format" To: –MPEG-7 Description

Experimental Results Filelinescharselemntsattrs MSNBC Partner XML Example MPEG-7 Result XSL Stylesheet: 370 lines of lightly commented code

Discussion Basic MPEG-7 Tools Semantic Encoding of MSNBC Keywords into MPEG-7 Structured Annotation DS (Who, What, Where, When, Why, How) Encoding Controlled Terms using namespaces Encoding Streaming Media Validity with the Availability DS Extending an MPEG-7 DS

MSNBC Video Distribution Entry tdy_fletcher_mideast_ Keywords: Israel, palestinian, Yasser Arafat Top News Order: 12 Peace hopes slip farther The slim hopes for peace in the Mideast are rapidly fading, NBC’s Martin Fletcher reports Monday from the outskirts of Jerusalem. Today’s show Barak, Sharon talk coalition What’s on Today What’s on Weekend Today What’s on Today

MSNBC tdy_fletcher_mideast_ :01:09 Peace hopes slip farther The slim hopes for peace in the Mideast are rapidly fading, NBC’s Martin Fletcher reports Monday from the outskirts of Jerusalem. Israel, palestinian, Yasser Arafat...

MPEG-7 link to stream ideast_ asx

Peace hopes slip farther

, The slim hopes for peace in the Mideast... Israel, palestinian, Yasser Arafat Israel Yasser Arafat</Keyword

Enhanced Israel, palestinian, Yasser Arafat Yasser Arafat palestinian Israel

Encoding Controlled Terms 1.Singingfish.com Genres are described in one namespace (urn:sf:genre). 2.MSNBC Genres are described in another namespace (urn:msnbc:category )

Encoding Controlled Terms International <xsl:variable name=“sfCategory"

Extending an MPEG-7 DS

<element name="Publisher" type="mpeg7:AgentType" minOccurs="0"/> <element name="Publisher" type="mpeg7:AgentType" minOccurs="0"/> Extending an MPEG-7 DS

Extending an MPEG-7 DS... us wa T14:20:00

Summary Quality search depends on quality metadata –MPEG-7 standards ease development costs –Controlled vocabularies MPEG-7 MDS can be used to interoperate XML Schema allows controlled extensions

Thank you singingfish.com

Optional MPEG-7 Background Slides

MPEG-7 Basics ISO/IEC Multimedia Content Description Interface Comprehensive set of audiovisual description tools. Enabled by key Internet standards: –W3C: XML, XML Schema –IETF standards: URI, URN, URL for resource naming and location Harmonized with other emerging metadata standards: –Dublin Core, MPEG-21, NewsML, SMPTE Metadata Dictionary, TV-Anytime, and more. Text and compressed binary encodings –Both encodings have streaming add, delete, update features for delivery over real-time transports: MPEG-2, MPEG-4, IP, etc. International Standard in October 2001 –Ballot period begins 14 March 2001

Basic elements Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Time, Duration, Medialocators Time, Duration, Medialocators Textual Annotation (free text, structured annotation, syntactic dependency, etc.) Controlled vocabularies, Agent, Place, Graph, etc. Textual Annotation (free text, structured annotation, syntactic dependency, etc.) Controlled vocabularies, Agent, Place, Graph, etc.

Content Management & Description Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Title, Creator, Creation location & date, Purpose, Classification, Genre, etc. (Author generated) Title, Creator, Creation location & date, Purpose, Classification, Genre, etc. (Author generated) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Datatype & structures Link & media localization Basic DSs Schema tools Viewpoint of the structure: Segments Spatial / temporal structure Audio, video low-level Ds Elementary semantic information. Viewpoint of the structure: Segments Spatial / temporal structure Audio, video low-level Ds Elementary semantic information.

Content Management & Description (Conceptual aspects) Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Datatype & structures Link & media localization Basic DSs Schema tools Viewpoint of conceptual notions Events, objects, abstract concepts, and their relation Viewpoint of conceptual notions Events, objects, abstract concepts, and their relation

Navigation and Access Navigation & Access Summary Variation Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Efficient support of : discovery, browsing, navigation, visualization / sonification Datatype & structures Link & media localization Basic DSs Schema tools

Navigation and Access Navigation & Access Summary Variation Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Datatype & structures Link & media localization Basic DSs Schema tools Substitution of the original content Adaptation to terminal, network, or user preferences Substitution of the original content Adaptation to terminal, network, or user preferences

Content Organization Navigation & Access Summary Variation Model Content organization Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Probability Model Statistical functions and structures to describe sample of AV content and classes of descriptors. Analytic model: Definition of cluster, classes and models to associate a semantic label to a set of data. Probability Model Statistical functions and structures to describe sample of AV content and classes of descriptors. Analytic model: Definition of cluster, classes and models to associate a semantic label to a set of data. Description and organization of collection of documents Description and organization of collection of documents Collection & Classification Datatype & structures Link & media localization Basic DSs Schema tools

User Interaction Navigation & Access Summary Variation Analytic Model Content organization Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Collection & Classification UserInteraction User preferences Datatype & structures Link & media localization Basic DSs Schema tools User identification and preferences: Filtering, search and browsing User identification and preferences: Filtering, search and browsing User preferences Usage History

MPEG-7 DDL XML Schema Data type extensions –MIME type, ISO country, region, currency codes –ISO Character set codes –Revised time data types to support arbitrary fractional seconds denominator for per-frame positioning T15:23:46N11F30 (11 th 30 FPS) Type-centric approach using root abstract types –Control available global elements –Allow extension via name spaces and mechanism

Basic Derivation of MPEG-7 Types <element name="Header" type="mpeg7:HeaderType" minOccurs="0" maxOccurs="unbounded"/>

Creation Description Scheme <element name="Title" type="mpeg7:TitleType maxOccurs="unbounded"/> … <element name="Creator“ type="mpeg7:CreatorType“ minOccurs="0" maxOccurs="unbounded"/> …