1. Internet Streaming Media Metadata Interchange with MPEG-7 Eric Rehm CTO, singingfish.com Thomson multimedia 4 May 2001, Hong Kong

2. Overview Brief look at Singingfish Indexing Internet streaming media Automating metadata delivery and processing Case Study: Using XSL to transform MSNBC schema to MPEG-7

3. singingfish.com Wholly-owned subsidiary of Thomson Multimedia B2B Streaming Media Search Service Pay per query business model Over 15 M streams indexed Live with customers since Jan 2000 –InfoSpace: Metacrawler, Dogpile –Inside Internet AG: Swiss-Search, Austria-Search Involved with MPEG-7 standards development since Sept 1999

5. Service Model

6. Indexing Streaming Media High quality metadata improves relevancy of multimedia search results Crawl….or…work directly with multimedia “Content Producers” to acquire quality metadata Solution: Implement FTP push/pull of metadata –Automated processing upon FTP close –Support bulk or incremental operations: add, update, delete, reset –Future: SOAP or other W3C XML protocol

7. Design Content Producer Program Metadata Engine

8. Development Goals Single metadata schema interface to a database –Control development costs –Partition engineering and content development Adapt to any “content partner” metadata –XML, CSV, Excel, Virage VDF, …. –Transform “content partner” metadata to MPEG-7 via: Custom applications (CSV, Excel)  MPEG-7 Proprietary XML schemas  XSL  MPEG-7

9. Case Study Create XSL transformation From: – MSNBC "Partner XML Format" To: –MPEG-7 Description

10. Experimental Results Filelinescharselemntsattrs MSNBC Partner XML Example 7311995816 MPEG-7 Result263447115174 XSL Stylesheet: 370 lines of lightly commented code

11. Discussion Basic MPEG-7 Tools Semantic Encoding of MSNBC Keywords into MPEG-7 Structured Annotation DS (Who, What, Where, When, Why, How) Encoding Controlled Terms using namespaces Encoding Streaming Media Validity with the Availability DS Extending an MPEG-7 DS

12. MSNBC Video Distribution Entry tdy_fletcher_mideast_001023 Keywords: Israel, palestinian, Yasser Arafat Top News Order: 12 Peace hopes slip farther The slim hopes for peace in the Mideast are rapidly fading, NBC’s Martin Fletcher reports Monday from the outskirts of Jerusalem. Today’s show Barak, Sharon talk coalition What’s on Today What’s on Weekend Today What’s on Today

13. MSNBC tdy_fletcher_mideast_001023 00:01:09 Peace hopes slip farther The slim hopes for peace in the Mideast are rapidly fading, NBC’s Martin Fletcher reports Monday from the outskirts of Jerusalem. Israel, palestinian, Yasser Arafat...

14. MPEG-7 link to stream http://www.msnbc.com/news/asx/video/28/tdy_fletcher_m ideast_001023.asx

15. Peace hopes slip farther

16. , The slim hopes for peace in the Mideast... Israel, palestinian, Yasser Arafat Israel Yasser Arafat</Keyword

17. Enhanced Israel, palestinian, Yasser Arafat Yasser Arafat palestinian Israel

18. Encoding Controlled Terms 1.Singingfish.com Genres are described in one namespace (urn:sf:genre). 2.MSNBC Genres are described in another namespace (urn:msnbc:category )

19. Encoding Controlled Terms International <xsl:variable name=“sfCategory" select="singingfish:mapper.map(string(category[1]/@id))">

20. Extending an MPEG-7 DS

21. <element name="Publisher" type="mpeg7:AgentType" minOccurs="0"/> <element name="Publisher" type="mpeg7:AgentType" minOccurs="0"/> Extending an MPEG-7 DS

23. Extending an MPEG-7 DS... us wa 2000-10-23T14:20:00

24. Summary Quality search depends on quality metadata –MPEG-7 standards ease development costs –Controlled vocabularies MPEG-7 MDS can be used to interoperate XML Schema allows controlled extensions

25. Thank you singingfish.com

26. Optional MPEG-7 Background Slides

27. MPEG-7 Basics ISO/IEC 15928 Multimedia Content Description Interface Comprehensive set of audiovisual description tools. Enabled by key Internet standards: –W3C: XML, XML Schema –IETF standards: URI, URN, URL for resource naming and location Harmonized with other emerging metadata standards: –Dublin Core, MPEG-21, NewsML, SMPTE Metadata Dictionary, TV-Anytime, and more. Text and compressed binary encodings –Both encodings have streaming add, delete, update features for delivery over real-time transports: MPEG-2, MPEG-4, IP, etc. International Standard in October 2001 –Ballot period begins 14 March 2001

28. Basic elements Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Time, Duration, Medialocators Time, Duration, Medialocators Textual Annotation (free text, structured annotation, syntactic dependency, etc.) Controlled vocabularies, Agent, Place, Graph, etc. Textual Annotation (free text, structured annotation, syntactic dependency, etc.) Controlled vocabularies, Agent, Place, Graph, etc.

29. Content Management & Description Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Title, Creator, Creation location & date, Purpose, Classification, Genre, etc. (Author generated) Title, Creator, Creation location & date, Purpose, Classification, Genre, etc. (Author generated) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Datatype & structures Link & media localization Basic DSs Schema tools Viewpoint of the structure: Segments Spatial / temporal structure Audio, video low-level Ds Elementary semantic information. Viewpoint of the structure: Segments Spatial / temporal structure Audio, video low-level Ds Elementary semantic information.

30. Content Management & Description (Conceptual aspects) Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Datatype & structures Link & media localization Basic DSs Schema tools Viewpoint of conceptual notions Events, objects, abstract concepts, and their relation Viewpoint of conceptual notions Events, objects, abstract concepts, and their relation

31. Navigation and Access Navigation & Access Summary Variation Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Efficient support of : discovery, browsing, navigation, visualization / sonification Datatype & structures Link & media localization Basic DSs Schema tools

32. Navigation and Access Navigation & Access Summary Variation Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Datatype & structures Link & media localization Basic DSs Schema tools Substitution of the original content Adaptation to terminal, network, or user preferences Substitution of the original content Adaptation to terminal, network, or user preferences

33. Content Organization Navigation & Access Summary Variation Model Content organization Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Probability Model Statistical functions and structures to describe sample of AV content and classes of descriptors. Analytic model: Definition of cluster, classes and models to associate a semantic label to a set of data. Probability Model Statistical functions and structures to describe sample of AV content and classes of descriptors. Analytic model: Definition of cluster, classes and models to associate a semantic label to a set of data. Description and organization of collection of documents Description and organization of collection of documents Collection & Classification Datatype & structures Link & media localization Basic DSs Schema tools

34. User Interaction Navigation & Access Summary Variation Analytic Model Content organization Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Collection & Classification UserInteraction User preferences Datatype & structures Link & media localization Basic DSs Schema tools User identification and preferences: Filtering, search and browsing User identification and preferences: Filtering, search and browsing User preferences Usage History

35. MPEG-7 DDL XML Schema Data type extensions –MIME type, ISO country, region, currency codes –ISO Character set codes –Revised time data types to support arbitrary fractional seconds denominator for per-frame positioning 2001-05-01T15:23:46N11F30 (11 th frame @ 30 FPS) Type-centric approach using root abstract types –Control available global elements –Allow extension via name spaces and mechanism

36. Basic Derivation of MPEG-7 Types <element name="Header" type="mpeg7:HeaderType" minOccurs="0" maxOccurs="unbounded"/>

37. Creation Description Scheme <element name="Title" type="mpeg7:TitleType maxOccurs="unbounded"/> … <element name="Creator“ type="mpeg7:CreatorType“ minOccurs="0" maxOccurs="unbounded"/> …