1. Multimedia Semantic Web and MPEG-7 Ana B. Benitez ana @ ee.columbia.edu Image and Advanced Television Lab (ADVENT) Department of Electrical Engineering Columbia University

2. Semantic Web and Multimedia Motivation: Rapid explosion of available multimedia on the Web Extract semantics from multimedia is even harder than from text! Multimedia Semantic Web: Describe multimedia documents Enable interoperable, scalable and intelligent applications Deal with multimedia and interact with users at human levels MPEG-7 – Multimedia Content Description Interface : Describe different aspects of multimedia documents at different abstraction levels Enable multimedia applications in general

3. MPEG-7 Suite of tools for describing: Structure and semantics ~ XML and RDF for generic data Audio and visual features – color, texture, melody and timbre Content management information – creator, media format and rights Free and structured annotations – who, when and where Text and graph classification schemes ~ thesauri and ontologies Summaries and variations – key frame hierarchy and abstracts Multimedia collections and models – statistical models and classifiers XML-Schema as description definition language ISO standard in September 2001!

4. MPEG-7 Description

5. Abstraction Levels

6. Structure and Semantic Tools Framework comparable to ER Modeling and Semantic Networks: Entities, attributes and relationships Structure descriptions: Segment entities: video, audio, multimedia segments, … Attributes: AV descriptors, spatio-temporal localization, … Relationships: temporal, spatial, … Semantic descriptions: Semantic entities: objects, events, concepts, states, places, times Attributes: label, definition, abstraction level, media occurrences Relationships: object-event, concept-semantic entity, … Multiple abstraction levels: media and formal Encoding of rules based on alphabet and rule graphs in the works

7. RDF/DAML+OIL vs MPEG-7 Pros RDF/DAML+OIL: Cardinality of properties different than zero or one Arbitrary union and intersection combinations of classes Pros MPEG-7: Richer and more flexible graphical structure: Total or partial graph morphisms to relate graphs Discover and reuse sub-descriptions but not as nodes, e.g., modern adaptation of Hamlet (poisson  heart medicine; drown  step front of car) Closer to how humans construct descriptions More types of entities (segments, object, events, …), attributes (AV features, definition, abstraction, …), relations (composition, dependency, spatio-temporal…), and abstraction levels (media, formal) Mechanism to encode rules to enable inference in progress Non-structure/semantic information (management, summaries, …)

8. Some Questions … What is the Semantic Web doing about multimedia? What is the position of the Semantic Web on MPEG-7?

9. Some Pointers To know more about MPEG-7: MPEG: http://www.cselt.it/mpeg MPEG-7 Industry Forum: http://www.mpeg7.org To know more about our work at MPEG-7: MPEG-7 Project at Columbia University: http://www.ctr.columbia.edu/~ana/MPEG7 ADVENT: http://www.ee.columbia.edu/advent, Research

10. The End Thanks for you attention!

11. Content Management Description Photographer Seungyup Columbia University 1998-09-19 image JPG http://www.alex&ana.jpg columbia:1919:alex&ana_image

12. Content Structure Description Alex shakes hands with Ana Alex... Ana <Relation xsi:type="DirectionalSpatialSegmentRelationType“ name="left“ target="#SR2"/>...

13. Locator Description T00:00:15 PT1M15S 4.34 1.43 4.33 10.3 5.03.33 5.34.43 2.37 4.34 1.43 4.33 10.3 5.03.33 5.34.43 2.37 4.34 1.43 4.33 10.3 5.03.33 5.34.43 2.37 …

14. Content Semantics Description Alex shakes hands with Ana Shake hands <Relation xsi:type="ObjectEventRelationType“ name="hasAgentOf“ target="#AO1"/> <Relation xsi:type="ObjectEventRelationType“ name="hasAccompanierOf“ target="#AO2"/> <Relation xsi:type="ConceptSemanticBaseRelationType“ name="hasPropertyOf" target="#C1"/> Columbia University September 9, 1998 Alex Ana Comradeship Associate Friend