A Motivating Scenario for Designing an Extensible Audio- Visual Description Language Monday 25 th of October, 2004 Raphaël Troncy, Jean Carrive, Steffen Lalande and Jean-Philippe Poli

Raphaël TroncyCoRIMedia - 10/25/20042 Description of the AV content Various uses / Different granularity : –identification of the content creator and the content provider: Dublin Core metadata, VRA core categories, TV Anytime metadata … –feature extraction from the video signal: storing and exchanging automatic tools results (MPEG-7) –structural decomposition in video segments corresponding to a logical structure of the program: time-code, spatial coordinates –semantic description of these segments: controlled vocabulary, thesaurus, free text annotation

Raphaël TroncyCoRIMedia - 10/25/20043 Description of the AV content (cultural heritage point of view) Segmentation –locate and date some events Description –type each segment with an AV genre –type each segment with a general thematic –give hints on the production –describe the scene (who, when, where, what, …) ⇒ needs a powerful description language

Raphaël TroncyCoRIMedia - 10/25/20044 Motivating scenario Generic application for describing manually TV programs w.r.t: –structural constraints: patterns represent the logical structure of a document –semantic constraints: the description of the content is machine understandable Let us define the temporal structure of a Sports Magazine

Raphaël TroncyCoRIMedia - 10/25/20045 MPEG-7, the natural candidate description language? ISO standard since December of 2001 Main components: –Descriptors (Ds) and Description Schemes (DSs) –DDL (XML Schema + extensions) Concern all types of media Part 5 - MDS

Raphaël TroncyCoRIMedia - 10/25/20046 MPEG-7: a non-suitable description language for this scenario 1.A non-extensible language closed set of descriptors 2.Exchange syntax rather than a real machine processable multimedia description language non object-based data model non modular language (universal approach) 3.No formal semantics provided applications cannot have access to the meaning of the documents ⇒ the DDL (XML Schema) fault ?

Raphaël TroncyCoRIMedia - 10/25/20047 MPEG-7: a non-suitable description language for this scenario ⇒ how to reconciliate the critical issue object-oriented semantic expression versus structural validation How to define new descriptors ? How to define new description schemes ? How to make the description machine understandable ?

Raphaël TroncyCoRIMedia - 10/25/20048 Our proposition: AVDL AVDL: a reduced yet extensible audio-visual description language –an object meta-model (an instance model specifies the vocabulary for and the rules followed by the descriptions) –an XML syntax –a semantics (closed to DL for the descriptors) Description Schemes –Descriptors –Properties –Structures Descriptions –valid instances w.r.t description schemes

Raphaël TroncyCoRIMedia - 10/25/20049 The meta class level

Raphaël TroncyCoRIMedia - 10/25/ The class level

Raphaël TroncyCoRIMedia - 10/25/ Location

Raphaël TroncyCoRIMedia - 10/25/ Document, Content and Media Distinction : –Document vs Content vs Media –Virtual content vs physical content Media: a content abstraction for decomposition –audio tracks, subtitles

Raphaël TroncyCoRIMedia - 10/25/ Defining Structures A structure defines how the descriptors may and have to be combined –allows a description control –allows an automatic completion of the descriptions AVDL provides some predefined structure models –containment : gives the list of the possible sub-segments of an AV segment (in space and in time) –regular expression : by analogy of grammar for temporal succession Other models are currently studied: temporal constraints, etc.

Raphaël TroncyCoRIMedia - 10/25/ AVDL Implementation XML Serialization –Independent from a schema language –Use XML Schema validation (mainly for datatypes) C# –Object inheritance –Use of the.NET reflexivity

Raphaël TroncyCoRIMedia - 10/25/ d-162.xml ds-17.xml avdl.xsd XML Serialization Audio-Visual Description Language Description Schemes Descriptions ds-17.xsd partial control transformation partial control

Raphaël TroncyCoRIMedia - 10/25/ XML Syntax (DS) <Constraint type="temporal" validation="full" method="system parser="XMLSchema">

Raphaël TroncyCoRIMedia - 10/25/ XML Syntax (Descriptions) <Media id="CPB mpg" name="CPB mpg" contentID="CPB mpg" frameHeight="288" frameWidth="352"/>...

Raphaël TroncyCoRIMedia - 10/25/ Memory.NET implementation d-162.xml ds-17.xml Description Schemes Descriptions ds-17.dll parsing read/write.NET instanciation

Raphaël TroncyCoRIMedia - 10/25/ Two kinds of applications Static Description Schemes –DS are well-known –The developer uses generated libraries Dynamic Description Schemes –DS are created by the application –Use of the dynamic instantiation mechanism (reflexivity) of.NET

Raphaël TroncyCoRIMedia - 10/25/ Carrying out the scenario Definition of new descriptors and properties –associating behavior with the corresponding classes –performing reasoning on the descriptions with the formal definitions in OWL Definition of logical and temporal structures –the description is controlled and validated by a grammar

Raphaël TroncyCoRIMedia - 10/25/ Conclusion and Future Work AVDL: a reduced yet extensible Audio-Visual Description Language –descriptors, properties, structures –XML syntax and DL semantics –.NET implementation and APIs About structure validation: –which constructors used ? which semantics ? Trade-of expressivity vs calculability –OWL Full is undecidable –constraints satisfaction problems can be complex