1. A Motivating Scenario for Designing an Extensible Audio- Visual Description Language Monday 25 th of October, 2004 Raphaël Troncy, Jean Carrive, Steffen Lalande and Jean-Philippe Poli

2. Raphaël TroncyCoRIMedia - 10/25/20042 Description of the AV content Various uses / Different granularity : –identification of the content creator and the content provider: Dublin Core metadata, VRA core categories, TV Anytime metadata … –feature extraction from the video signal: storing and exchanging automatic tools results (MPEG-7) –structural decomposition in video segments corresponding to a logical structure of the program: time-code, spatial coordinates –semantic description of these segments: controlled vocabulary, thesaurus, free text annotation

3. Raphaël TroncyCoRIMedia - 10/25/20043 Description of the AV content (cultural heritage point of view) Segmentation –locate and date some events Description –type each segment with an AV genre –type each segment with a general thematic –give hints on the production –describe the scene (who, when, where, what, …) ⇒ needs a powerful description language

4. Raphaël TroncyCoRIMedia - 10/25/20044 Motivating scenario Generic application for describing manually TV programs w.r.t: –structural constraints: patterns represent the logical structure of a document –semantic constraints: the description of the content is machine understandable Let us define the temporal structure of a Sports Magazine

5. Raphaël TroncyCoRIMedia - 10/25/20045 MPEG-7, the natural candidate description language? ISO standard since December of 2001 Main components: –Descriptors (Ds) and Description Schemes (DSs) –DDL (XML Schema + extensions) Concern all types of media Part 5 - MDS

6. Raphaël TroncyCoRIMedia - 10/25/20046 MPEG-7: a non-suitable description language for this scenario 1.A non-extensible language closed set of descriptors 2.Exchange syntax rather than a real machine processable multimedia description language non object-based data model non modular language (universal approach) 3.No formal semantics provided applications cannot have access to the meaning of the documents ⇒ the DDL (XML Schema) fault ?

7. Raphaël TroncyCoRIMedia - 10/25/20047 MPEG-7: a non-suitable description language for this scenario ⇒ how to reconciliate the critical issue object-oriented semantic expression versus structural validation How to define new descriptors ? How to define new description schemes ? How to make the description machine understandable ?

8. Raphaël TroncyCoRIMedia - 10/25/20048 Our proposition: AVDL AVDL: a reduced yet extensible audio-visual description language –an object meta-model (an instance model specifies the vocabulary for and the rules followed by the descriptions) –an XML syntax –a semantics (closed to DL for the descriptors) Description Schemes –Descriptors –Properties –Structures Descriptions –valid instances w.r.t description schemes

9. Raphaël TroncyCoRIMedia - 10/25/20049 The meta class level

10. Raphaël TroncyCoRIMedia - 10/25/200410 The class level

11. Raphaël TroncyCoRIMedia - 10/25/200411 Location

12. Raphaël TroncyCoRIMedia - 10/25/200412 Document, Content and Media Distinction : –Document vs Content vs Media –Virtual content vs physical content Media: a content abstraction for decomposition –audio tracks, subtitles

13. Raphaël TroncyCoRIMedia - 10/25/200413 Defining Structures A structure defines how the descriptors may and have to be combined –allows a description control –allows an automatic completion of the descriptions AVDL provides some predefined structure models –containment : gives the list of the possible sub-segments of an AV segment (in space and in time) –regular expression : by analogy of grammar for temporal succession Other models are currently studied: temporal constraints, etc.

14. Raphaël TroncyCoRIMedia - 10/25/200414 AVDL Implementation XML Serialization –Independent from a schema language –Use XML Schema validation (mainly for datatypes) C# –Object inheritance –Use of the.NET reflexivity

15. Raphaël TroncyCoRIMedia - 10/25/200415 d-162.xml ds-17.xml avdl.xsd XML Serialization Audio-Visual Description Language Description Schemes Descriptions ds-17.xsd partial control transformation partial control

16. Raphaël TroncyCoRIMedia - 10/25/200416 XML Syntax (DS) <Constraint type="temporal" validation="full" method="system parser="XMLSchema">

17. Raphaël TroncyCoRIMedia - 10/25/200417 XML Syntax (Descriptions) <Media id="CPB86006610.mpg" name="CPB86006610.mpg" contentID="CPB86006610.mpg" frameHeight="288" frameWidth="352"/>...

18. Raphaël TroncyCoRIMedia - 10/25/200418 Memory.NET implementation d-162.xml ds-17.xml Description Schemes Descriptions ds-17.dll parsing read/write.NET instanciation

19. Raphaël TroncyCoRIMedia - 10/25/200419 Two kinds of applications Static Description Schemes –DS are well-known –The developer uses generated libraries Dynamic Description Schemes –DS are created by the application –Use of the dynamic instantiation mechanism (reflexivity) of.NET

20. Raphaël TroncyCoRIMedia - 10/25/200420 Carrying out the scenario Definition of new descriptors and properties –associating behavior with the corresponding classes –performing reasoning on the descriptions with the formal definitions in OWL Definition of logical and temporal structures –the description is controlled and validated by a grammar

21. Raphaël TroncyCoRIMedia - 10/25/200421 Conclusion and Future Work AVDL: a reduced yet extensible Audio-Visual Description Language –descriptors, properties, structures –XML syntax and DL semantics –.NET implementation and APIs About structure validation: –which constructors used ? which semantics ? Trade-of expressivity vs calculability –OWL Full is undecidable –constraints satisfaction problems can be complex