new trends in multimedia - MPEG-21: concepts and applications 1 Rik Van de WalleGhent, April 2008 Rik Van de Walle Multimedia Lab Ghent University, Belgium web: multimedialab.ugent.be MPEG-21: concepts and applications

new trends in multimedia - MPEG-21: concepts and applications 2 Rik Van de WalleGhent, April 2008 media representation standards some milestones Still images o JPEG, JPEG2000, GIF, TIFF,... CD-ROM storage o MPEG-1 Digital TV o MPEG-2 Video over PSTN and mobile networks (Public Switched Telephone Network) o H.263 Object-based AV coding o MPEG-4 State-of-the-art video coding o MPEG-4 AVC/H.264 o MPEG-21 Scalable Video Coding

new trends in multimedia - MPEG-21: concepts and applications 3 Rik Van de WalleGhent, April 2008 MPEG-1/2/4/7: technology for...

new trends in multimedia - MPEG-21: concepts and applications 4 Rik Van de WalleGhent, April 2008 MPEG-21: what is it about ? Started in October 1999 General goal o infrastructure for delivery and consumption of MM content o 'the big picture' Some more specific objectives o understand how the elements of the framework fit together o where gaps in the framework exist: identify new standards o integration of different standards within MPEG outside MPEG

new trends in multimedia - MPEG-21: concepts and applications 5 Rik Van de WalleGhent, April 2008 MPEG-21: what is it about ? A multimedia framework 'to enable transparent and augmented use of multimedia resources'

new trends in multimedia - MPEG-21: concepts and applications 6 Rik Van de WalleGhent, April 2008 some MPEG-21 applications 'A billion content providers for a billion users...' Some examples o digital libraries / archiving applications (e.g., interaction with OAI) o interactive TV / broadcasting o publishing o event reporting / trade transactions o e-health (e.g., content adaptation) o mobile applications / lightweight applications

new trends in multimedia - MPEG-21: concepts and applications 7 Rik Van de WalleGhent, April 2008 who is the MPEG-21 User ? Definition within MPEG-21: User o any entity interacting in the MPEG-21 environment o any entity using a Digital Item o including individuals, organizations, corporations, communities consortia, governments, standards bodies,... o User roles include content creation, content consumption content providers/distributors rights holders...

new trends in multimedia - MPEG-21: concepts and applications 8 Rik Van de WalleGhent, April 2008 MPEG-21 parts Currently: quite a lot of parts... o Technical Report: Vision, Technologies and Strategy o Digital Item Declaration o Digital Item Identification o Intellectual Property Management and Protection o Rights Expression Language o Rights Data Dictionary o Digital Item Adaptation o Reference Software

new trends in multimedia - MPEG-21: concepts and applications 9 Rik Van de WalleGhent, April 2008 MPEG-21 parts Currently: quite a lot of parts... o File Format o Digital Item Processing o Evaluation Tools for Persistent Association o Test Bed for MPEG-21 Resource Delivery o Scalable Video Coding o Conformance o Even Reporting o Binary Format

new trends in multimedia - MPEG-21: concepts and applications 10 Rik Van de WalleGhent, April 2008 declaring 'Digital Items' – why ? Currently, multimedia applications are based on transfer/processing/presentation/... of o different media types, with different representations still images (JPEG2000, GIF, PNG,...) video (MPEG-4, QuickTime, ASF,...) audio (WAV, MP3,...) text (txt, doc,...)... o metadata descriptive information about actual data DRM information (e.g., copyright statement) configuration information...

new trends in multimedia - MPEG-21: concepts and applications 11 Rik Van de WalleGhent, April 2008 declaring 'Digital Items' – why ? Example: a digital music album o without the concept of 'digital item declaration' a 'flat directory' containing a loose set of –audio files –lyrics files –a readme.txt file –a config.ini file ('to be used by advanced users only') –playTrack.exe (only executable on a single platform/OS) –... no 'structure'

declaring 'Digital Items' – structure aria title: Nessun Dorma track number: aria title: Nessun Dorma track number: nessunDorma.txt type: lyrics composer: Giacomo Puccini opera: Turandot copyright: Ricordi & co... nessunDorma.txt type: lyrics composer: Giacomo Puccini opera: Turandot copyright: Ricordi & co... nessunDorma.mp3 type: audio format: mp3 duration: 200 s bitrate: 192 kbps copyright: EMI... nessunDorma.mp3 type: audio format: mp3 duration: 200 s bitrate: 192 kbps copyright: EMI... aria title: O mio babbino caro track number: aria title: O mio babbino caro track number: babbinoCaro.doc type: lyrics composer:Giacomo Puccini opera: Gianni Schicci copyright: DECCA... babbinoCaro.doc type: lyrics composer:Giacomo Puccini opera: Gianni Schicci copyright: DECCA... babbinoCaro.wav type: audio format: wav duration: 170 s bitrate: 128 kbps copyright: DECCA... babbinoCaro.wav type: audio format: wav duration: 170 s bitrate: 128 kbps copyright: DECCA... ACA01039.jpg type: album cover art format: image/jpeg size: 300x400 copyright: EMI... ACA01039.jpg type: album cover art format: image/jpeg size: 300x400 copyright: EMI... title: concert recording date: July 2003 location: Covent Garden... title: concert recording date: July 2003 location: Covent Garden... concert.mov type: concert video type: video/mov duration: 4500 s bitrate: 500 kbps size: 320x240 copyright: DECCA... concert.mov type: concert video type: video/mov duration: 4500 s bitrate: 500 kbps size: 320x240 copyright: DECCA...

new trends in multimedia - MPEG-21: concepts and applications 13 Rik Van de WalleGhent, April 2008 declaring 'Digital Items' – MPEG-21 solution basic concepts A Digital Item is... o structured, with a standard representation, identification and metadata o fundamental unit of distribution and transaction o expressed by Digital Item Declaration Language (DIDL) ISO/IEC XML schema Terminology o resourceindividual assets o metadatainfo about or pertaining to the DI o structurerelation between DI parts

new trends in multimedia - MPEG-21: concepts and applications 14 Rik Van de WalleGhent, April 2008 declaring 'Digital Items' – MPEG-21 solution basic concepts As a result, MPEG-21 DIs are a composition of o different resources audio video textual... o metadata describing the content (e.g., MPEG-7) terminal characteristics (e.g., MPEG-21 DIA) rights expressions (e.g., MPEG-21 REL)

new trends in multimedia - MPEG-21: concepts and applications 15 Rik Van de WalleGhent, April 2008 declaring 'Digital Items' – MPEG-21 solution basic concepts resources metadata MPEG-1 MPEG-2 MPEG-4 MPEG-7 new resource formats new metadata formats (e.g., registration) structure

MPEG-21 Digital Item Declaration basic concepts Digital Item Declaration Language (DIDL) o a generic container structure o a set of building blocks o expressed in XML o allows declaration of any Digital Item Descriptor Container Descriptor Item Component Resource Descriptor Item Component Resource

MPEG-21 DID example Prokofiev: Romeo and Juliet Valery Gergiev <Resource ref='Prokofiev_RnJ.mp3' type='audio/mp3'/> … Prokofiev: Romeo and Juliet Valery Gergiev <Resource ref='Prokofiev_RnJ.mp3' type='audio/mp3'/> …

new trends in multimedia - MPEG-21: concepts and applications 18 Rik Van de WalleGhent, April 2008 MPEG-21 DID Language (DIDL) building blocks DID technology is described in three normative sections o model describes set of abstract terms and concepts a Digital Item is the digital representation of 'a work' a DI is the thing that is acted upon within the model DIs are –managed/handled/processed –described –exchanged –collected –...

new trends in multimedia - MPEG-21: concepts and applications 19 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks o representation DID elements are represented in XML normative description of their syntax and semantics o schema normative XML schema comprising entire grammar of the DID

new trends in multimedia - MPEG-21: concepts and applications 20 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks item o grouping of sub-items and/or components that are bound to relevant descriptors o descriptors contain information about the item o may contain choices allow items to be customized or configured o may be conditional o NOTE item that contains no sub-items can be considered a logically indivisible work item that does contain sub-items can be considered a compilation

new trends in multimedia - MPEG-21: concepts and applications 21 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks container o structure allowing items and/or containers to be grouped o groupings of items and/or containers can be used to form logical packages (for transport or exchange) logical shelves (for organization)

new trends in multimedia - MPEG-21: concepts and applications 22 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks (example)....

new trends in multimedia - MPEG-21: concepts and applications 23 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks component o binding of a resource to its descriptors o descriptors will typically contain control or structural information about the resource bit rate character set start points encryption information o NOTE a component itself is not an item components are building blocks of items

new trends in multimedia - MPEG-21: concepts and applications 24 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks descriptor o associates information with the enclosing element o can be a component –thumbnail of an image –textual component a textual statement resource o individually identifiable asset a video or audio clip images, textual asset o must be locatable via an unambiguous address

new trends in multimedia - MPEG-21: concepts and applications 25 Rik Van de WalleGhent, April 2008 MPEG-21 DIDL building blocks statement o literal textual value o contains information, but not an asset o examples: descriptive info control info revision tracking info

MPEG-21 DIDL building blocks (example) Image Image

new trends in multimedia - MPEG-21: concepts and applications 27 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Identification Scope: how to o uniquely identify Digital Items and parts thereof (including resources) o uniquely identify IP related to the Digital Items and parts thereof o uniquely identify Description Schemes o use identifiers to link Digital Items with related information such as descriptive metadata o identify different types if Digital Items

new trends in multimedia - MPEG-21: concepts and applications 28 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Identification DII specification does NOT o specify new identification systems for content elements for which identification and description schemes already exist does not attempt to replace the ISRC (as defined in ISO 3901) for sound recordings but allows ISRCs to be used within MPEG-21 Identifiers can be associated with Digital Items o by including them in a Statement element

new trends in multimedia - MPEG-21: concepts and applications 29 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Identification – example MPEG-21 Music Album backcover.jpg cover.jpg covernotes.html track1.aac metadata track2.aac metadata track2.aac metadata lyrics1.txt metadata lyrics2.txt metadata Identifier: ISRC FR-Z Descriptions: lead singer composer concert hall conductor Identifier: ISMN: M Descriptions: composer lyrics writer language

new trends in multimedia - MPEG-21: concepts and applications 30 Rik Van de WalleGhent, April 2008 MPEG-21 IPMP Defines how to manage and protect Digital Items o taking IPMP in MPEG-4 as its starting point o standardizes how IPMP tools can be retrieved from remote locations o exchange of messages between IPMP tools and the terminal Management and protection of o rights expressions o Digital Item Adaptation descriptions o...

new trends in multimedia - MPEG-21: concepts and applications 31 Rik Van de WalleGhent, April 2008 MPEG-21 REL and RDD REL RDD based on XrML 2.0 based on 2rdd consortium of companies MPEG-21 REL and RDD are closely alligned standards in 2003

new trends in multimedia - MPEG-21: concepts and applications 32 Rik Van de WalleGhent, April 2008 MPEG-21 REL and RDD Rights Expression Language (REL) o machine-readable language o can declare rights and permissions o using the terms as defined in the Rights Data Dictionary Rights Data Dictionary (RDD) o provides a set of clear, consistent, structured, and integrated definitions of terms o terms may be used in the REL o provides framework for interoperability of terms used in management of rights and permissions

new trends in multimedia - MPEG-21: concepts and applications 33 Rik Van de WalleGhent, April 2008 simple/single digital objects complex digital objects LANL Use Case archiving/publishing complex digital objects

new trends in multimedia - MPEG-21: concepts and applications 34 Rik Van de WalleGhent, April 2008 LANL Use Case MPEG-21 DIDL – Data Model

new trends in multimedia - MPEG-21: concepts and applications 35 Rik Van de WalleGhent, April 2008 LANL Use Case MPEG-21 DIDL – Descriptors Syntax for Semantic Enriching of Web Pages Jeremy J. Carroll Metadata, Dublin Core element OCLC Online Computer Library Center, Inc. text/html en RDF = Resource Description Framework dc = Dublin Core (metadata standard for library aplications)

new trends in multimedia - MPEG-21: concepts and applications 36 Rik Van de WalleGhent, April 2008 LANL Use Case MPEG-21 DIDL – Rights … Copyright2003; American Physical Society … MPEG-21 r:license

new trends in multimedia - MPEG-21: concepts and applications 37 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Adaptation Description schemes o describing the environment in which digital items are used o user preferences o accessibility requirements o network and terminal characteristics o consumption environment (e.g., lighting conditions) Resource Adaptation Digital Item Adaptation tools o Session Mobility (see further) o (generic) Bitstream Syntax Description Language ((g)BSDL)

new trends in multimedia - MPEG-21: concepts and applications 38 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Adaptation Descriptor Adaptation Engine Resource Adaptation Engine Adapted Digital Item Digital Item

new trends in multimedia - MPEG-21: concepts and applications 39 Rik Van de WalleGhent, April 2008 Digital Item Processing – why ? Scope of Digital Item Processing o declaration of a Digital Item defines 'structure' of a DI static declaration –what happens when a DI arrives at a terminal ? –nothing o Digital Item Methods/Processing allow Users to add functionality to a Digital Item Declaration o on receipt of a DID list of DI Methods that can be applied to the Digital Item is presented to the User User chooses a Method which is then executed (poor) analogy: DVD menus

new trends in multimedia - MPEG-21: concepts and applications 40 Rik Van de WalleGhent, April 2008 Digital Item Processing – why ? Digital Item Methods o provide a way for DI Users of the DI to specify a selection of preferred procedures by which the DI should be handled at the level of the DI itself o example: music album DI 'AddTrack' DIM o NOT intended to be utilized for implementing the processing of media resources themselves

new trends in multimedia - MPEG-21: concepts and applications 41 Rik Van de WalleGhent, April 2008 Digital Item Processing – MPEG-21 solution basic concepts 'Methods' vs. 'Operations' o DIMs are defined as 'a list of operations' o Digital Item Base Operations (DIBOs) describe base operations on which DIMs are built analogy –DI Method = programming language –DIBOs = standard library of functions

new trends in multimedia - MPEG-21: concepts and applications 42 Rik Van de WalleGhent, April 2008 Digital Item Processing – MPEG-21 solution basic concepts DIBOs o specify high level normative interface to the basic functionality of the MEPG ‑ 21 Peer Digital Item Method Language (DIML) o specifies normative language for defining interoperable DIMs and from which the DIBOs are able to be called Digital Item extension Operations (DIxO) o specify normative mechanism for enabling more complex functionality in an efficient way

relationship DID – other parts of MPEG-21

new trends in multimedia - MPEG-21: concepts and applications 44 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Processing building blocks One DIML has been chosen: ECMAScript One set of normative operations: DIBOs o implemented in any language any scripting language any 'full' programming language only APIs are normatively defined o will be atomic operations o having access to MultiMedia Middleware API (see further) o having ECMAScript bindings DIBOs can be called from within DIMs o having DIxO bindings DIxOs can be called from within DIMs DIxOs can call DIBOs as well

new trends in multimedia - MPEG-21: concepts and applications 45 Rik Van de WalleGhent, April 2008 MPEG-21 Digital Item Processing building blocks DIxOs have access to o all normative DIBOs o other DIxOs DIBO Extension Language: o multiple solutions possible o already 1 solution identified: MPEG-J-based normative annex on how to call MPEG-J-based DIxOs informative annex on the execution environment for MPEG-J-based DIxOs o include complex methods efficiently o downloadable DIxOs are callable from DIML

Digital Item Method DIBO MultiMedia Middleware API (M3W API) MPEG-21 Digital Item Processing how does it work ?

Digital Item Method DIBO ECMAScript native language normative language normative API & semantics non-normative implementation

Digital Item Method DIBO DIxO MultiMedia Middleware API (M3W API) MPEG-21 Digital Item Processing how does it work ?

Digital Item Method DIBO ECMAScript identified platform native language DIxO normative language normative API & semantics normative platform non-normative API normative DIxO invocation non-normative implementation

POPE: Publish Once, Play Everywhere

new trends in multimedia - MPEG-21: concepts and applications 51 Rik Van de WalleGhent, April 2008 application – overview digital media - multimedia o bits and bytes o MPEG - Moving Picture Experts Group mobile multimedia applications o problem statement o session mobility o standardized solution: MPEG-21 demo: prototype application (during lab visit) o streaming video o lightweight: PDA as a terminal (Personal Digital Assistant)

new trends in multimedia - MPEG-21: concepts and applications 52 Rik Van de WalleGhent, April 2008 introduction The old analogue times - TV paradigm o video data modeled as a sequence of pictures with a certain number of lines o one audio channel added to the video signal o video and audio have an analogue representation o user chooses among the available broadcast programs o programs are consumed 'as is'

new trends in multimedia - MPEG-21: concepts and applications 53 Rik Van de WalleGhent, April 2008 mobile applications - Session Mobility Problem statement o increasing demand to access multimedia anywhere at any time o new powerful multimedia devices different terminal capabilities different network capabilities o increasing demand for easy switching between different devices without complex user interaction i.e., 'session mobility' o 'going mobile' is a key objective

new trends in multimedia - MPEG-21: concepts and applications 54 Rik Van de WalleGhent, April 2008 mobile applications - Session Mobility student watching a lesson on a set- top box student continuing the session on a PDA finishing the e- learning session on a tablet PC tv/set-top box Pocket PC Session Transfer Media stream Tablet PC Servers containing online classes

new trends in multimedia - MPEG-21: concepts and applications 55 Rik Van de WalleGhent, April 2008 SM protocols simple Device A o collects information about the current session o stops the current session location of current media stream (e.g., the URI) position in current media stream (e.g., 50 sec) status of the current session (e.g., pause) status of the application (e.g., full screen) Device A transfers session data to device B Device B o reconstructs session for the received data load the media stream go to the correct position reconstruct the status of the session reconstruct the status of the application

new trends in multimedia - MPEG-21: concepts and applications 56 Rik Van de WalleGhent, April 2008 SM protocols simple dead period end of session on device A start of session on device B

new trends in multimedia - MPEG-21: concepts and applications 57 Rik Van de WalleGhent, April 2008 SM protocols advanced Modified protocol to remove the dead period o device A collects information about current session does not stop the current session o device A transfers session data to device B o device B reconstructs session for the received data requests updates of the session information until both sessions are synchronized o device A stops its session

new trends in multimedia - MPEG-21: concepts and applications 58 Rik Van de WalleGhent, April 2008 SM protocols advanced

new trends in multimedia - MPEG-21: concepts and applications 59 Rik Van de WalleGhent, April 2008 SM and heterogeneous devices Different terminal capabilities o cause difficulties for transferring sessions o example watching a movie encoded at CIF (352x288) on a QCIF (176x144) display Different network capabilities o example playing a movie encoded at 512 kbps on a device with a 128 kbps network connection Requirement o common (proprietary or standardized) format o for storage of session information Solution: MPEG-21 technology

new trends in multimedia - MPEG-21: concepts and applications 60 Rik Van de WalleGhent, April 2008 MPEG-21 Session Mobility introduction Provides a solution to the problems for session transfer between heterogeneous devices o differences in capabilities including choices that allow the configuration of a Digital Item o common format for the session information standardized in Digital Item Adaptation specification

new trends in multimedia - MPEG-21: concepts and applications 61 Rik Van de WalleGhent, April 2008 prototype application architecture

new trends in multimedia - MPEG-21: concepts and applications 62 Rik Van de WalleGhent, April 2008 prototype application implementation platforms Client A – Windows platform o.Net Framework (C#) o Pentium 4, 2.8 GHz running Windows XP o Windows Media Player 9 Client B – Windows Mobile platform o.Net Compact Framework (C#) o iPAQ 5550, Intel XScale 400 MHz running Windows Mobile 2003 o Windows Media Player 9 for Pocket PC

new trends in multimedia - MPEG-21: concepts and applications 63 Rik Van de WalleGhent, April 2008 prototype application performance measurements 4 different measurements o parsing a Digital Item o generating a Session Mobility Digital Item o parsing a Session Mobility Digital Item o transferring and resuming a session

new trends in multimedia - MPEG-21: concepts and applications 64 Rik Van de WalleGhent, April 2008 prototype application performance measurements Processing cost is insignificant on both platforms o parsing a Digital Item average iPAQ ms o generating a Session Mobility Digital Item average iPAQ 14.4 ms o parsing a Session Mobility Digital Item average iPAQ 63.1 ms o transferring and resuming a session average iPAQ ms

new trends in multimedia - MPEG-21: concepts and applications 65 Rik Van de WalleGhent, April 2008 conclusions MPEG-21 offers very generic framework for setting up and managing advanced multimedia applications Session Mobility between devices with different capabilities o interoperability is a key requirement o MPEG-21 Session Mobility is offering such interoperability o by standardizing a format for session information Overhead introduced by MPEG-21 o not significant compared to the consumption of the actual multimedia o MPEG-21 can easily be implemented on lightweight platforms