MPEG-7 What is MPEG-7 ? MPEG-7 is a multimedia content description standard. These descriptions are based on catalogue (e.g., title, creator, rights), semantic (e.g., the who, what, when, where information about objects and events) structural (e.g., the colour histogram - measurement of the amount of colour associated with an image or the timbre of an recorded instrument) features of the AV content MPEG-7 describes AV data whose representation is defined by MPEG-1, 2 and 4. MPEG-7 uses XML Schema as the language of choice for content description
MPEG-7 What is XML? XML stands for EXtensible Markup Language XML is a markup language much like HTML XML is a text format, designed to describe data XML tags are not predefined in XML. You must define your own tags XML uses a Document Type Definition (DTD) or an XML Schema to describe the data XML with a DTD or XML Schema is designed to be self-descriptive
MPEG-7 XML Tags. the XML declaration - defines the XML version XML does not DO anything. XML is created to structure, store, and to send information within tags. <?xml version="1.0" encoding="ISO-8859-1"?> <note> <to>John</to> <from>Jani</from> <heading>Reminder</heading> <body>Don't forget me to ring me this weekend!</body> </note> the XML declaration - defines the XML version and the character encoding used in the document Tags of the XML file
MPEG-7 XML Attributes. Attributes must always be within quotes … and attributes. <?xml version="1.0" encoding="ISO-8859-1"?> <note date="12/11/99"> <to>John</to> <from>Jani</from> <heading>Reminder</heading> <body>Don't forget me this weekend!</body> </note>
MPEG-7 XML is free and extensible XML tags are not predefined. You must "invent" your own tags. The tags used to mark up HTML documents and the structure of HTML documents are predefined. The author of HTML documents can only use tags that are defined in the HTML standard (like <p>, <h1>, etc.). XML allows the author to define his own tags and his own document structure. The tags in the example above (like <to> and <from>), are not defined in any XML standard. These tags are "invented" by the author of the XML document.
MPEG-7 What can MPEG-7 do ? MPEG-7 describes multimedia content so users can search, browse, and retrieve that content more efficiently and effectively than they could using today’s mainly text-based search engines. It’s a standard for describing the features of multimedia content. Search Engine: Searching & filtering Classification Manipulation Summarization Indexing MPEG-7 Scope: Description Schemes (DSs) Descriptors (Ds) Language (DDL) Ref: MPEG-7 Concepts Feature Extraction: Content analysis (D, DS) Feature extraction (D, DS) Annotation tools (DS) Authoring (DS)
MPEG-7 What MPEG-7 does not do?: Not standardize the (automatic) extraction of AV descriptions/features. Not specify the search engine (or any other program) that can make use of the description. It will be left to the creativity and innovation of search engine companies to manipulate and massage the MPEG-7-described content into search indices that can be used by their browser and retrieval tools
MPEG-7 Anatomy of MPEG-7 MPEG-7 defines a multimedia library of methods and tools and will standardize: A set of descriptors: A descriptor (D) is a representation of a feature that defines the syntax and semantics of the feature representation. A set of description schemes: A description scheme (DS) specifies the structure and semantics of the relationships between its components, which may be both descriptors and description schemes. A Description Definition Language (DDL): A language that specifies description schemes. It also allows for the extension and modification of existing description schemes. MPEG-7 adopted XML Schema Language as the MPEG-7 DDL. One or more ways (textual, binary) to encode descriptions: A coded description is a description that’s been encoded to fulfil relevant requirements such as compression efficiency, error resilience, and random access.
MPEG-7 Audio Description Framework The audio description framework consists of: Sound effect description tools – descriptions of the temporal and spectral components of audio segments Instrumental timbre description tools – descriptions of the perceptual features of instrument sounds e.g. attack, brightness, richness Spoken content description - detailed description of words spoken within an audio stream Uniform silence segment – description lattice of combined word and phone lattices for each speaker in an audio stream Melodic descriptors - compact descriptions for melodic information, which allows for efficient and robust melodic similarity matching, for example, in query-by-humming
MPEG-7 Visual Basic Structures (5 types) Grid Layout: The grid layout is a splitting of the image into a set of equally sized rectangular regions 2D-3D Multiple View: 2D/3D Descriptor specifies a structure which combines 2D Descriptors representing a visual feature of a 3D object seen from different view angles. Time Series: defines a temporal series of descriptors in a video segment and provides image to video-frame matching and video-frames to video-frames matching functionalities. Spatial 2D Coordinates: defines two kinds of 2D spatial coordinate system: “local” (all images mapped into same position) and “integrated” (images mapped into different positions) Temporal Interpolation: describes a temporal interpolation using connected polynomials. This can be used to approximate an object position in a video.
MPEG-7 Visual Description Framework There are six types of visual descriptors: Colour - Descriptors of colour model, quantiser, dominant colour, histograms for groups of images or frames Texture - Descriptors of regularity, coarseness & directionality, the spatial distribution of different types of edges, Shape - Descriptors of object region-based shape, contour-based shape, 3D shape, and 2D-3D Multiple Views. Motion – Descriptors of camera motion, object motion trajectory, parametric object motion, and motion activity e.g high speed car chase Localization – Descriptions of enables localized regions within images or frames by specifying them with a scalable representation of a Box or a Polygon Face Recognition – Description of a face as a set of vectors
MPEG-7 Generic Description Framework Generic descriptions are features, which are used in audio, visual, and text descriptions, and therefore “generic” to all media e.g. “vector”, “time”, etc: Content description: representation of perceivable information Content management: information about the media features, the creation and the usage of the AV content; Content organization: representation the analysis and classification of several AV contents; Navigation and access: specification of summaries and variations of the AV content; User interaction: description of user preferences and usage history pertaining to the consumption of the multimedia material.
MPEG-7 Architecture The transport of the MPEG-7 data can occur on a variety of delivery systems. E.g. MPEG-2 Transport Streams, IP (Internet Protocol), or MPEG-4 (MP4) files or streams. The delivery layer encompasses mechanisms allowing synchronization, framing and multiplexing of MPEG-7 content. MPEG-7 content may be delivered independently or together with the content they describe.
MPEG-7 Architecture MPEG-7 Systems generic method for binary encoding, called BiM, a description (in textual XML form) can be compressed, partitioned, streamed, and reconstructed at terminal side. The BiM, in order to gain its compression efficiency, relies on a schema analysis phase where internal tables are computed to associate binary code to XML elements, types and attributes.
MPEG-7 Applications Digital libraries, Education (image catalogue, musical dictionary, Bio-medical imaging catalogues…) Multimedia editing (pers onalised electronic news service, media authoring) Cultural services (history museums, art galleries, etc.), Multimedia directory services (e.g. yellow pages, Tourist information, Geographical information systems) Broadcast media selection (radio channel, TV channel,…) Journalism (e.g. searching speeches of a certain politician using his name, his voice or his face), E-Commerce (personalised advertising, on-line catalogues, directories of e-shops,…) Surveillance (traffic control, surface transportation, non-destructive testing in hostile environments, etc.), Investigation services (human characteristics recognition, forensics), Home Entertainment (systems for the management of personal multimedia collections, including manipulation of content, e.g. home video editing, searching a game, karaoke,…) Social (e.g. dating services),