Ch 4 : Multimedia Database Science and Technology Faculty Informatics Arini, ST, MT Com

I. Multimedia Data Multimedia data : – From the presentation view point, multimedia data is huge & involves time dependent /independent characteristics. Presentation & subsequent interactions – Temporal : video, audio, animation – Spatial : image – Complex structure demands deriving semantics from contents » Need for Content-Based Access : color, texture, shape – Collaborative Support Environment – Consisting of Alphanumeric : text, hypertext Image : bitmap graphic (2D) and vector graphic (3D) Video : analog (videocassette, laserdisc) and digital (Quick time, real player, win player ) Animation : 2D, 3D (virtual reality : modeling object) Audio : digital audio

II. Multimedia Data Management Multimedia data management system is to allow efficient storage, manipulation using of multimedia data in all its varied forms. – Storage, retrieval, integration and presentation requirements of multimedia data differ significantly from those for the traditional data. Implemented to : Multimedia Database – Content Multimedia : MM-DBMS

III. Multimedia Database Management System (MM-DBMS) A framework that manages different types of data potentially represented in a wide diversity of formats on a wide array of media sources – To provide a suitable environment for using and managing multimedia database information MM-DBMS must include : – The traditional DBMS functions (e.g., database definition and creation, data retrieval, data access and organization, data independence, privacy, integration, Integrity control, version control and concurrency support but applied to various multimedia data types. – The functional requirements imposed on a MMDBMS can be grouped into two categories : Data representation requirements Data manipulation requirements.

4.1. MM-DBMS Functional Requirements Data representation requirements – Support for Generalization/Specialization Hierarchy (Book; Newspaper) – Attribute Specification (Title, author, number) – Ordering of Documents – The presentation of the paragraphs, images, and drawings of a multimedia document could depend on the users accessing the document. – Data independence Separate the database and the management from the application programs Data manipulation requirements. – Integration Data items do not need to be duplicated for different programs – Concurrency control allows concurrent transactions 5

– Data manipulation requirements. – Persistence Data objects can be saved and re-used by different transactions and program invocations – Privacy Access and authorization control – Integrity control Ensures database consistency between transactions – Recovery Failures of transactions should not affect the persistent data storage – Query support Allows easy querying of multimedia data – Flexible Acquisition – Efficient Storage – Efficient Retrieval MM Database6

4.2. Data Structure Multimedia data Raw Data Registering Data Descriptive Data (Uncompressed Image) (Size & coding details of raw data) (Textual numerical annotations) MediaRawRegisteringDescriptive TextCharactersCoding scheme (ASCII), length / end symbol Key words, information for structuring ImagesPixelsHeight/ Width of picture, Mode of Compression, if JPEG, tables for quantization purpose Pic.Date = 21/04/07 Pic.Reason = Birthday Etc Motion Video Sequence PixelsFrames/second, coding details, frame types… Scene description AudioSample sequence Audio coding (PCM,…), resolution of samples Content of audio passages in short form

4.3. Data Model Traditional : – Relational Data Model Store everything as First Normal Form tables Access, Oracle, MySQL Semantic : – Object Oriented Data Model Store everything as classes of objects ODL, OQL – Object-relational Data Model Fundamentally relations but are not First Normal Form table form POSTGRES

4.4. MM Database Architectures Based on Principle of Autonomy Each media type is organized in a media-specific manner suitable for that media type Need to compute joins across different data structures Relatively fast query processing due to specialized structures The only choice for legacy data banks MM Database9

MM Database Architectures (cont.) Based on Principle of Uniformity A single abstract structure to index all media types Abstract out the common part of different media types (difficult!) - metadata One structure - easy implementation Annotations for different media types MM Database10

MM Database Architectures (cont.) Based on Principle of Hybrid Organization A hybrid of the first two. Certain media types use their own indexes, while others use the "unified" index An attempt to capture the advantages of the first two Joins across multiple data sources using their native indexes MM Database11

Organizing Multimedia Data Based on the Principle of Uniformity Consider the following statements about media data and they may be made by a human or may be produced by the output of an image/video/text content retrieval engine. – The image photol.gif shows Jane Shady, “Big Spender” and an unidentified third person, in Sheung Shui. The picture was taken on January 5, – The video-clip videol.mpg shows Jane Shady giving “Big Spender” a briefcase (in frames ). The video was obtained from surveillance set up at Big Spender’s house in Kowloon Tong, in October, – The document bigspender.txt contains background information on Big Spender, a police’s file. MM Database12

Querying SMDSs (Uniform Representation) Querying SMDS based on top of SQL. Basic functions include: FindType(Obj): This function takes a media object Obj as input, and returns the output type of the object. For example, FindType(iml.gif) = gif. FindType(moviel.mpg) = mpg. FindObjWithFeature(f): This function takes a feature f as input and returns as output, the set of all media objects that contain that feature. For example, FindObjWithFeature(john)= {iml.gif,im2.gif,im3.gif,videol.mpg:[1,5]}. FindObjWithFeature(mary)= {videol.mpg:[1,5],videol.mpg:[15,50]}. MM Database13

4.5. Data Retrieval : 4 Types 1. Conventional database system – This is the widely-used approach to manage and search for structured data. – All data in a database system must conform to some predefined structures and constraints (i.e., schema’s). – To formulate a database query the user must specify which data objects are to be retrieved, the database tables from which they are to be extracted and predicate on which the retrieval is based. – A query language for the database will generally be of the artificial kind, one with restricted syntax and vocabulary, such as SQL.

4.5. Data Retrieval : 4 Types 2. Information retrieval (IR) system : – IR system is mainly used to search large text collections, in which the content of the (text) data is described by an indexer using keywords or a textual abstract, and keywords or natural language is used to express query demands. Use Keyword Querying – For example for an image or video we have to describe it in words or in a way need to store lot of metadata (textual form).

4.5. Data Retrieval : 4 Types 3. Content based retrieval (CBR) system – This approach is used to retrieve desired multimedia objects from a large collection on the basis of features (such as color, texture and shape, etc.) that can be automatically extracted from the objects themselves. Use Visual querying and semantic querying – Although keyword can be treated as a ”feature” for text data, traditional information retrieval has much more higher performance than content-based retrieval because keyword has the proven ability to represent semantics, while no features have shown convincing semantic describing ability. – But major drawback of this method is that it lacks precision.

4.5. Data Retrieval : 4 Types 4. Graph or tree pattern matching : – This approach aims to retrieve object sub- graphs from an object graph according to some denoted patterns.

V. 2 Search Types Metric Combinatorial : image, audio (music), video

VI. MM-DBMS Structure Data Analysis – Data structure (4.2 above) Data Modeling – Data Model (4.3 above) Data Storage – DBMS Architecture (4.4 above) Data Retrieval : – 4 Types data retrieval (depend on query and searching method) (V. above) Multimedia Communication – Multimedia Networking