Naresuan University Multimedia Paisarn Muneesawang

Naresuan University What is Multimedia? Apps that involve more than conventional data types (e.g., text, drawing and images) Best examples are continuous media (e.g., animations, audio, and video) –Called continuous media because of time basis

Naresuan University Why is Multimedia? Human brain is much more efficient at processing and interpreting visual and audio information –than text, text+graphics Human beings and technology developments –image: 2-dimensional data –video: 3-dimensional data+audio –virtual reality

Naresuan University Challenges Multimedia data size Real-time nature of multimedia Why is the semantic nature of multimedia data a problem?

Naresuan University Multimedia Data Size

Naresuan University The First Challenge is Size A single good quality colored image could require 6 Mb. A video object consist of a sequence of such images (called frames) will be very large. –30 frames per second, a five minute video clip would require 54 Gb. A typical sequence of audio will occupy 8 Kb for each second. Data size will effect the storage, retrieval and transmission of multimedia. Thus data compression techniques are crucial.

Naresuan University Multimedia Data Acquisition Data Capture Loss Sampling Predict/ Model TransformSend DelayReorder ReceiveRestoreDisplay

Naresuan University Digitization of Data First Step: capturing analogue signal. Second Step: sampling the signal and convert it to digital value, –Theory of sampling by Nyquist: sampling rate >= (2)*(BW ) to accurately reproduce an analog signal. Sampling rate  data size

Naresuan University Multimedia System Design Should be designed based on human visual and audio sensory systems: –The human voice is in the frequency range 0-4kHz requiring 8kHz rate for digitization. –But, the intelligible part of human speech is carried in the voice band ( Hz). Thus, we can use smaller bandwidth. Should be designed based on applications: –Image in medical application may require higher resolution than images transmitted on the web application.

Naresuan University Reducing the Media Object’s Size Compression is the technique to remove redundant data. –For example, by abbreviating any repeated information in an image and eliminating information that is difficult for the human eye to see. Compression/decompression process –Lossy compression  some information is loss by the process. –Lossless compression  non of the original information is lost. Many compression methods attempt to address the same objectives: –Reduced BW and/or storage; –Decode signal should be as close as possible to the original; –Lowest possible implementation strategy; –Application to as many signal types as possible; –Robustness; –Scalability; –extensibility

Naresuan University Media Data and Format Standards Media dataSpecial natureTypical digital file formats Text dataASCII, RTF, HTML, SGML, XML Audio dataTime dependence with digitized data size can affect quality WAV, AU, MPEG-MP3 Image dataResults from pictures, drawings and photographs JPEG, GIF, TIFF, BMP, PNG Video dataTime-dependent sequence of video frames AVI, FLI, GIF, JPEG, MPEG

Naresuan University How to Choose the Formats Multimedia data is available in a range of file formats. Although this variety could appear confusing, this gives the designer great flexibility. We may select a format that: –Suit the application and the requirements of different users. This computer-generated graphic image can be stored in several different file formats. bit map: 1088 Kb JPEG: 748 Kb GIF: 109 Kb. Note: JPEG is designed for photographs so it is not necessarily the best format for computer graphic files.

Naresuan University Digital File Formats, Applications and Features Digital video format ApplicationFeatures MPEGMultimediaKey frame plus motion encoding INDEOWeb pagesWavelet progression CENEPAKComputerVideo efficient SORENSONComputer/videophoneVideo efficient QUICKTIMEComputerIncorporates compression AVIComputerIncorporates compression

Naresuan University Important File Formats Image: JPEG is a standard of the ISO and can operate in four modes: sequential, progressive, lossless, and hierarchical. High quality compression for audio and video is the MPEG standards of the Moving Picture Expert Group. –MPEG-1: Interactive CDROM, video CDROM, Audio-MP3 –MPEG-2: Digital TV, DVD, Audio-Music –MPEG-4: Multimedia for fixed and mobile web. (for low bandwidth applications) –MPEG-7: Search video and audio content –MPEG-21: Multimedia framework for interoperability

Naresuan University How Large is a Video Object Time640x480320x240160x120 Compressed JPEG 25:1 640x480 Compressed MPEG 100:1 640x480 1 s27 Mb6.75 Mb Mb270 Kb 1 min1.6 Mb400 Mb10065 Mb16 Mb 1 h97 Gb24 Gb6 Gb3.9 Gb970 Mb 1000 h97 Tb24 Tb6 Tb

Naresuan University Real-Time Nature of Multimedia

Naresuan University The Real-Time Nature of Multimedia Video and audio are continuous periodic media: –The frames of the video must run in the correct sequence and at an acceptable rate, otherwise it becomes meaningless. The effect of time to the media objects is referred to as the real-time nature of multimedia. This relationship with time will have significant for the way the media objects are stored, retrieved, transmitted and synchronized.

Naresuan University Video Data Video is a continuous media but for database storage and manipulation such as random access it is important to be able to deal with portions of video object. Video Segmentation―cutting long video into portions: shot, scene, and clip –Shot define a low level syntactic building blocks of video sequence. –Scene is the logical grouping of shots into semantic unit. –Clip is not clearly defined so it can last from a few seconds to several hours.

Naresuan University Video Segmentation

Naresuan University Story Clip Scene Shot a Shot b

Naresuan University Organization of Video Data

Naresuan University Why is the Semantic Nature of Multimedia Data a Problem?

Naresuan University Why is the Semantic Nature of Multimedia Data a Problem? “A picture is worth a thousand words” Subjectivity: Different people have difference opinions. Multimedia data, instead of having an explicit nature, have an implicit semantic nature.

Naresuan University Describing Multimedia Content Traditional database systems: Metadata is know as: –Schema definitions, indexes, users, integrity constraints, security constraints etc Information that is of interest to the system itself. But, in multimedia database management systems (MMDBMS) metadata can refer to information about individual objects. Metadata = Characteristics of the media data: –Texture for images; –Frequencies for audio; –Speech ― keywords such as identification of the speakers, place, time; –Video clip ― camera motion and lighting

Naresuan University Techniques for Describing Multimedia Image annotation: a simple text description (keywords) –Time consuming for a large image database –Difficult to describe an image precisely in text Content-based technique: An image is described by specific features in the image, such as color or texture. –Description is generated automatically –Description is the content itself (low-level feature)

Naresuan University Image Content with Low-Level Description

Naresuan University Metadata for Multimedia Text metadata + content descriptors A new standard MPEG-7 will have a major impact on the issue of metadata. MPEG-7 multimedia contents include: –Low-level descriptions of each individual object in a scene, such as shape, color, and movement; –High-level abstract descriptions of the scene, the objects it contains and the events taking place; –Audio information such as key, mood and tempo. MPEG-7 makes the process of searching and retrieving an image or a video clip much easier.

Naresuan University Key-Word Search

Naresuan University Content-Based Retrieval

Naresuan University Content-Based Retrieval