2. CGMB 324: MULTIMEDIA SYSTEM DESIGN Chapter 07: Multimedia Element V – Video Part II

3. Objectives Upon completing this chapter, you should be able to:  understand some of the common image and video processing techniques  understand the basic concepts of video  apply video in a multimedia system

4. Image & Video Processing Techniques

5. Video & Still Image Processing  Image processing = process of manipulating a bitmap image so that the image can be enhanced, restored, distorted, or analyzed.  Methods:  Pixel Point-to-point processing – operations on one pixel at a time. Does not change the location of pixel. E.g. Pix_out (x,y) = Pix_in (x,y) (operator) C C is a constant and operator is ADD, SUB, DIV etc.

6. Video & Still Image Processing  Methods:  Histogram sliding – change the overall visible effect of brightening or darkening the image.  A constant is ADDED or SUBTRACTED from all pixels in an image.  A histogram is constructed from a frequency table.  The pixel intensities are shown on the X-axis and the number of occurences of each intensity is represented by the Y-axis

7. Histogram x Axis - Pixel Intensity (0-255) y Axis - Number Of Pixels

8. Video & Still Image Processing  Histogram Stretching/Shrinking – increase or decrease the contrast by multiply or divide each pixel value by a constant.  Pixel Threshold – To set a limit on the dark and bright areas of a picture.

9. Inter-Frame Image Processing  Inter-frame image processing:  Operates on two images Pix_Out(x,y) = ( Image1(x,y) ) Operator ( Image2(x,y) ) where: Operator = arithmetic or logical operations such as ADD, SUB, MUL, DIV, AND, OR, XOR)

10. Inter-Frame Image Processing  Methods:  Image Averaging – minimizes or cancels the effects of random (Gaussian) Noise.  Successive image frames are added on a pixel-by- pixel basis and the result is divided by the number of frames to get the average value of the pixel.

11. Inter-Frame Image Processing  Methods:  Image Subtraction – determine the change from one frame to the next for image comparisons for key frame detection or motion detection.  Can also be used to get rid of common background.  If two frames have very little similarity (below a threshold level), then the new frame is a key frame.

12. Basics of Video

13.  Analog video is represented as a continuous (time varying) signal.  Digital video is represented as a sequence of digital images.

14. Types of Color Video Signals  Component video  A video signal that has been split into 2 or more components  Each primary is sent as a separate video signal.  Does not carry audio  The primaries can either be RGB or a luminance-chrominance transformation of them (e.g., YIQ, YUV).  Carry information in 3 separate signal – 1 luma and 2 chroma component  Best color reproduction  Requires more bandwidth and good synchronization of the three components

15. Types of Color Video Signals  Composite video  Carry picture only – does not carry audio  color (chrominance) and luminance signals are mixed into a single carrier wave.  Some interference between the two signals is inevitable.  S-Video (Separated video, e.g., in S-VHS)  a compromise between component analog video and the composite video.  It uses two lines, one for luminance and another for composite chrominance signal.

16. Video Cables Composite Video (RCA) S-Video Component Video

17. HDMI Cable  The High-Definition Multimedia Interface (HDMI) is a licensable compact audio/video connector interface for transmitting uncompressed digital streams, which include s Radio Frequency (RF) coaxial cable, composite video, S-Video, component video, D-Terminal, and VGA.  HDMI connects DRM-enforcing digital audio/video sources such as a set-top box, an HD DVD disc player, a Blu-ray Disc player, a personal computer, a video game console, or an AV receiver to a compatible digital audio device and/or video monitor such as a digital television (DTV).  HDMI began to appear in 2006 on consumer HDTV camcorders and high-end digital still cameras

18. NTSC Video  NTSC (National Television Standards Committee)  20 lines reserved for control information at the beginning of each field (2 fields)  So a maximum of 485 lines of visible data  Laserdisc and S-VHS have actual resolution of ~420 lines  Ordinary TV -- ~320 lines

19. NTSC Video  525 scan lines per frame  352 x 240 resolution  30 frames per second (or be exact, 29.97 fps, 33.37 msec/frame)  Interlaced, each frame is divided into 2 fields, 262.5 lines/field  uses the YIQ color model.

20. PAL Video  PAL (Phase Alternating Lines)  625 scan lines per frame  352 x 288 resolution  25 frames per second (40 msec/frame)  Interlaced, each frame is divided into 2 fields, 312.5 lines/field  Uses YUV color model  Used in Great Britain and parts of Europe, Africa, Asia, and the Middle East

21. Digital Video  Advantages:  Direct random access  good for non-linear editing (NLE)  No problem for repeated recording & playback  Ease of making indefinite number of exact copies (mass production)  Longer lasting than analog video (not really subject to the same level of degradation)

22. ATSC Digital Television Standard  (ATSC -- Advanced Television Systems Committee)  The ATSC Digital Television Standard was recommended to be adopted as the Advanced TV broadcasting standard by the FCC Advisory Committee on Advanced Television Service on November 28, 1995.  It covers the standard for HDTV (High Definition TV).  Will replace NTSC television system

23. ATSC Digital Television Standard  The video scanning formats supported by the ATSC Digital Television Standard are shown in the following table. Vertical LinesHorizontal PixelsAspect RatioPicture Rate 1080192016:960I 30P 24P 720128016:960P 30P 24P 48070416:9 & 4:3 60I 60P 30P 24P 4806404:3 60I 60P 30P 24P

24. ATSC Digital Television Standard  The aspect ratio for HDTV is 16:9 as opposed to 4:3 in NTSC, PAL, and SECAM.  A 33% increase in horizontal dimension.  In the picture rate column, the "I" means interlaced scan, and the "P" means progressive (non-interlaced) scan.  Both NTSC rates and integer rates are supported (i.e., 60.00, 59.94, 30.00, 29.97, 24.00, and 23.98).

25. ATSC Digital Television Standard  At 1920 x 1080, 60I (which CBS and NBC have selected), there will be 1920 x 1080 x 30 = 62.2 million pixels per second.  Considering 4:2:2 chroma subsampling, each pixel needs 16 bits to represent it, the bit rate is 62.2 x 16 / 8= 124.375 MB/sec (approx. 1 Gb/sec)

26. Interlaced & Progressive  Standard 480-line NTSC TV broadcasts, VCR, DVD and laserdisc signals are sent in an "Interlaced Scan" format.  A TV screen first draws the image's odd lines, one at a time sequentially from top to bottom (which takes 1/60 of a second), and then fills in the even lines (taking another 1/60 of a second).  That is, the full picture (top to bottom) is first drawn with half its information hollowed out, and then the other half is filled in -- the entire process taking 1/30 of a second.

27. Interlaced & Progressive  A newer and superior scanning method called "Progressive" permits the entire picture to be drawn sequentially from top to bottom without the odd/even interlacing.  Some newer DVD players now have outputs for both an interlaced and progressive scan image.  HDTV signals are now being broadcast in both progressive and interlaced formats:  720p (720 lines of resolution in progressive scan format)  1080i (interlaced).

28. Applying Video In Multimedia Systems

29. Applying Video In MM Systems  Video changed the very nature of entertainment when it first went mainstream.  Not only could we hear things, but we could also see it.  Apart from that, it provided a window into the everyday lives of other people.  Video opened the door to a huge film industry which capitalizes on our desires and wants – fantasy, action, adventure, horror etc. - things that we could not dream of in reality, but can easily enjoy through video.

30. Applying Video In MM Systems  Inherently, in a contest between images and video, the latter usually wins hands down as well.  People would much rather see a video of a person doing something than a just a picture of it.  How can we use this knowledge to work for us?  How can multimedia systems benefit from the element of video and where should it be applied?

31. Video is used for practically everything people wish (and would pay) to see, including sci-fi, fantasy, adventure, horror and sex. Applying Video In MM Systems

32.  Because video is arguably the most taxing multimedia element in a computer system, it must be used wisely and sparingly.  Firstly, we must eliminate all the other multimedia elements before deciding to use video.  For example, why show a 25 MB video interview of a person, when a single good image, Q&A text, and perhaps a short audio sample of the person’s voice would suffice?  Collectively, the space and resource requirements will be less than video.

33. Applying Video In MM Systems  Also, why use two pages of text, some animation and several diagrams to show how to perform a magic trick, when a short, clear video (including inherent audio explanation within the video), would describe it more clearly and quickly?  There are no fixed rules when deciding on this matter. It depends on a number of things such as the subject matter, audience, system requirements etc.  Generally, the point is to find a compromise which gives us the highest quality, lowest system requirements and gets our message across as quickly and as clearly as possible.

34. Applying Video In MM Systems – Quality Concerns  Never assume that video will be appreciated over images or any other element on the grounds that it is simply video.  Poor quality video suffers from the same discrimination that poor quality (low resolution) images do.  Any video worth posting (using in the system) has to be of exceptional quality wherever possible.  If the characters are blur, scan lines are visible, or the sound is too soft or poor (this can sometimes be improved), you might as well not use it.

35. Applying Video In MM Systems – Quality Concerns  This is why companies behind software such as RealOne player and Windows Media Video strive to improve their video compression technology so as to deliver near DVD quality streaming content.  Quicktime also delivers high quality video with its Sorenson 3 codec. You often see this applied in downloadable movie trailers.  Even other websites which offer video for download, use the quality of the video they deliver as grounds to charge a fee – and people nowadays are willing to pay for high quality video.

36. Applying Video In MM Systems – Psychology  We must also not forget the role of psychology in video.  Ask yourself, what is it about a video download that would make you want to take the time to download it and then, sit through viewing it?  Is there anything a website (for example) could do, to encourage you to want to see the video it has available?  Perhaps providing (as a hyperlink) a carefully selected thumbnail image of an interesting scene from the video might help.

37. Applying Video In MM Systems – Psychology  The old adage, “bigger is better” holds true for many things; especially video.  Psychologically, if the video is large enough (has a high resolution, such as 352x240 – NTSC VCD standard as a minimum), users will appreciate it more than a small one, like 240x180.  This, of course, depends on whether the large video is clear and that it does not suffer from the ‘blocky effect’.

38. The video screenshot on the left (NTSC, 352x240, 29.97 fps), is of reasonable quality and acceptable size for a multimedia application. Notice how the text in the background is clear enough to be read. The one on the right (PAL, 352x288, 25 fps), is also an acceptable size for a multimedia system, but unfortunately, the quality isn’t very good. There is no point in making the video this size, if the quality isn’t good. In fact, making it bigger, only degrades the quality further.