CSC 8610 & 5930 Multimedia Technology Lecture 6 Digital Video Representation

2 Today in class (3/19) 6:15Recap, Reminders 6:20Lecture – Digital Audio Repr. (cont’d) 6:40Lecture – Digital Video Representation 7:30Break 7:45Digital Video Repr. (cont’d) 8:00Project 1 status, discussion 8:15Audacity discussion & demos? 9:00Wrap

3 BASIC TERMINOLOGY

4 Lower Field vs Upper Field

5 Interlaced vs. Progressive Scan Interlaced – Developed for CRT (Cathode Ray Tube) technology – Divides scans into odd and even lines – Alternately refreshes odd lines, then even lines – Slight delay between refreshes causes “jaggedness” or INTERLACE ARTIFACTS – DEINTERLACING can compensate somewhat Progressive – Computer monitors – Scans entire picture line by line – Eliminate flicker seen in interlaced

6 Deinterlace To remove the interlace artifact Common method: – discard one field – fill in the gaps by duplicating or interpolating the other field

7 Deinterlaced Result

8 Sampling and Quantization of Motion Temporal: – sampling rate: how frequent you take a snapshot of the motion frame rate higher sampling rate: higher frame rate higher frame rate  more frames for the same duration  larger file size

9 Sampling and Quantization of Motion Each snapshot: a frame – an image – digitized based on the same concepts of sampling and quantization of images

10 Counting Time in Digital Video Timecode: to number frames SMPTE (Society of Motion Pictures and Television Engineers) video timecode – number frames in hours, minutes, seconds, and frames – drop-frame timecode – non-drop-frame timecode

11 Frame Size Resolution of the frame image Measured in pixel dimensions No ppi setting: Unlike digital images, there is no pixel per inch (ppi) setting for video because video is not intended for print but for on screen display.

12 Frame Size Examples Frame size NTSCstandard definition 720 x 480 pixels high definition HDV format 1280 x 720 pixels 1440 x 1080 pixels PALstandard definition 720 x 576 pixels

13 Frame Aspect Ratio the ratio of a frame's viewing width to height NOT equivalent to ratio of the frame’s pixel width to height.

14 Frame Aspect Ratio Examples 4:316:9 Example: Standard definition NTSC standard format Examples: Standard definition NTSC wide-screen format High definition digital video High definition TV

15 VIDEO IMAGE WILL BE DISTORTED IF IT IS DISPLAYED ON A SYSTEM WITH A DIFFERENT PIXEL ASPECT RATIO.

16 Distortion Pixel Apect RatiosDistortion video frame's = display system'snone video frame's < display system'sstretched horizontally video frame's > display system'sstretched vertically

17 VIDEO STANDARDS AND COLOR MODELS

18 Digital Video Standards Standard definition High definition Digital Television

19 The Term DV Common simple abbreviation for digital video DV compression and DV format: specific types of digital video compression and format respectively In this context – DV refers to the specific types of digital video – DV is NOT used to abbreviate digital video

20 High Definition HDV Format Picture Format720/25p, 720/30p, 720/50p, 720/60p 1080/50i 1080/60i Pixel Dimensions 1280   1080 Frame Aspect Ration16:9 Pixel Aspect Ratio Data RateVideo data only:approx. 19 Mbpsapprox. 25 Mbps Color Sampling MethodYUV 4:2:0 Audio SettingSampling rate and bit depth: 48 kHz 16-bit Bit rate after compression: 384 kbps

21 HDV Picture Format Notation 1080/60i frame height frame/field frequency "i": interlaced "p": progressive

22 Frame Size (Resolution) Comparison between Standard Definition and High Definition By viewing frame size

23 Frame Size (Resolution) Comparison between Standard Definition and High Definition By pixel dimensions

24 Frame Size (Resolution) Comparison between Standard Definition and High Definition A frame from a 1080i video

25 Frame Size (Resolution) Comparison between Standard Definition and High Definition Same frame as 720p

26 Frame Size (Resolution) Comparison between Standard Definition and High Definition Same frame as standard definition DV wide-screen (16:9)

27 Frame Size (Resolution) Comparison between Standard Definition and High Definition Same frame as standard definition DV standard 4:3

28 Digital Television (DTV) Signals of DTV are broadcast and transmitted digitally Need a digital TV set to watch

29 Digital Television (DTV) Standard definition – 704  480, 16:9 and 4:3, progressive and interlaced – 640  480, 4:3, progressive and interlaced High definition – 1920  1080, 16:9, progressive and interlaced – 1280  720, 16:9, progressive and interlaced – MPEG-2

30 VIDEO FILE TYPES

31 Common Video File Types File TypeAcronym ForOriginally Created By File Info & Compression Platforms.movQuickTime movieApple Also audio-only Can be streamed "Fast start" Common compression methods: H.264, Sorenson Video, Animation Apple QuickTime player, which is available for Mac and Windows.aviAudio Video Interleave IntelCommon compression methods: Microsoft RLE, Intel Indeo Video Primarily used on Windows but Apple QuickTime player can play AVI files.mpg.mpeg MPEGMotion Picture Experts Group For DVD-video High definition HDV Cross-platform

32 Common Video File Types File TypeAcronym ForOriginally Created By File Info & Compression Platforms.flvFlash VideoAdobe Progressive download Can be streamed Common compression methods: H.264, Sorenson Spark, On2 VP6 Cross-platform Requires Adobe Media Player to play.rmReal VideoReal System Can have very high degree of compression Choose the compression level based on network connection speed Can be streamed Cross-platform Requires Real player to play.wmvWindows MediaMicrosfotRequires Windows Media Player to play

33 Considerations for File Type File size restriction Intended audience Future editing

34 File Size Restriction For Web: – high compression – streaming video CD-ROM or DVD-ROM playback: – use data rate that can be handled by your target audience's computer DVD-video: – MPEG-2

35 Intended audience Multiple platforms – cross-platform formats: Apple QuickTime, MPEG, Flash video, Real Video How your target audience is going to watch your video?

36 Future Editing If the video will be used as a source for future editing: – Lower compression level – Choose uncompressed, if the frame size is small the video duration is extremely short you have enough disk space

37 Digital Video File Size Optimization Video tends to have very large file size compared to other media. Why should we care file size optimization? – A large file requires more disk space. – A large file takes longer to transfer. – Data transfer can be expensive (because data plans are not unlimited) – High data rate may cause choppy playback of the video. (Data rate will be explained later in this lecture.)

38 Effect of File Size vs. Data Rate on Video Playback Data rate: – If high: choppy playback – Amount of data to be processed per second Larger file size can have a low data rate if it is a long video Smaller file size can have a high data rate if it is a short video File size: – If high: Requires larger storage space Not unnecessary choppy playback – The impact of file size on smoothness of playback also depends on the video duration.

39 General Strategies for Reducing Video Data Rate Basic ideas: A video is a sequence of images + audio Apply strategies for reducing digital image and audio file size.

40 General Strategies for Reducing Video File Size General Strategies for reducing digital image file size – reduce frame size – reduce frame rate – choose a video compressor that allows higher compression – choose the lower picture quality option Reduce duration of the video so you have less frames – not always possible – will not impact data rate

41 Why General Strategies for Reducing Digital Image File Size Work for Video Reduce frame size because: – you have less pixels for each frame Reduce frame rate because: – you have less frames Video compression with high compression because: – some data are discarded Lower picture quality option because: – some data are discarded

42 Strategies Least Used for Reducing Video Data Rate Reduce bit depth – Not all video formats support lower bit depth – Live videos need 24-bit to look natural – Some compressors do not support lower bit depth Reduce sampling rate, bit depth, and channel numbers of the audio – size of the audio is insignificant compared to that of the picture component in a video

43 COMPRESSION OF VIDEO

44 Compression Basic idea: Want to represent the same content by using less data

45 Compression and Decompression Compression: – To reduce file size – Takes time – Often takes more time for higher compression Decompression: – A compression video file must be decompressed before it is played. – The decompression method or algorithm depends on how it is originally compressed.

46 Compression and Decompression Compression and decompression always go together as a pair. Codec: compressor/decompressor

47 Types of Compression Methods Spatial compression Temporal compression Lossless vs. lossy compression Symmetrical and asymmetrical compression

48 Spatial Compression Compact individual frames as if they are independent digital images Examples of algorithms: – Run-length encoding (RLE) – JPEG compression Example codecs: – QuickTime Animation – QuickTime PlanarRGB – Microsoft RLE

49 Spatial Compression Types of video that spatial compression is good for: – contain large areas of solid colors, such as cartoon animation Disadvantage: – Less compressed, i.e., relatively large file size compared to other types of compression

50 Temporal Compression Exploits the repetitious nature of image content over time in video Saving more information for selected frames, i.e. less compressed. These are called key frames. All other frames stores only the difference from the previous key frame, instead of full frame Advantage: – Effective if the change between a frame and its previous key frame is small

51 Temporal Compression Compressed well for: – video that contains continuous motion Not compressed well for: – video with frequent flickering and scene changes Example codecs that use temporal compression: – H.264 – Sorenson Video

52 Lossy vs. Lossless Compression Lossy compression: Reduce data by discarding or altering some of the original data Lossless compression: Preserve the original data but reduce file size by encoding the data specially

53 Lossy Compression Usually much smaller file size than lossless compression Lower picture quality Often try to maintain perceptual quality when deciding what data are to be discarded Discarded data cannot be recovered

54 Lossless Compression Usually much larger file size than lossy compression Example codecs: – QuickTime Animation – PlanarRGB (set at the maximum quality setting)

55 Symmetrical and Asymmetrical Compression Symmetrical codec: Same amount of time in compression and decompression Asymmetrical codec: – Amount of time to compress and decompress are significantly different – Preferable: Fast decompression so less wait time to play back the video

56 MPEG

57 MPEG Moving Pictures Experts Group Committee who derives standards for encoding video Allow high compression MPEG-1, MPEG-2, MPEG-4

58 What Happend to MPEG-3? NOT MP3 (which is audio format) Intended for HDTV HDTV specifications was merged into MPEG-2

59 MPEG-1 Video quality comparable to VHS Originally intended for Web and CD-ROM playback Frame sizes up to 352  240 pixels Video format for VCD (VideoCD) before DVD became widespread

60 MPEG-2 Supports DVD-video, HDTV, HDV standards For DVD video production: Export video into DVD MPEG-2 format For HDV video production: Export video into HDV's MPEG-2 format

61 MPEG-4 Newer standard of MPEG family Different encoding approach from MPEG-1 and MPEG-2 (will discuss after MPEG-1 and MPEG-2 compression in this lecture)

62 Applications of MPEG-4 Cover a wide range of data rate Low end of the data rate: Video playback on mobile devices High end of the data rate: – HDTV – Handheld and portable game devices (e.g., Sony PSP)

63 WAYS OF PLAYING VIDEO

64 Two Ways of Playing Video Play from disk Play over a network

65 Play from Disk An entire clip needs to be on disk before it can be played Played from hard drive, CD, or DVD

66 Play over a Network The video can be played while it is being downloaded Can be played from disk Streaming video Progressive download

67 Streaming Video Play video as soon as enough data has arrived Examples: – Streaming QuickTime – Real Video – Window Media Video (WMV)

68 Streaming Video Require a streaming server to stream video Allow saving several different compression levels of a video in a single file The server chooses the compression level to match the speed of network connection Buffering: Wait time depends on network speed

69 Progressive Download Play video as soon as enough data has arrived Does not require special servers Example: – QuickTime fast-start Created by saving the QuickTime movie as self-contained using QuickTime Pro