Video Digital Multimedia, 2nd edition Nigel Chapman & Jenny Chapman Chapter 7 This presentation © 2004, MacAvon Media Productions Video Digital Multimedia, 2nd edition Nigel Chapman & Jenny Chapman Chapter 7 This presentation © 2004, MacAvon Media Productions

Data Size PAL uncompressed 768x576 pixels per frame 191–192 Data Size PAL uncompressed 768x576 pixels per frame x 3 bytes per pixel (24 bit colour) x 25 frames per second ≈ 31 MB per second ≈ 1.85 GB per minute Data Size PAL uncompressed 768x576 pixels per frame x 3 bytes per pixel (24 bit colour) x 25 frames per second ≈ 31 MB per second ≈ 1.85 GB per minute

Data Size NTSC uncompressed 640x480 pixels per frame 191–192 Data Size NTSC uncompressed 640x480 pixels per frame x 3 bytes per pixel (24 bit colour) x 30 frames per second (approx) ≈ 26 MB per second ≈ 1.6 GB per minute Data Size NTSC uncompressed 640x480 pixels per frame x 3 bytes per pixel (24 bit colour) x 30 frames per second (approx) ≈ 26 MB per second ≈ 1.6 GB per minute

Digitization In the camera – DV + Firewire 193–195 Digitization In the camera – DV + Firewire In the computer – video capture card Digitization in camera (DV) means less noise Less noise allows better compression Digitization In the camera – DV + Firewire In the computer – video capture card Digitization in camera (DV) means less noise Less noise allows better compression

197 Streamed Video Play back a video stream as it arrives over a network (like broadcast TV), instead of downloading an entire video clip and playing it from disk (like renting a DVD) Streamed Video Play back a video stream as it arrives over a network (like broadcast TV), instead of downloading an entire video clip and playing it from disk (like renting a DVD)

198 HTTP Streaming Start playing a downloaded clip as soon as enough of it has arrived Starts when the (estimated) time to download the rest is equal to the duration of the clip HTTP Streaming Start playing a downloaded clip as soon as enough of it has arrived Starts when the (estimated) time to download the rest is equal to the duration of the clip

198 X axis, Time Y axis, % remaining. Starting at 100% on Y axis download starts and runs at a 45 degree angle to the finish at 0% at distance along the X axis. A horizontal line parallel to X axis the proceeds from the 100% mark until the start playing time. At which point we have a line descendinfinsh at the same point on the time lineg at an angle of 26 degrees to the vertical to

Video Standards Digital video devices must conform to standards 199 Video Standards Digital video devices must conform to standards Digital standards must maintain compatibility with older analogue standards for broadcast TV Video Standards Digital video devices must conform to standards Digital standards must maintain compatibility with older analogue standards for broadcast TV

Interlacing Required for TV, so encountered in captured footage 200 Interlacing Required for TV, so encountered in captured footage Each frame is divided into two fields Field 1: odd lines; Field 2: even lines Fields are transmitted one after the other Frame is built out of the interlaced fields Interlacing Required for TV, so encountered in captured footage Each frame is divided into two fields Field 1: odd lines; Field 2: even lines Fields are transmitted one after the other Frame is built out of the interlaced fields

Analogue Standards PAL (Phase Alternating Line) 199 Analogue Standards PAL (Phase Alternating Line) Western Europe, Australia & New Zealand, China,… NTSC (National Television Standards Committee) North America, Japan, Taiwan, parts of South America,… Analogue Standards PAL (Phase Alternating Line) Western Europe, Australia & New Zealand, China,… NTSC (National Television Standards Committee) North America, Japan, Taiwan, parts of South America,…

Analogue Standards SECAM (Séquential Couleur avec Mémoire) 199 Analogue Standards SECAM (Séquential Couleur avec Mémoire) France and former Soviet Union Standard only used for transmission Uses PAL cameras etc Analogue Standards SECAM (Séquential Couleur avec Mémoire) France and former Soviet Union Standard only used for transmission Uses PAL cameras etc

PAL NTSC NTSC Frame has 625 lines, 576 are picture 202 PAL Frame has 625 lines, 576 are picture 25 frames (50 fields) per second NTSC Frame has 525 lines, 480 are picture 29.97 frames (59.94 fields) per second (Often quoted as 30 frames per second) PAL Frame has 625 lines, 576 are picture 25 frames (50 fields) per second NTSC Frame has 525 lines, 480 are picture 29.97 frames (59.94 fields) per second (Often quoted as 30 frames per second)

CCIR 601 Digital video standard, properly called Rec. ITU-R BT.601 202–203 CCIR 601 Digital video standard, properly called Rec. ITU-R BT.601 720 luminance samples (Y), 2x360 colour difference samples (B−Y and R−Y) per line PAL 720x576 pixels; NTSC 720x480 pixels Pixels are not square CCIR 601 Digital video standard, properly called Rec. ITU-R BT.601 720 luminance samples (Y), 2x360 colour difference samples (B−Y and R−Y) per line PAL 720x576 pixels; NTSC 720x480 pixels Pixels are not square

4:2:2 sub-sampling Used in CCIR 601 203 4:2:2 sub-sampling Used in CCIR 601 Twice as many Y samples as each of the colour difference samples Co-sited: same pixel is used for all three samples Reduces data rate to just over 20MB per second 4:2:2 sub-sampling Used in CCIR 601 Twice as many Y samples as each of the colour difference samples Co-sited: same pixel is used for all three samples Reduces data rate to just over 20MB per second

DV Consumer format, also known as mini-DV 204 DV Consumer format, also known as mini-DV DVCAM, DVPRO use different tape formats, but generate the same data stream 4:1:1 chrominance sub-sampling Data rate constant 25Mbits per second Compression ratio 5:1 DV Consumer format, also known as mini-DV DVCAM, DVPRO use different tape formats, but generate the same data stream 4:1:1 chrominance sub-sampling Data rate constant 25Mbits per second Compression ratio 5:1

DV sub-sampling PAL DV 4:2:0 chrominance sub- sampling 210

DV sub-sampling NTSC DV 4:1:1 chrominance sub-sampling 210

MPEG ISO/IEC Motion Picture Experts Group 204–206 MPEG ISO/IEC Motion Picture Experts Group Series of standards including MPEG-1 intended for video CD MPEG-2 used in DVD and broadcast MPEG-4 for low bitrate multimedia ISO/IEC Motion Picture Experts Group Series of standards including MPEG-1 intended for video CD MPEG-2 used in DVD and broadcast MPEG-4 for low bitrate multimedia

204–205 MPEG Profiles & Levels Profiles define subsets of the features of the data stream Levels define parameters such as frame size and data rate Each profile may be implemented at one or more levels Notation: profile@level, e.g. MP@ML MPEG Profiles & Levels Profiles define subsets of the features of the data stream Levels define parameters such as frame size and data rate Each profile may be implemented at one or more levels Notation: profile@level, e.g. MP@ML

MPEG-2 Profiles & Levels 205 MPEG-2 Profiles & Levels MPEG-2 Main Profile at Main Level (MP@ML) used for DVD video CCIR 601 scanning 4:2:0 chrominance sub-sampling 15 Mbits per second Most elaborate representation of MPEG-2 compressed data MPEG-2 Profiles & Levels MPEG-2 Main Profile at Main Level (MP@ML) used for DVD video CCIR 601 scanning 4:2:0 chrominance sub-sampling 15 Mbits per second Most elaborate representation of MPEG-2 compressed data

205–206 MPEG-4 Designed to support a range of multimedia data at bit rates from 10kbps to >1.8Mbps Applications from mobile phones to HDTV Video codec becoming popular for Internet use, is incorporated in QuickTime, RealMedia and DivX MPEG-4 Designed to support a range of multimedia data at bit rates from 10kbps to >1.8Mbps Applications from mobile phones to HDTV Video codec becoming popular for Internet use, is incorporated in QuickTime, RealMedia and DivX

MPEG-4 Profiles & Levels 205 MPEG-4 Profiles & Levels Visual Simple Profile (SP), suitable for low bandwidth streaming over Internet Visual Advanced Simple Profile (ASP) suitable for broadband streaming SP@L1 (Level 1 of Simple Profile), 64 kbps, 176x144 pixel frame ASP@L5, 8000 kbps, full CCIR 601 frame MPEG-4 Profiles & Levels Visual Simple Profile (SP), suitable for low bandwidth streaming over Internet Visual Advanced Simple Profile (ASP) suitable for broadband streaming SP@L1 (Level 1 of Simple Profile), 64 kbps, 176x144 pixel frame ASP@L5, 8000 kbps, full CCIR 601 frame

Video Compression Spatial (intra-frame) compression 206–208 Video Compression Spatial (intra-frame) compression Compress each frame in isolation, treating it as a bitmapped image Temporal (inter-frame) compression Compress sequences of frames by only storing differences between them Always some compression because of sub- sampling Video Compression Spatial (intra-frame) compression Compress each frame in isolation, treating it as a bitmapped image Temporal (inter-frame) compression Compress sequences of frames by only storing differences between them Always some compression because of sub-sampling

Spatial Compression Image compression applied to each frame 207 Spatial Compression Image compression applied to each frame Can therefore be lossless or lossy, but lossless rarely produces sufficiently high compression ratios for volume of data Lossless compression implies a loss of quality if decompressed then recompressed Ideally, work with uncompressed video during post-production Spatial Compression Image compression applied to each frame Can therefore be lossless or lossy, but lossless rarely produces sufficiently high compression ratios for volume of data Lossless compression implies a loss of quality if decompressed then recompressed Ideally, work with uncompressed video during post-production

Temporal Compression Key frames are spatially compressed only 207–208 Temporal Compression Key frames are spatially compressed only Key frames often regularly spaced (e.g. every 12 frames) Difference frames only store the differences between the frame and the preceding frame or most recent key frame Difference frames can be efficiently spatially compressed Temporal Compression Key frames are spatially compressed only Key frames often regularly spaced (e.g. every 12 frames) Difference frames only store the differences between the frame and the preceding frame or most recent key frame Difference frames can be efficiently spatially compressed

Motion JPEG Purely spatial compression Apply JPEG to each frame 209–210 Motion JPEG Purely spatial compression Apply JPEG to each frame Used by most analogue capture cards No standard, but MJPEG-A format widely supported Motion JPEG Purely spatial compression Apply JPEG to each frame Used by most analogue capture cards No standard, but MJPEG-A format widely supported

DV Compression Starts with chrominance sub-sampling of CCIR 601 frame 210–211 DV Compression Starts with chrominance sub-sampling of CCIR 601 frame Constant data rate 25Mbits per second Higher quality than MJPEG at same rate Apply DCT, quantization, run-length and Huffman coding on zig-zag sequence – like JPEG – to 8x8 blocks of pixels DV Compression Starts with chrominance sub-sampling of CCIR 601 frame Constant data rate 25Mbits per second Higher quality than MJPEG at same rate Apply DCT, quantization, run-length and Huffman coding on zig-zag sequence – like JPEG – to 8x8 blocks of pixels

210–211 DV Compression If little or no difference between fields (almost static frame), apply DCT to block containing alternate lines from odd and even fields If motion between fields, apply DCT to two 8x4 blocks (one from each field) separately, leading to more efficient compression of frames with motion DV Compression If little or no difference between fields (almost static frame), apply DCT to block containing alternate lines from odd and even fields If motion between fields, apply DCT to two 8x4 blocks (one from each field) separately, leading to more efficient compression of frames with motion

DV Compression Shuffling 210–211 DV Compression Shuffling Construct video segments by taking 8x8 blocks from five different areas of the frame, to ‘average’ amount of detail Calculate coefficients for whole video segment, making more efficient use of available bytes DV Compression Shuffling Construct video segments by taking 8x8 blocks from five different areas of the frame, to ‘average’ amount of detail Calculate coefficients for whole video segment, making more efficient use of available bytes

211–212 MPEG-1 Compression Spatial compression based on quantization and coding of DCT coefficients Temporal compression based on motion compensation Record displacement of object plus changed pixels in area exposed by its movement MPEG-1 Compression Spatial compression based on quantization and coding of DCT coefficients Temporal compression based on motion compensation Record displacement of object plus changed pixels in area exposed by its movement

MPEG-1 Compression I-pictures purely intra-frame compressed 212–213 MPEG-1 Compression I-pictures purely intra-frame compressed P-pictures ‘predictive’ Difference frames based on earlier I- or P- pictures B-pictures ‘bi-directionally predictive’ Difference frames based on preceding and following I- or P-pictures MPEG-1 Compression I-pictures purely intra-frame compressed P-pictures ‘predictive’ Difference frames based on earlier I- or P-pictures B-pictures ‘bi-directionally predictive’ Difference frames based on preceding and following I- or P-pictures

MPEG-1 Compression Group of Pictures (GOP) 213–214 MPEG-1 Compression Group of Pictures (GOP) Repeating sequence of I-, P- and B-pictures Always begins with an I-picture Display order – frames in order they will be displayed Bitstream order – re-ordered so that every P- or B-picture comes after frames it depends on, allowing reconstruction of the complete frames MPEG-1 Compression Group of Pictures (GOP) Repeating sequence of I-, P- and B-pictures Always begins with an I-picture Display order – frames in order they will be displayed Bitstream order – re-ordered so that every P- or B-picture comes after frames it depends on, allowing reconstruction of the complete frames

MPEG-1 Compression Source Input Format (SIF) 4:2:0 chroma sub-sampled 214 MPEG-1 Compression Source Input Format (SIF) 4:2:0 chroma sub-sampled 352x240 pixel frame MPEG-1 compressed SIF video at 30 frames per second has data rate of 1.86Mbits per second (CD video – 40mins of video at that rate) MPEG-1 can be scaled up to larger frames, but cannot handle interlacing MPEG-1 Compression Source Input Format (SIF) 4:2:0 chroma sub-sampled 352x240 pixel frame MPEG-1 compressed SIF video at 30 frames per second has data rate of 1.86Mbits per second (CD video – 40mins of video at that rate) MPEG-1 can be scaled up to larger frames, but cannot handle interlacing

215 MPEG-4 Standard defines an encoding for multimedia streams made up of different sorts of object – video, still images, animation, 3-D models… Higher profiles divide a scene into arbitrarily shaped video objects each one may be compressed and transmitted separately, scene is composed at the receiving end by combining them SP and ASP restricted to rectangular objects, usually complete frames MPEG-4 Standard defines an encoding for multimedia streams made up of different sorts of object – video, still images, animation, 3-D models… Higher profiles divide a scene into arbitrarily shaped video objects each one may be compressed and transmitted separately, scene is composed at the receiving end by combining them SP and ASP restricted to rectangular objects, usually complete frames

MPEG-4 Compression Refinement of MPEG-1 compression 215 MPEG-4 Compression Refinement of MPEG-1 compression I-pictures compressed by quantizing and Huffman coding DCT coefficients Improved motion compensation leads to better quality than MPEG-1 at same bit rates MPEG-4 Compression Refinement of MPEG-1 compression I-pictures compressed by quantizing and Huffman coding DCT coefficients Improved motion compensation leads to better quality than MPEG-1 at same bit rates

MPEG-4 Compression Simple Profile P-pictures only 215–216 MPEG-4 Compression Simple Profile P-pictures only Efficient decompression, suitable for PDAs etc Advanced Simple Profile (ASP) B-pictures Global Motion Compensation Sub-pixel motion compensation MPEG-4 Compression Simple Profile P-pictures only Efficient decompression, suitable for PDAs etc Advanced Simple Profile (ASP) B-pictures Global Motion Compensation Sub-pixel motion compensation

216–219 Older Codecs Cinepak – Longest established, high compression ratio, takes much longer to compress than to decompress Intel Indeo – Similar to Cinepak, but roughly 30% faster compression Sorenson – More recent, higher quality and better compression ratios than other two All three based on vector quantization Quality of all three inferior to MPEG-4 Older Codecs Cinepak – Longest established, high compression ratio, takes much longer to compress than to decompress Intel Indeo – Similar to Cinepak, but roughly 30% faster compression Sorenson – More recent, higher quality and better compression ratios than other two All three based on vector quantization Quality of all three inferior to MPEG-4

216 Vector Quantization Divide each frame into small rectangular blocks (’vectors’) Code Book – collection of constant vectors representing typical patterns (edges, textures, flat colour,…) Compress by replacing each vector in image by index of vector from code book that most closely resembles it Vector Quantization Divide each frame into small rectangular blocks (’vectors’) Code Book – collection of constant vectors representing typical patterns (edges, textures, flat colour,…) Compress by replacing each vector in image by index of vector from code book that most closely resembles it

Editing Making a constructed whole from a collection of parts 223–230 Editing Making a constructed whole from a collection of parts Selection, trimming and organization of raw footage Apply transitions (e.g. dissolves) between shots Combination of picture with sound No changes made to the footage itself Editing Making a constructed whole from a collection of parts Selection, trimming and organization of raw footage Apply transitions (e.g. dissolves) between shots Combination of picture with sound No changes made to the footage itself

Post-Production Changing or adding to the material 230–236 Post-Production Changing or adding to the material Most changes are generalizations of image manipulation operations (e.g. colour correction, blurring and sharpening,…) Compositing – combining elements from different shots into a composite sequence Animating elements and combining animation with live action Post-Production Changing or adding to the material Most changes are generalizations of image manipulation operations (e.g. colour correction, blurring and sharpening,…) Compositing – combining elements from different shots into a composite sequence Animating elements and combining animation with live action

Preparing for Delivery 236–237 Preparing for Delivery Compromises required to bring resource requirements of video within capabilities of delivery media (e.g. networks) and low-end machines Reduce frame size (e.g. downsample to quarter frame) Reduce frame rate (12fps is OK for smooth motion, flicker not a problem on computer) Reduce colour depth Preparing for Delivery Compromises required to bring resource requirements of video within capabilities of delivery media (e.g. networks) and low-end machines Reduce frame size (e.g. downsample to quarter frame) Reduce frame rate (12fps is OK for smooth motion, flicker not a problem on computer) Reduce colour depth