1. Digital Media Dr. Jim Rowan ITEC 2110 Video Part 2

2. Coping with Video Size Consider human vision limitations 1) Chrominance sub-sampling Compression - two versions 2) Spatial 3) Temporal differencing vectoring

3. Coping with Video Size Consider human vision limitations 1) Chrominance sub-sampling Compression - two versions 3) spatial 4) Temporal differencing vectoring

4. Chrominance sub-sampling Humans can’t distinguish changes in color as well as they can distinguish luminance changes –http://en.wikipedia.org/wiki/Chroma_subsamplinghttp://en.wikipedia.org/wiki/Chroma_subsampling In our cameras… –Of every 4 frames –store the luminance for each frame –only store a proportion of the color info –4:2:0

5. http://en.wikipedia.org/wiki/Chroma_subsampling#Sampling_systems_and_ratios Chrominance sub-sampling Luminance, Cr, Cb

6. Aside from screen size and frame rate... Consider human vision limitations 1) Chrominance sub-sampling Compression - two versions 2) spatial 3) Temporal differencing vectoring Coping with Video Size

7. Spatial compression Individual images can be compressed using the techniques discussed in the bitmapped section Doesn’t result in very much compression for video Doesn’t take into consideration the other frames that come before or after it Our video cameras do this, compressing each frame to jpeg

8. Aside from screen size and frame rate... Consider human vision limitations 1) Chrominance sub-sampling Compression - two versions 2) spatial 3) Temporal differencing vectoring Coping with Video Size

9. Temporal Compression differencing Use the Difference in two frames –A naive approach can result in good compression –Works well for a small amount of movement Security cameras spend most of their time “seeing” the same thing all night long –A Tarantino film? not so much… Most pixels change with nearly every frame Captured Frame 1 Captured Frame 2 Saved Frame 1 Saved Frame 2

10. Image Differencing To subtract one image from the next Do it one pixel at a time –red minus red –green minus green –blue minus blue Store the difference To play it back –Play frame one 1 –Add frame 2 to frame 1 Next, an example in black & white

11. Example 1, the difference of two identical images

12. The result ===>

13. Example 2, the difference of two similar images

14. The result ===>

15. Temporal Compression vectoring When an OBJECT moves –compute its trajectory –fill in the resulting exposed background –BUT there’s a problem... –why isn’t this an easy thing to do? Captured Frame 1 Captured Frame 2 Stored Object Movement Vector Stored Background

16. More on differencing The differencing can happen in a forward manner and a backward manner It might be more economical (in data size) to create a frame from a frame that follows it...

17. MPEG-2 iFrame pFrame bFrame GOP http://en.wikipedia.org/wiki/Inter_frame http://en.wikipedia.org/wiki/I- frames#Intra_coded_frames_.28or_sli ces_or_I-frames_or_Key_frames.29

18. iFrame: -a keyframe -spatially compressed (a fully specified image) pFrame -predicted frame -contains only the difference between the current frame and the previous frame (smaller in size than iFrame) bFrame -bi-predicted frame -contains difference between current and both the preceding and following frames -even smaller in size than iFrame

19. 3 mpeg 2 video streams All spatially (intra-frame) compressed Spatially compressed and predictive (difference) Spatially compressed forward and backward predictive (difference) The largest Smaller… but more computation Smallest… but more computation and it is transported out of order! Play sequence: 1 2 3 4 5 6 7 8 9 10 Transmit sequence: 1 4 2 3 7 5 6 10 8 9 Group(s) Of Pictures (GOPs) http://www.spiritalchemy.com/blog/wp-content/uploads/2009/10/IPB_frame_example.jpg

20. So… How does this ===> happen?

22. Macroblocks It’s COMPLICATED… Images are broken up and transmitted as macroblocks They can be a variety of sizes –Typically 8X8 or 16X16 When missing (or there are video errors) –May use a lower resolution block (that was previously received) –May just use a solid color block instead

23. Video Compression What does this? Coder/Decoder - Codec –encodes and decodes video Can be symmetric it takes as long to compress as decompress Can be asymmetric it takes longer to compress or decompress than it does to decompress to compress http://en.wikipedia.org/wiki/Video_codec

24. Video Compression Can be very complex Each image is just jpeg compressed Each image is jpeg compressed; the p-frames are the result of frames being subtracted from the i-frames Each image is jpeg compressed; the p-frames are the result of frames being subtracted from the i-frames; b-frames are the result of frames being subtracted from i-frames and future p-frames

25. A final worry... Codecs compress video to make it smaller There are a variety of codecs to do this Which to choose? –It is a tradeoff between compression technique, its computational complexity and its artifacts

26. So... How do codecs vary? compression and decompression complexity –affects the artifacts that are created –affects the time required to carry them out –affects the volume of the data stream created –affects the type and expense of the equipment used –affects whether or not it can be implemented in hardware of software WMV, DivX, Cinepak, Intel Indeo & Sorenson

27. Which is the original? Notice the aritfacts? http://www.100fps.com/codec_quality_comparison.htm

28. That’s It!