1. VIDEO COMPRESSION FUNDAMENTALS
Pamela C. Cosman
June 2, 2010

2. Compressing Digital Video
Exploit spatial redundancy within frames (like JPEG: transforming, quantizing, variable length coding)
Exploit temporal redundancy between frames
Only the sun has changed position between these 2 frames
Previous Frame
Current Frame

3. Simplest Temporal Coding - DPCM
Frame 0 (still image)
Difference frame 1 = Frame 1 – Frame 0
Difference frame 2 = Frame 2 – Frame 1
If no movement in the scene, all difference frames are 0. Can be greatly compressed!
If movement, can see it in the difference images

4. Difference Frames Differences between two frames can be caused by
Camera motion: the outlines of background or stationary objects can be seen in the Diff Image
Object motion: the outlines of moving objects can be seen in the Diff Image
Illumination changes (sun rising, headlights, etc.)
Scene Cuts: Lots of stuff in the Diff Image
Noise

5. Difference Frames
If the only difference between two frames is noise (nothing moved), then you won’t recognize anything in the Difference Image
But, if you can see something in the Diff Image and recognize it, there’s still correlation in the difference image
Goal: remove the correlation by compensating for the motion

9. Types of Motion
Frame n
Frame n+1 (Rotation)
Translation: simple movement of typically rigid objects
Camera pans vs. movement of objects
Frame n+2 (Zoom)
Rotation: spinning about an axis
Camera versus object rotation
Zooms –in/out
Camera zoom vs. object zoom (movement in/out)
Frame n
Frame n+1

10. Describing Motion Translational Rotational Zoom
Move (object) from (x,y) to (x+dx,y+dy)
Rotational
Rotate (object) by (r rads) (counter/clockwise)
Zoom
Move (in/out) from (object) to increase its size by (t times)
Which is easiest? Which are we most likely to encounter?

11. Motion Estimation Determining parameters for the motion descriptions
For some portion of the frame, estimate its movement between 2 frames- the current frame and the reference frame
What is some portion?
Individual pixels (all of them)?
Lines/edges (have to find them first)
Objects (must define them)
Uniform regions (just chop up the frame)

12. General Idea
For a region PC in the current frame, find a region PR in the search window in reference frame so that Error(PR,PC) is minimized
Issues: Error measures, search techniques, choice of search window, choice of reference frame, choice of region PC
Current
Portion of
interest PC
Search
window
Reference
Frame
Frame

13. Block-based Motion Estimation
PC is a block of pixels (in the current frame)
The search window is a rectangular segment (in the reference frame)
T=1 (reference)
T=2 (current)

14. Motion Vectors
A motion vector (MV) describes the offset between the location of the block being coded (in the current frame) and the location of the best-match block in the reference frame
T=1 (reference)
T=2 (current)

15. Motion Compensation The blocks being predicted are on a grid
The blocks used for prediction are NOT 1 3 1 2 3 4 2 4 5 6 7 8 6 7 8 5 9 10 11 12 10 9 12 11 13 14 15 16 14 13 15 16

16. Motion Vector Search 1. Mean squared error 2. Mean abs. error
Select a block in the reference frame to minimize
Σ(b(Bref)-b(Bcurr))2
2. Mean abs. error
Select block to minimize
Σ|b(Bref)-b(Bcurr)|
Given error measure, how to efficiently determine best-match block in search window?
Full search: best results, most computation
Logarithmic search – heuristic, faster
Hierarchical motion estimation

17. Motion Compensation
The standards do not specify HOW the encoder will find the motion vectors (MVs)
The encoder can use exhaustive/fast search, MSE /MAE/other error metric, etc.
The standard DOES specify
The allowable syntax for specifying the MVs
What the decoder will do with them
What the decoder does is to grab the indicated block from reference frame, and glue it in place

18. Standard specifies bit stream
This allows future encoders of better performance to remain compatible with existing decoders.
Also allows for commercially secret encoders to be compatible with standard decoders
The video compression standards define syntax and semantics for the bit stream between encoder and decoder
Encoder is not specified by MPEG except that it produces a compliant bit stream
Compliant decoder must interpret all legal MPEG bit streams
ENCODER
DECODER
bit stream
Standard defines this
not this
Today’s Ho-Hum Encoder
Today’s Decoder
Tomorrow’s Nifty Encoder
Today’s decoder still works!
Very secret Encoder

19. Motion Compensation Example
Frame n+1
Frame n
(0,0)
(-16,0)
(5,0)
(16,7)
(5,2)
(20,-24)
(-20,-18)
MOTION COMPENSATED n+1

20. Objects versus Macroblocks
Real moving objects will not coincide with boundaries of macroblocks
If encoder sends MV=(MotX,MotY), object well coded, but background poorly coded
If encoder sends MV=(0,0), background well coded, but moving object poorly coded
Either approach is valid
background
moving object
Prediction error
Moving object well encoded with motion vector
Background well encoded (no motion vector)

21. Motion Compensation This glued together frame is called
the motion compensated frame
The encoder can also form the difference between the motion compensated frame and the actual frame.
This is called the motion compensated difference frame
This difference frame formed using MC should have less correlation between pixels than the difference frame formed without using MC

22. Motion Compensated Difference Frames
Suppose we are doing lossless coding
Encoder has sequence of frames: …, F(n-2), F(n-1)
Now we are going to encode F(n)
Past frames have been losslessly encoded, so the decoder knows F(n-1) perfectly already
Encoder sends the motion vectors for frame F(n) relative to frame F(n-1), to form motion compensated frame M(n)
Encoder knows M(n), Decoder knows M(n)

23. Encoding Difference Frames
Encoder forms motion compensated difference frame: MCD(n) = F(n) – M(n)
Encoder losslessly encodes MCD(n)
Decoder can then do
F(n) = MCD(n) + M(n) → knows F(n) exactly
If we didn’t do motion compensation, encoder could do frame diff: FD(n) = F(n) – F(n-1)
Encode FD(n) losslessly, then decoder does
F(n) = FD(n) + F(n-1) → knows F(n) exactly

24. Motion compensated difference frames
Decoder knows F(n-1) and, once you send the motion vectors, it knows M(n)
Reference Frame F(n-1)
Original Frame F(n)
Difference Image FD(n)=F(n)-F(n-1)
Motion compensated frame M(n)
Motion compensated difference image MCD(n) =F(n) – M(n)
Send FD(n)
Send Motion Vectors
Send MCD(n)

25. Motion Compensated Difference Frames
But we are NOT doing lossless coding
Encoder has sequence of frames: …, F(n-2), F(n-1)
Now we are going to encode F(n)
Past frames have been lossy encoded, so the decoder has versions …, G(n-2), G(n-1)
Encoder knows …, G(n-2), G(n-1) also
Encoder sends the motion vectors for frame F(n) relative to frame G(n-1), to form motion compensated frame M(n)

26. Encoding Difference Frames
Encoder forms motion compensated difference frame: MCD(n) = F(n) – M(n)
Encoder lossy encodes MCD(n)
Call the decoder version MCD*(n)
If the decoder received MCD(n) exactly, could do: F(n) = MCD(n) + M(n)
But with MCD*(n), decoder can do
G(n) = MCD*(n) + M(n)
→ knows F(n) approximately

27. Motion estimation philosophy
Goal of motion estimation is NOT to provide a careful analysis of the actual motion
Goal is to achieve a given quality of representation of the video while globally minimizing the bit rate required to send
The motion information
The prediction error information
Most of the time, for a given representation quality
fewer bits to send MV+MCD(n) instead of sending FD(n)
fewer bits to send FD(n) instead of sending F(n) itself.

28. Motion Compensation for Chrominance
Luminance is highly correlated, more so than chrominance
The “best” motion vectors are available by searching in the luminance plane
Motion vectors for chrominance are not computed separately, simply scaled as needed

29. Motion Estimation/Compensation Summary
At the encoder:
For each block in the frame being coded, examine the search window(s) in the reference frame to find the best match block (do this for luminance only)
Form the MC difference image = original image minus motion compensated image
Scale the motion vectors for the chrominance, form the motion compensated chrominance frames, and form chrominance difference image

30. Motion Estimation/Compensation Summary
At the decoder:
Decode the reference frames (Y,Cr,Cb)
For each block in a temporally coded Y frame, use the motion vector to select a block from the reference frame and glue it in place
Add the Y difference image
For each block in temporally coded Cr,Cb frames, first scale the motion vector, then do the previous 2 steps with Cr and Cb data

31. Accuracy of Motion Vectors
Digital images are sampled on a grid. What if the actual motion does not move in grid steps?
Solution: interpolation of grid points in reference frame adds a half-pixel grid
Reference frame effectively has 4 times as many positions for the best match block to be found A h B v m C D

32. Progress of Video Compression

33. Progress of Video Compression

34. Progress of Video Compression

35. Temporal Location of Reference
The reference frame need not occur before the temporally coded frames which use it
Why? Scene changes, allow better matches

36. Flavors of Motion Estimation
1. Forward predicted blocks: the best-match block occurs in the reference frame before the block’s frame
2. Backward predicted blocks: the best-match block occurs in the reference frame after the block’s frame
3. Interpolatively predicted blocks: the best-match block is the average of the best-match blocks from reference frames before & after
The motion compensation direction can be selected independently for each block in a frame.

37. Motion Vector Coding
How are the motion vectors actually encoded for transmission to the decoder?
Start by taking the difference between the current motion vector and the most recent previous one of the same type (forward/backward/interpolative)
Encode the difference using variable length coding
Horizontal and vertical components coded separately

38. MPEG Frame Types
Intra (I) pictures: coded by themselves, as still images. No temporal coding. No motion vectors.

39. MPEG Frame Types
Forward Motion Compensated predicted (P) pictures – forward motion compensated from the previous I or P frame

40. MPEG Frame Types
Motion Compensated interpolated (B) pictures – forward, backward, and interpolatively motion compensated from previous/next I/P frames

41. MPEG Frame Structure Terminology
A block contains 8x8 pixels
The DCT unit
A macroblock (MB) contains 4 blocks from the luminance, plus the corresponding chrominance blocks
4 blocks from each of Cr/Cb if 4:4:4 format
2 blocks from each of Cr/Cb if 4:2:2 format
1 block from each of Cr/Cb if 4:1:1 or 4:2:0 format
The motion compensation unit

42. MPEG Frame Structure Terminology
A slice is a collection of macroblocks, tracing in a raster scan from upper left to lower right
The resynchronization unit
A picture is a frame, either progressive (non-interlaced) or interlaced
The primary coding unit
A Group of Pictures (GOP) contains ≥ 1 frame.
The unit for random access into the sequence

43. MPEG GOP Structure A Group of Pictures (GOP) may contain
All I pictures
I & P pictures only
I, P, & B Pictures
A common GOP format for 30 frames/sec:
I-picture spacing 15 frames (1/2 second)
P-picture spacing 3 frames (1/10 second) I B P 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

44. Frame Ordering Display order (encoder input order):
But consider coding dependencies:
Frame 2 (B) needs frame 4 (P) to be decoded first, etc.
So better transmit frame 4 before frame 2 B I P -1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 I B P 1 -1 4 2 3 7 5 6 10 8 9 13 11 12 16 14 15

45. Group of Pictures IIIII…: Every picture is intra-coded.
Fully decodable without reference to any other picture
Editing is straightforward
Requires about 2.5 more bit rate than bidirectional
IBBPBBPB…: Forward and bidirectional
Best compression factor
Needs large decoder memory
Hard to edit
Most useful for final delivery of post-produced material (e.g., broadcast) because no editing requirement

46. Group of Pictures IPPPPIPP…: Forward predicted only.
Needs less decoder memory
IBIBIB…: bidirectional compromise
Some of the bit rate advantage of bidirectional coding
Not nearly the full latency penalty of bidirectional
Editable with moderate processing.
For example, if the video after a B picture needs to be deleted, the B frame would not be decodable.
Solution is to decode the B frame first, re-encode it using forward prediction only. Some quality loss.