1. Introduction to Image and Video Coding Algorithms

2. Outline Transform-based Image and Video Coding
Linear Transformation – DCT
Quantization
Scalar Quantization
Vector Quantization
Entropy Coding
Video Coding – Motion Compensation

3. Transform-based Image Coding
Input
Image
Binary bit
stream
Linear
Transform
Quanti-
zatioin
Entropy
Coding

4. Linear Transform
If the signal is formatted as a vector, a linear transform can be formulated as a matrix-vector product that transform the signal into a different domain.
Examples:
K-L Expansion
Discrete Fourier Transform
Discrete cosine transform
Discrete wavelet transform
Energy compaction property: The transformed signal vector has few, large coefficients and many nearly zero small coefficients. These few large coefficients can be encoded efficiently with few bits while retaining the majority of energy of the original signal.

5. Block-based Image Coding
An image is a 2D signal of pixel intensities (including colors).
A block-based image coding scheme partitions the entire image into 8 by 8 or 16 by 16 (or other size) blocks. Then the coding algorithm is applied to individual blocks independently.
Blocks may be overlapping or non-overlapping.
Advantage: parallel processing can be applied to process individual blocks in parallel. For hand-held devices, only one block needs be loaded into main memory each time.

6. JPEG Image Coding Algorithms
DCT Q
DPCM
Zig Zag
Scan DC
Huffman AC
8x8
block
Quantization
Matrix
Code books
JPEG Encoding Process

7. JPEG Decoding JPEG Decoding Process DC DC IDPCM 8x8 Huffman block IDCTIQ AC
Huffman AC
JPEG Decoding Process

8. Pre-Processing Color sub-sampling
A color image is converted from RGB to YUV color space. Each pixel in each dimension is 1 byte.
Sub-sample U-V planes: 4:1:1 scheme.
For every 16 by 16 block of a color image, six 8 by 8 blocks are encoded.
Level shifting: Each pixel value is subtracted by 128 so it ranges (–128, 127).
Four 88 blocks of luminance pixels, plus two 88 sub-sampled chrominance components makes a 16 by 16 macro-block

9. Discrete Cosine Transform
88 two-dimensional separable DCT:
DCT is chosen because it leads to superior energy compaction for natural images.
F(0,0): DC coefficient ranges (-128x64/4,127x16) needs 12 bits to represent (including sign bit). 12 bits are more than enough for the remaining AC coefficients (u > 0, or v > 0)

10. Inverse DCT (IDCT) 88 two-dimensional separable IDCT:
IDCT can be computed using the same routine as DCT

11. DCT Basis Functions

12. Quantization of DCT Coefficients

13. DPCM of DC coefficients
DC coding: All DC coefficients of each 8 by 8 blocks of the entire image are combined to make a sequence of DC coefficients.
Next, DPCM is applied:
DiffDC(blocki) = DC(blocki) – DC(blocki–1)
Then DiffDCs will be encoded using Hoffman entropy
Example:
Original:
1216  1232  1224  1248  1248  1208
After DPCM:
1216  +16  -8  +24  0  -40

14. Huffman Entropy Coding
Task: to assign a variable-length binary code to a finite set of alphabets.
Goal: to minimize the average length (number of bits) per alphabet.
Approach: Shorter code for alphabet occurred more frequently. Longer for infrequent ones.
Optimal solution:
When the averaged code length approaches the entropy of the source.
Huffman coding:
Code words are derived from a (perhaps) un-balanced binary tree.
Arithmetic coding is another entropy coding method.

15. Huffman Encoding of DC Coefficients
Encoding and decoding of Huffman code is done via look-up table.
In JPEG, DC coefficients (after DPCM) are first grouped according to their magnitudes. Each category is assigned as a symbol and a Hoffman table is given. For example, –7 to –4 and 4 to 7 are listed as category 3 which has a code "00“.
If the number is positive, the binary representation of the number will be append to the Hoffman code of the category number directly. For example, 6 is encoded as If the number is negative, the appended code is the 1’s complement of that number. For example, -5 is encoded as
Question: Given such a table, how to devise a dedicated hardware to implement the encoding procedure?

16. JPEG Huffman Table: Categories

17. JPEG DC Entropy Coding Example:
-9: category 4. Hence Base code = 101
1’s complement of (-9) = 1C(1001) = 0110
Code word = =
Note that category 3 occurs most frequent and hence has shortest base code word.

18. AC Coefficients Zig-Zag scan order
AC coefficients are first weighted with a quantization matrix:
C(i,j)/q(i,j) = Cq(i,j)
Then quantized.
Then they are scanned in a zig-zag order into a 1D sequence to be subject to AC Huffman encoding.
Question: Given a 8 by 8 array, how to convert it into a vector according to the zig-zag scan order? What is the algorithm? 1 2 6 7 15 16 28 29 3 5 8 14 17 27 30 43 4 9 13 18 26 31 42 44 10 12 19 25 32 41 45 54 11 20 24 33 40 46 53 55 21 23 34 39 47 52 56 61 22 35 38 48 51 57 60 62 36 37 49 50 58 59 63 64
Zig-Zag scan order

19. AC Coefficients Huffman Encoding
The symbols for encoding AC coefficient consists both the number of significant bits, as well as runs of 0s preceding the nonzero AC coefficient. For example,
–1 is encoded as:
This is according to the table below:
Number
Run/Category
Base code
Length
Final code 5
0/3
100 6 02
1/2
111001 8
00-1
2/1
11011

20. Huffman Decoding A look-up table procedure.
Challenge: How to perform decoding fast?
Example: a Huffman table for six symbols:
The decoding process can be modeled as a finite state machine with the following state diagram. It decodes one bit of input bit stream per clock cycle.
Question: How to make this process fast enough to match any input bit rate? a
0/A b c
0/B d e
1/-
0/-
0/C,1/D
0/E,1/F
Symbol
Codeword A B 10 C
1100 D
1101 E
1110 F
1111

21. Video Coding
Video coding is often implemented as encoding a sequence of images. Motion compensation is used to exploit temporal redundancy between successive frames.
Examples: MPEG-I, MPEG-II, MPEG-IV, H.323, H.263, H.263+, etc.
Existing video coding standards are based on JPEG image compression as well as motion compensation.

22. MPEG Encoding + +  DCT Q VLC Q-1 IDCT
Buffer control
Current frame x(t) r
Bit stream
Buffer +
DCT Q
VLC 
Q-1
IDCT ^
x(t): predicted frame
Q[r(t)]: reconstructed residue + ~
x(t): reconstructed current frame ~
Motion
Estimation &
Compensation
x(t)
x(t-1)
Frame
Buffer
This is a simplified block diagram where the encoding of intra coded frames is not shown.
Motion vectors

23. MPEG Decoding + VLD Q-1 IDCT VLD: Variable Length Decoding
Received bit stream
Bit stream
Buffer
VLD
Q-1
IDCT ^
x(t): predicted frame
Q[r(t)]: reconstructed residue + ~
x(t): reconstructed current frame ~
Motion
Compensation
x(t-1)
Frame
Buffer
Motion vectors

24. Motion Estimation Three types of frames:
Intra (I): the frame is coded as if it is an image
Predicted (P): predicted from an I or P frame
Bi-directional (B): forward and backward predicted from a pair of I or P frames.
A typical frame arrangement is (subscripts are used to distinguish them):
I1 B1 B2 P1 B3 B4 P2 B5 B6 I2
P1, P2 are both forward-predicted from I1. B1, B2 are interpolated from I1 and P1, B3, B4 are interpolated from P1, P2, and B5, B6 are interpolated from P2, I2.

25. Forward Motion Estimation1 2 3 4 2 4 1 3 8 5 6 7 8 5 7 6 9 11 12 9 10 11 12 10 13 15 16 13 14 15 16 14
Current frame constructed From different parts of reference frame
Reference frame

26. Block Motion Estimation
MAD: Mean absolute difference between the I,j-th pixel of the current block x(i,j) and the (I+m,j+n)-th pixel of the reference frame.
(-pm,n  p) is the motion vector corresponding to the macro-block. M and N are search range.
It is similar to DPCM in the temporal domain, and has less to do with object motion.
motion vector
current block
search area
reference frame
current frame

27. Video sequence : Tennis frame 0
Prepared by Surin Kittitornkun

28. Video sequence : Tennis frame 1
Prepared by Surin Kittitornkun

29. Frame Difference
Prepared by Surin Kittitornkun

30. What is motion estimation?
Prepared by Surin Kittitornkun

31. What is motion compensation ?
Prepared by Surin Kittitornkun

32. Motion Compensated Frame Difference
Prepared by Surin Kittitornkun

33. 6-Level Nested Do Loop Do h=0 to Nh-1 End do j Do v=0 to Nv-1 End do i
MV(h,v)=(0,0)
Dmin(h,v)=
Do m=-p to p (-1)
Do n=-p to p (-1)
MAD(m,n)=0
Do i=hN to hN+N-1
Do j=vN to vN+N-1
MAD(m,n)= MAD(m,n)
+|x(i,j)-y(i+m,j+n)|
End do j
End do i
If Dmin(h,v)>MAD(m,n)
Dmin(h,v)=MAD(m,n)
MV(h,v)=(m,n)
End if
End do n
End do m
End do v
End do h