JPEG 2000 Image Coding System Coding of Still Pictures JPEG 2000 Part I Final Committee Draft Version 1.0 April, 2000

Why another JPEG? JPEG2000 can provide higher compression efficiency  for low bit-rate application larger images (up to 64k by 64k) superior performance for natural images (e.g. photo) computer-generated images (e.g. graphics) compound documents (e.g. bi-level text) five progression orders progressive by quality (SNR) or by resolution  lossless and lossy compression in a single stream ROI (Region of Interest) coding

JPEG v.s. JPEG2000 (Main Differences) DCT v.s. DWT (Discrete Wavelet Transform) Way of coding coefficients JPEG Zigzag scan Huffman coding, run-length coding JPEG2000 Coefficient bit modeling Scan algorithm Probability estimation Arithmetic entropy coding

Main Ideas of JPEG2000 DWT Way of coding coefficients Coefficient bit modeling Scan algorithm: bitplane and sub-bitplane (coding pass) Probability estimation Arithmetic entropy coding Tag tree to save tree information in header For progressive transmission ROI coding Five progression orders

Image -> Tiles Apply DWT for each tile Tile

Basic Coding Scheme of JPEG2000 DWT Scan Algorithm Probability Estimation Entropy Coding Tile of a Image Bit Stream DC level shift (Component Transform) Coefficient bit modeling Quantization

DC Level Shift e.g. The ith component is unsigned. The bit depth of the ith component is 8. (  Each sample value is in (0, 255))  I(x, y) = I(x, y) – 2 (  Each sample value is in (-128, 127)) (8 – 1)

Component Transform To decorrelate redundancy between components  to improve compression efficiency Component transformation is applied to the three first components. ICT (Irreversible Component Transform) For lossy compression RCT (Reversible Component Transform) For lossy and lossless compression

ICT (Irreversible Component Transform) The Forward ICT: Given I0, I1, I2 in the first three components: Y0 = I I I2 Y1 = I I I2 Y2 = 0.5 I I I2 If the first three components are R, G, and B, then the Forward ICT can be seen as a approximation of a YCbCr transformation. The Inverse ICT: I0 = Y Y2 I1 = Y0 – Y1 – Y2 I2 = Y Y1

RCT (Reversible Component Transform) The Forward RCT: Given I0, I1, I2 in the first three components: Y0 = floor( (I0 + 2 I1 + I2) / 4 ) Y1 = I2 – I1 Y2 = I0 – I1 The Inverse RCT I1 = Y0 – floor( (Y2 + Y1) / 4 ) I0 = Y2 + I1 I2 = Y1 + I1

High Pass: Y(2n+1) = X(2n+1) - floor( (X(2n) + X(2n+2)) / 2 ) Example of DWT: Lifting-based (5, 3) filter Low Pass: Y(2n) = X(2n) + floor( (X(2n-1) + X(2n+1) + 2) / 4 ) Extension Run …

Image “ Lena ” (256x256) Resolution levels (each consists of 1 or 3 sub-bands) LL0 HL1, LH1, HH1 HL2, LH2, HH2 Precincts (  positions) (packet partitions) e.g. 8 precincts(16x32) Code-blocks e.g. 2 code-blocks(16x16) HL3, LH3, HH3 Example of DWT (con ’ t) HL3 HH3 LH3

Example of DWT (con ’ t) (100,100,100,100,100,100, 0, 0, 100,100,100,100,100,100) (100, 100, 113, 0, 88, 100, 100)(0, 0, 50, -50, 0, 0, 0) D DWT (for row) HighLow 1D DWT

Example of DWT (con ’ t) DWT

Quantization Each of the transform coefficients f(u, v) is quantized to the value q(u, v). q = sign(f) Quantization step-size = 2 (1+ ) Pair (Exponent, mantissa ) for sub-band b is obtained either explicitly or implicitly. Nominal dynamic range is the bit depth for sub- band b to represent the original image. |f|  b  b R -  b b 2 11  b  b  b R b

Quantization (con ’ t) Explicit quantization (, ) are signaled in the bit stream for every sub-band. Implicit quantization (, ) is signaled in the bit stream for the LL sub-band. (, ) are derived implicitly from (, ).  b  b  0  0  b  b  0  0

Quantization (con ’ t) RR+1 R+2 R+1 R+2 R is the bit depth of the original image Two level analysis gain R b for each sub-band b R 

LSB Bitplane – > > > > MSBLSB MSB Code-block

Code-block Scan Pattern LSB MSB Bitplane0 Bitplane1 Bitplane2 Bitplane3

Sub-bitplane (Coding Pass) Each bitplane is coded in 3 sub-bitplanes (coding passes) Significance propagation (Magnitude) Refinement Clean-up Each coefficient bit in a bitplane is coded only one of the 3 sub-bitplane Depend on the “ significance state ” of that coefficient and its 8 neighboring coefficients

Significance State State of a coefficient at a particular bitplane If a coefficient has the first “ 1 ” bit at or before the given bitplane is considered “ significant ”,  significance state = 1 otherwise it is considered “ insignificant ”.  significance state = 0

Example of Sub-bitplane Coding Order … Bitplane3 Bitplane2 Bitplane1 Bitplane0 (Significance state = 1)  The coefficient is significant S R C C SR C S RC

(Neighbor) Context: Function of Neighbors States LSB MSBBitplane Bitplane2 Significance state (after coding bitplane3) V0D0D1 H0H1 D2V1D3 8 neighbors states = 0  context = 0

Bits in a Sub-bitplane significance state = 1 context ≠ 0 Significance propagationRefinementClean-up Significance propagationRefinementClean-up true false more important (significant)(insignificant) (most likely to be significant)

Bits in a Sub-bitplane (con ’ t) Bits coded in each sub-bitplane (coding pass) Significance propagation Bits from “ most likely to become significant ” coefficients  (significance state = 0) and (context ≠ 0) Refinement Bits from already significant coefficients  (significance state = 1) Clean-up Bits from all remaining coefficients  (significance state = 0) and (context = 0)

Coding Operations 4 coding operations Significance coding (9 context bins) Sign coding (5 context bins) Magnitude refinement coding (3 context bins) Clean-up coding (run-length coding and 9 context bins) Coding operations in each sub-bitplane Significance propagation Significance coding and sign coding Refinement Magnitude refinement coding Clean-up Clean-up coding and sign coding

9 Context Bins for Significance and Clean-up Codings V0D0D1 H0H1 D2V1D3 LL and LHHLHHContext (label) ΣHΣVΣDΣDΣHΣVΣDΣDΣ(H+V)ΣDΣD 22 ≧3≧3 8 1 ≧1≧1 ≧1≧1 1 ≧1≧ ≧1≧1 01 ≧1≧ ≧2≧ ≧2≧2 00 ≧2≧2 ≧2≧ e.g. ΣH = 0 ΣV = 1 ΣD = 0 For HL 5

5 Context Bins for Sign Coding HVContext (label)XOR bit  expected value = 0 (expected sign = +)  expected value = 1 (expected sign = -) V0 H0H1 V e.g. H = 0 V = H = 0 V = - 10 HVHV Code (sign bit) XOR (XOR bit) bit

3 Context Bins for Magnitude Refinement Coding First refinement for this coefficient Context label 16 Σ(H+V+D) = 0 Context label 15 Context label 14 false true V0D0D1 H0H1 D2V1D3

If the four contiguous coefficient bits currently satisfy Run-length Coding for Cleanup Coding Code 0 (using context label 17) Code the four bits (using context label 18) Four bits are all 0 true false Code 1 (using context label 17) significance state = 0 context = 0 UNIFORM context Run-length context

Huffman Coding 設法讓 “ 較常出現 ” 的 symbol ，使用較短 的 code 。 Symbolabcd 出現機率 3/41/81/16 Fixed-Length Code Huffman Code 資訊量 (= log(1/p) ) 平均資訊量 (Entropy) (= Σ p*log(1/p) ) 0.415(3/4) + 3(1/8) + 4(1/16) + 4(1/16) = 平均長度 2(3/4) + 2(1/8) + 2(1/16) + 2(1/16) = 2 1(3/4) + 2(1/8) + 3(1/16) + 3(1/16) = 1.375

AC: Arithmetic (Entropy) Coding 3/4 1/8 1/16 d c b a a b b d a Encode babda

Example of AC for Two Symbols A = 1 0 A = 0.9 Encode 0  A = A – 0.1 C = C C = 0.1C = A: the interval C: the code 0 A = 0.81 C = Encode 0  A = A – 0.09 C = C A = C = Encode 0  A = A – C = C

Example of AC for Two Symbols (con ’ t) A = 1 0 A = 0.1 Encode 1  A = 0.1 C = C C = A: the interval C: the code 0 A = 0.01 C = Encode 1  A = 0.01 C = C …

AC for Two Symbols Two symbols (0 and 1) LPS (Less Probable Symbol) Probability Qe MPS (More Probable Symbol) Probability (1-Qe) E.g. Pr(0) = 0.9 Pr(1) = 0.1 MPS = 0 LPS = 1 Qe = 0.1

The MQ-Coder 1-Qe Qe MPS LPS 0 A = 1 0 A – Qe Encode MPS  A = A – Qe C = C + Qe C = Qe 0.75 ≦ A ＜ 1.5 C = 0 1-Qe Qe MPS LPS 0 A = 1 0 Qe Encode LPS  A = Qe C = 0 A ～ 1

The MQ-Coder (con ’ t) Keep the interval A in the range 0.75 ≦ A ＜ 1.5 (by using renormalization, if needed), and save code in C Code MPS A = A – Qe (i.e. sub-interval for the LPS) C = C + Qe Code LPS A = Qe (i.e. sub-interval for the MPS)

Renormalization of The MQ-Coder Out one bit (most significant bit) of C Double A and C  A<<=1, C<<=1 Obtain a new probability estimate (state) for the context

Finite State Machine of The MQ-Coder Pr(0) = 0.7 Pr(1) = 0.7 Pr(0) = 0.86 Pr(0) = 0.94 Pr(0) = 0.5 Pr(0) = 0.67Pr(0) = 0.51 Pr(0) = Slowly change Pr(0) Fast change Pr(0) MPS = 0 …… … …

Initialization and Termination of The MQ-Coder Initialization Start at the initial state  Pr(LPS) = Qe Termination Terminate coding operations Generate the required terminate marker

Usage of The MQ-Coder Each context bin invokes a MQ-Coder. The initial state of each MQ-Coder: UNIFORM context (label 18) the state with Pr(0) = 0.5, and no state-transaction Run-length context (label 17) the state with Pr(0) = All-zero-neighbors context (label 0) the state with Pr(0) = Other contexts (labels 1-16) the state with Pr(0) = 0.5

Layer A layer consists of some number of consecutive sub-bitplanes (coding passes) e.g. layer0 = {the first 2 sub-bitplanes} layer1 = {the following 3 sub-bit-planes} … layer0layer1 higher sub-bitplaneslower sub-bitplanes …layer2

Packet Header 2 sub-bitplanes from code-block0 1 sub-bitplane from code-block1 5 sub-bitplanes from code-block2 … e.g.

Tag Tree A way of representing a 2D array in a hierarchical way To save information in a packet header Only the information needed for the current code-block is stored in the packet header Ability to progressive transmit

Example of Tag Tree (A>0, A=B=2) 011 (C=n1=3) 0001 (n2>C=3, n2=C+3) 1 (D=B) 01 (n3>D, n3=D+1) A B C n1n2 D n3 A, B, C, n1 n2 D, n3 Minimum of the sub-tree

Information in Packet Header Bit for zero ( “ 0 ” ) or non-zero ( “ 1 ” ) length packet For each precinct of each sub-band For all code-blocks in the precinct in raster order Inclusion information (  the layer ID where the code-block first appears in the precinct of the sub-band) If included at the 1 st time  tag tree else one bit (the code-block is included(1)/not included(0)) If the code-block is included If included at the 1 st time Zero bit-planes information  tag tree # of sub-bitplanes Length for the code-block …

ROI (Region of Interest) ROI is a part of an image coded earlier in the codestream than the rest part (background) A tile can be either Outside the ROI Inside the ROI Partly inside the ROI ROI coding Arbitrary region shapes Any number of regions Maxshift method

ROI Coding 4 steps: Generate ROI mask M(x, y) Select the scaling value s Maxshift method s ≧ (bit depth of max(BG)) Scale down all BG by s Write s into codestream 1(x, y) is needed for ROI 0otherwise (  (x, y)  BG (background))  M(x,y) =

ROI Mask Generation e.g …… Sample in ROI High Low

ROI Maxshift Method ROI coefficients Coefficient values ROI coefficients max(BG) Maxshift e.g. max(BG) = 10 s = 4 Shift-left 4 bits for each coefficient in ROI Mask

Advantages of ROI Maxshift Method Support for arbitrary region shapes and any number of regions with minimal complexity No need for sending shape information shape codec ROI mask at decoder side Decoder as simple as non-ROI capable decoder Support progression by accuracy and by quality

Error Resilience Error resilience tools Packet level Entropy coding level

Error Resilience Tools at Packet Level Packets Re-synch marker SOP (Start of Packet) at packet boundaries Locate errors in a packet Short packet format (packed packet) Move the packet headers to either Tile-part header (PPT) Main header (PPM) of the codestream

Error Resilience Tools at Entropy Coding Level Code-blocks Re-synch marker at code-block boundaries Locate errors in a code-block Bit-planes Put segmentation symbol at the end of each bit- plane for error detection Sub-bitplanes (coding passes) The AC terminate on each sub-bitplane some sub-bitplanes (  selective AC bypass) last sub-bitplane Reset context probabilities on each sub-bitplane

Selective AC bypass SPAC, terminateAC MRAC, terminateAC CAC, terminateACAC, terminate SP (significance propagation) AC, terminateAC MR (magnitude refinement) AC, terminateAC C (cleanup)AC, terminateAC C (cleanup)AC, terminateAC … SPAC, terminateACRaw MRAC, terminateACRaw, terminate CAC, terminateACAC, terminate 5 … SPAC, terminateACRaw MRAC, terminateACRaw, terminate CAC, terminate Final AC terminate on each sub-bitplane AC terminate on last sub-bitplane ………… …………

Progression Order The interleaving of the packets can progress along 4 axes: Component e.g. Y, Cb, Cr Resolution e.g. LL0, HL1,LH1,HH1, HL2,LH2,HH2, … Position Layer

Layer-resolution-component- position Progressive (1) layer0layer1 higher sub-bitplaneslower sub-bitplanes LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0posotion1posotion2 … …

Resolution-layer-component- position Progressive (2) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

Resolution-position-component- layer Progressive (3) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

Position-component-resolution- layer Progressive (4) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

Component-position-resolution- layer Progressive (5) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

What we have learned? Operations of DWT Intelligent scan order More important coefficient bits are coded earlier. MQ coder (Arithmetic coder) for two symbols Tag tree ROI coding

Other Topics Rate control (Annex 10) Visual frequency weighting (Annex J.8) TCQ (Trellis coded quantization) Postprocessing