Compression of 3D images using disparity based coder Department Of Communication System Engineering 2007 Compression of 3D images using disparity based coder Presenting: Shay Yannay & Arik Wasserman Supervising: Dr. Adi Stern & Dr. Ofer Hadar

Introduction & Motivation 3D imaging has great potential in the fields of home entertainment, industry, medicine etc. Auto-stereoscopy: the ability to represent a 3D image which is visible without the need of special equipment or lighting source. One way of implementation – Light Field and multiview array. Our goal – to purpose an efficient method for encoding such data for storing and transmitting.

Multiview array & Light Field Main Scheme Encoder (recursive) Multiview array & Light Field Disparity Map Map Estimation Image Compensation Export

LightField & Multiview Array Light Field - A reliable representation of a 3D scene independent of it’s complexity. Acquisition  process & Display: Integral Image system (IIS). Viewing zone Projection screen multiview lenslet projector Arranging a 2-dimentional array from the captured pictures : A Multiview Image.

Amount of Required Information – Example… Proper viewing condition: 1 pixel distance between each adjacent pictures. Image resolution - 256X256 x Array size – 32X32 x Bit Per Pixel (RGB) – 24 Total: 192MB So compression is crucial! We will try to take advantage of the redundancy between the pictures in order to compress the formed data base.

What is a Disparity? The offset of identical objects in two adjacent pictures is known as disparity. The direction of the offset is known – according to the image position. We need only to get the magnitude of the shift. For optimum (theory) – disparity values per each pixel. In practice - the distance between blocks (8X8) is taken. Hence, we can build for each picture a Disparity-Map.

Building the Disparity Map MSE 5 Disparity Value 0 7 15 23

Disparity Map Array

Encoding principals: step 1– Map Estimation Estimated Map Missing Map

Encoding principals: step 2– Image Compensation Missing Image Estimated Map 5

The Coder – Decision tree To meet a reconstruction quality of a missing image, we define a threshold parameter – Qmin (as PSNR). The purpose: to achieve Qmin at the decoder while saving the minimum necessary. Therefore, there are 3 possible methods for reconstructing a missing image: The compensation is based upon the estimated map. The compensation is based upon the original map (from initial array). The compensation is done as in (2) and an error is also added. The compensation stage goes along with a decision tree:

Compensate image using estimated disparity map Save nothing PSNR>Qmin Yes No Compensation With Original Disparity Map Save Original Disparity Maps PSNR>Qmin Yes No Calculate residue ( original-estimated ) Save Original Map + Residue

Encoding- recursion order Demo- 9X9 Light Field array : A C A C B C A C A

Encoding- recursion order Demo- 9X9 Light Field array : A C A C A C B C B C A C A C A C B C B C A C A C A

Encoding- recursion order Demo- 9X9 Light Field array : A C A C A C A C A C B C B C B C B C A C A C A C A C A C B C B C B C B C A C A C A C A C A C B C B C B C B C A C A C A C A C A C B C B C B C B C A C A C A C A C A

Disparity Coder Syntax Encoder Data Base Encoder Header Decoder Guide JPEG(4 Base ‘A’ Images) Huffman(Encoder Disparity Map) JPEG(Error)

Disparity Coder - Benchmark We tested our encoder Based on several parameters: Block size - 4x4 ; 8x8 ; 16x16 Qb & Qc - Qmin threshold is assigned for each ‘Compensation type’ in the decision tree : Qb – for B type images ; Qc – for C type images. {Qb;Qc | 30->46 , 30->46 – incremented by 1 dB}.

Disparity Coder - Benchmark Performance As A function of Qc & Qb Psnr(Qc,Qb) BitRate(Qc,Qb) Display Performance Time(Qc,Qb) Encoding Decision Ratio (%)

Images Array Quality Measurements

Images Array Bitrate – [bpp]

Images Array Encoding Elapsed Time

Encoding Decision Ratio (%) Encoding with Estimated Map Encoding with Original Map Encoding with Original Map + resideu

Encoding Decision Ratio (%) Encoding with Estimated Map Encoding with Original Map Encoding with Original Map + resideu

Encoding Decision Ratio (%) Encoding with Estimated Map Encoding with Original Map Encoding with Original Map + resideu

Encoding Decision Ratio (%) Encoding with Estimated Map Encoding with Original Map Encoding with Original Map + resideu

Encoding- Performance Reconstructed Multi-view Array Original Images Reconstructed Images Q(global) = 40, compression ratio  1:200

Disparity Coder – Compression Ratio

Further work Testing various image types - synthetic and real, - higher content complexity, - larger array sizes (wider viewing angle) Optimizing the algorithm (block size, minimizing residue saving) Implementing concepts of video coding Decoder and GUI

Questions?