Light Field Compression Using 2-D Warping and Block Matching Shinjini Kundu Anand Kamat Tarcar EE398A Final Project 1 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Outline Motivation and Goals Overview of Our Method Results and Analysis Summary Future Work References 2 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Motivation Light field images are used in computer graphics to compute new views of a scene without need for scene geometry model 1. Need to compress large set of images Exploit inter-view coherence to achieve compression. 1. M. Levoy and P. Hanrahan, “Light field rendering,” in Computer Graphics (Proceedings SIGGRAPH 96), August 1996, pp EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Light Fields Represents a 3D scene or object from all viewing positions and directions – 2D array of 2D images – Difficult to Acquire – Very Large Perfect representation requires images of the order of the resolution

Light Field Views EE398A - Compression of Light Fields using 2-D Warping and Block Matching 5

Light Field Data Set 8.4 MB uncompressed data sets Credit: Andrew Adams 6 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Related Work Intra-frame coding – Vector quantization, DCT coding, transform coding yield compression ratios of less than 30:1 Inter-frame coding (compression in the hundreds, thousands) – Disparity compensation – 3D geometry models – Blockwise Compression ideal: maximally use coherence between two images 7 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Our Method: 2-D Warping Each consecutive view is a projection of the previous view due to constant predictable movement of camera Find this relation between the views by obtaining projection matrix for each pair of views Predict the view and encode the residual 8 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Our Encoding Scheme 9 EE398A - Compression of Light Fields using 2-D Warping and Block Matching Reconstructed Previous View Previous Frame 2-D Warped Lagrangian Cost Function Cost=R1+λD1 Cost=R2+λD2 2D Warping Algorithm 2-D DCT for the Residual Residual and MV ? Input View -- Use for Reconstruction

Notes DCT used on 8x8 blocks to encode residual Laplacian distribution assumed for motion vectors Projection matrix was encoded by normalizing values with respect to 10, and assuming Laplacian distribution of bitrate. The min and max values are encoded separately using binary encoding. H = EE398A - Compression of Light Fields using 2-D Warping and Block Matching 10

1. Feature match by correlation 2. Projective matrix computed Lagrangian Mode Decision using two references 3. Clipped edges are interpolated using motion compensation 11 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Getting a predicted projection: Step 1: Feature matching by Correlation Features detected by Harris corner detection algorithm, and matching points identified by maximum correlation 12 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Computing the Homography Matrix A homography is an invertible transformation from the real projective plane to the projective plane that maps straight lines to straight lines EE398A - Compression of Light Fields using 2-D Warping and Block Matching 13

Results for 2-D Projection Warping EE398A - Compression of Light Fields using 2-D Warping and Block Matching 14

Results for 2-D Projective Warping EE398A - Compression of Light Fields using 2-D Warping and Block Matching 15

Results for 2D Projective Warping EE398A - Compression of Light Fields using 2-D Warping and Block Matching 16

Compression Ratios EE398A - Compression of Light Fields using 2-D Warping and Block Matching 17

Conclusion Advantages: decreased coding complexity, and increased rate/PSNR as well as compression Experimental results demonstrate improved coding efficiency with our 2D warp method when compared with MVC. 18 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Future Work Possible Optimize the code to give better PSNR values and check performance by introducing extra modes like copy mode Explore other methods of using inter-view redundancy in detail like disparity compensation at sub-pel accuracy Run for larger data sets and optimize complexity of the algorithm EE398A - Compression of Light Fields using 2-D Warping and Block Matching 19

Summary Light fields represent a 3D scene using sequence of 2-D images Large amounts of data Can use redundancy between images using 2- D warping with motion compensated block matching Results in a sleek method for compression Performance wise.. EE398A - Compression of Light Fields using 2-D Warping and Block Matching 20

Acknowledgement Prof. Girod for pointing us in the right direction Mina Makar for his help Chuo-Ling Chang for DAPBT code Huizhong Chen and Derek Pang for their help Prof. Peter Kovesi for open source matlab function library Prof. Levoy’s group and Andrew Adams for access to light field images 21 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Questions? EE398A - Compression of Light Fields using 2-D Warping and Block Matching 22

Other Projects Use Motion Compensation with Directional Transforms o Result: Gain in PSNR due to directionality is approximately 0.1dB at high Quantization; almost nil increase seen at low quantization So, We adapted the direction of out project to study a new approach of compression presented next. EE398A - Compression of Light Fields using 2-D Warping and Block Matching 23

Results with Motion Compensation and DAPBT for Crystal light field 24 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

Results with Motion Compensation and DAPBT for Lego light field EE398A - Compression of Light Fields using 2-D Warping and Block Matching 25

This is how blocking is done and direction selection happens! IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field EE398A - Compression of Light Fields using 2-D Warping and Block Matching 26

For Lego light field IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field EE398A - Compression of Light Fields using 2-D Warping and Block Matching 27