1. Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Carsten Clemens Clemens@nue.tu-berlin.de Error Concealment for Stereoscopic Sequences ITG Fachausschusstagung 3.2, Juni 2006

2. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences2 Monoscopic Error Concealment strategies are not well suited for stereoscopic scenario Assumtions –independently coded views of a stereo image pair remaining redundancies between the channels, which can be utilized for error concealment –block based coding (16x16 blocks) Introduction

3. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences3 Algorithm Overview Identification of corresponding region –feature extraction –feature matching along epipolar line –selection of matches (M-estimator/RANSAC) Projective Transformation –initial parameter set from matches –optimization by Newton Method Smoothing –only in case of discontinuities of depth

4. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences4 Matching and Transformation projective transformation:

5. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences5 Selection of feature matches M-estimator –uses all matches with different weights –In some cases the transformation fails, because pixels from outside the image were warped into the erroneous burst. RANSAC (random sample consensus) –uses a number of subsamples (four feature matches) –minimize the sum of squared residues of the boundary region: RANSAC yields better results than M-estimator

6. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences6 Adapted Newton Method Find the optimal transformation parameter by minimizing a cost function C(k): b is the Border Region of the erroneous block burst

7. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences7 Adapted Newton Method Iteration step: Problem I: Local minimum solution Initial Parameter set is of prime importance Cost function C(k) over horizontal and vertical translation parameter

8. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences8 Adapted Newton Method Problem II: Convergence of Newton method Successivly decreasing of border pixel size L after every minimum search Speed of convergence of the Newthon algorithm for different border sizes L

9. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences9 3D Block Smoothing In case of great discontinuities in depth (variation of disparity) we perform a linear smoothing algorithm towards the surrounding pixel region (3D- BS) Minimization of the intersample variance between neighboring samples and to the block borders

10. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences10 HQ EC: Results / Example

11. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences11 Results / Subjective Evaluation Subjective Simulation Results: –Double Stimulus Continuous Quality Scale Method (DSCQS) as phsychovisual test with 15 subjects –Shutterglasses (StereoGraphics) DMOS

12. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences12 Fast EC: Algorithm Overview Block Search –Directional Diamond Search –SAD Side Match Distortion

13. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences13 Fast EC: Matching Example Hall SMD determined for each possible position Position with minimum SMD selected Block used for reconstruction

14. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences14 Fast EC: Subjective Evaluation BALLOONDMOSS.D.conf. interval Interpolation5,691,19±0,65 SBS4,391,12±0,62 TBS2,522,64±1,45 TSBS2,491,37±0,75

15. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences15 Simple Matching - Example Balloons 720x480, corrupted frame and features

16. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences16 Simple Matching - Example Reference frame (temporal) with features

17. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences17 Simple Matching - Example Reference blocks

18. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences18 Simple Matching - Example Reconstructed frame

19. Communication Systems Group C. Clemens Technische Universität Berlin Error Concealment for Stereoscopic Sequences19 Publications K. Günther, C. Clemens, and T. Sikora A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment PCS 2004, San Francisco C. Clemens, M. Kunter, S. Knorr, and T. Sikora: A hybrid approach for error concealment in stereoscopic images WIAMIS '04, Lissabon M. Kunter, S. Knorr, C. Clemens, and T. Sikora: A gradient based approach for stereoscopic error concealment ICIP '04, Singapore S. Knorr, C. Clemens, M. Kunter, and T. Sikora Robust Concealment for Erroneous Block Bursts in Stereoscopic Images 3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece K. Günther, C. Clemens, and T. Sikora A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment PCS 2004, San Francisco C. Clemens, M. Kunter, S. Knorr, and T. Sikora: A hybrid approach for error concealment in stereoscopic images WIAMIS '04, Lissabon M. Kunter, S. Knorr, C. Clemens, and T. Sikora: A gradient based approach for stereoscopic error concealment ICIP '04, Singapore S. Knorr, C. Clemens, M. Kunter, and T. Sikora Robust Concealment for Erroneous Block Bursts in Stereoscopic Images 3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece