Hole Filling 관련 논문 2013. 06. 10 장호욱

DIBR 개요 정의: “the process of synthesizing 'virtual’ views of a scene from still or moving images and associated per-pixel depth information” 기존 view -> 새로운 view 기존 영상 + depth map

DIBR 개요 단점: virtual view 생성시 occlusion(hole) 발생 Solution 가) Depth map preprocessing Smooth depth map - sharp transitions cause hole - image distortion would be happened 나) Inpainting reconstruct an image through interpolation, extrapolation, background mirroring 다) Layered Depth Images store more than one pair of color and depth values for each pixel of original image

Depth map smoothing

Inpainting Should consider neighbor’s texture info. & structural info.

조사 논문 “A Local Depth Image Enhancement Scheme For View Synthesis”, Yongzhe Wang, Dong Tian, Anthony Vetro, Mitsubishi Electric Research Labs “Depth Adaptive Hierachical Hole-Filling for DIBR-Based 3D Videos”, Madhhour Solh, Ghassan Algeib, Georgia Insitiute of Technology “Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding”, Mounira Ebdelli, Christine Guillemot, Olivier Le Meur, INRIA “Depth Image-Based Rendering with Spatio-Temporally Consistent Texture Synthesis for 3D Video with Global Motion”, Martin Köppel, Technical University Berlin, Germany; Xi Wang, Dimitar Doshkov, Thomas Wiegand, Patrick Ndjiki-Nya, Fraunhofer Heinrich-Hertz-Institute

A Local Depth Image Enhancement Scheme For View Synthesis 개요 - depth images are likely to exhibit noise : produce artifacts in the rendered views - various depth enhancement techniques : global mode filtering technique : joint bilateral filter : SIFT based depth improvement - propose local depth image enhancement technique : references the color information of the stereo image pair : sparse depth features serve as an additional source of candidate depth values ※ stereo view -> virtual view 환경

A Local Depth Image Enhancement Scheme For View Synthesis I : color image D: depth image F sparse depth feature L : left view R : right view V : virtual view

A Local Depth Image Enhancement Scheme For View Synthesis Depth Enhancement Algorithm - depth candidate: input depth value d₁ neighboring depth value : selected based on the minimal collocated color intensity difference of the neighboring pixels, d₂, d₃ : associated depth value of the nearest sparse depth feature d₄(stage 1) : input depth value d₁ : median value from preceding five pixels of the same line d₂ : median value from proceding five pixels of the same column d₃(stage 2)

A Local Depth Image Enhancement Scheme For View Synthesis Depth Enhancement Algorithm cost calculation: stereo cost: measures the consistency in color intensity between two views color difference cost: measures the color intensity difference between the current location and the candidate pixel location depth difference cost: measures the depth value difference between the current pixel and the candidates

A Local Depth Image Enhancement Scheme For View Synthesis

A Local Depth Image Enhancement Scheme For View Synthesis

Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos

Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos Hierarchical approach for hole filling

Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos Depth Adaptive Preprocessing disocclusion areas are combination of background and foreground pixels blur can be reduced by assign higher weights to depth values of background pixels

Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos Pixels with low disparity values that are close to the minimum are considered background information and given higher weights Pixels with high disparity values that are close to the maximum are considered foreground and are given lower weights The transition between low and high disparity must be smooth.

Hierarchical Hole Filling Process Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos Hierarchical Hole Filling Process

Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos

Examplar-based Video Inpainting 처리 개요 Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding Examplar-based Video Inpainting 처리 개요 estimates the motion for each pixel in the image estimate the motion of each pixel: determine whether a pixel p belongs to (Mc(p) = 1) or (Mc(p) = 0) inpainting the moving objects according to priority P(p) = C(p)D(p) most similar patch with input patch is selected calculate SSD between vectors of five components (R, G, B, Vx, Vy) when a patch is filled, the Mc(p) of the filled pixels is updated with the Mc values of the copied pixels inpainting the stationary background ★The search for the best matching patch: entire images of the sequence => within a motion-compensated window ★template matching => more elaborate neighbor embedding techniques

Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding

DEPTH IMAGE BASED RENDERING WITH ADVANCED TEXTURE SYNTHESIS

Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding

Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding - k-nearest neighbor 중 하나를 선택하는 대신 combine the first K candidates Locally linear embedding (LLE) : approximating each input data point by a linear combination of its K-nearest neighbors Non-negative Matrix Factorization (NMF) : searches for two lower dimensional nonnegative matrices whose product gives a good approximation of the input data matrix

Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding

DEPTH IMAGE-BASED RENDERING WITH SPATIO-TEMPORALLY CONSISTENT TEXTURE SYNTHESIS FOR 3-D VIDEO WITH GLOBAL MOTION 전후 프레임 정보를 사용하여 temporal consistency를 확보하는 알고리즘 제안

DEPTH IMAGE-BASED RENDERING WITH SPATIO-TEMPORALLY CONSISTENT TEXTURE SYNTHESIS FOR 3-D VIDEO WITH GLOBAL MOTION