1. Hole Filling 관련 논문
장호욱

2. DIBR 개요
정의: “the process of synthesizing 'virtual’ views of a scene from still or moving images and associated per-pixel depth information”
기존 view -> 새로운 view
기존 영상 + depth map

3. DIBR 개요 단점: virtual view 생성시 occlusion(hole) 발생 Solution
가) Depth map preprocessing
Smooth depth map
- sharp transitions cause hole
- image distortion would be happened
나) Inpainting
reconstruct an image through interpolation, extrapolation, background mirroring
다) Layered Depth Images
store more than one pair of color and depth values for each pixel of original image

4. Depth map smoothing

5. Inpainting
Should consider neighbor’s texture info. & structural info.

6. 조사 논문
“A Local Depth Image Enhancement Scheme For View Synthesis”, Yongzhe Wang, Dong Tian, Anthony Vetro, Mitsubishi Electric Research Labs
“Depth Adaptive Hierachical Hole-Filling for DIBR-Based 3D Videos”, Madhhour Solh, Ghassan Algeib, Georgia Insitiute of Technology
“Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding”, Mounira Ebdelli, Christine Guillemot, Olivier Le Meur, INRIA
“Depth Image-Based Rendering with Spatio-Temporally Consistent Texture Synthesis for 3D Video with Global Motion”, Martin Köppel, Technical University Berlin, Germany; Xi Wang, Dimitar Doshkov, Thomas Wiegand, Patrick Ndjiki-Nya, Fraunhofer Heinrich-Hertz-Institute

7. A Local Depth Image Enhancement Scheme For View Synthesis개요
- depth images are likely to exhibit noise
: produce artifacts in the rendered views
- various depth enhancement techniques
: global mode filtering technique
: joint bilateral filter
: SIFT based depth improvement
- propose local depth image enhancement technique
: references the color information of the stereo image pair
: sparse depth features serve as an additional source of candidate depth values
※ stereo view -> virtual view 환경

8. A Local Depth Image Enhancement Scheme For View Synthesis
I : color image
D: depth image
F sparse depth feature
L : left view
R : right view
V : virtual view

9. A Local Depth Image Enhancement Scheme For View Synthesis
Depth Enhancement Algorithm
- depth candidate: input depth value d₁
neighboring depth value
: selected based on the minimal collocated color intensity difference of the neighboring pixels, d₂, d₃
: associated depth value of the nearest sparse depth feature d₄(stage 1)
: input depth value d₁
: median value from preceding five pixels of the same line d₂
: median value from proceding five pixels of the same column d₃(stage 2)

10. A Local Depth Image Enhancement Scheme For View Synthesis
Depth Enhancement Algorithm
cost calculation:
stereo cost: measures the consistency in color intensity between two views
color difference cost: measures the color intensity difference between the current location and the candidate pixel location
depth difference cost: measures the depth value difference between the current pixel and the candidates

11. A Local Depth Image Enhancement Scheme For View Synthesis

12. A Local Depth Image Enhancement Scheme For View Synthesis

13. Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos

14. Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos
Hierarchical approach for hole filling

15. Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos
Depth Adaptive Preprocessing
disocclusion areas are combination of background and foreground pixels
blur can be reduced by assign higher weights to depth values of background pixels

16. Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos
Pixels with low disparity values that are close to the minimum are considered background information and given higher weights
Pixels with high disparity values that are close to the maximum are considered foreground and are given lower weights
The transition between low and high disparity must be smooth.

17. Hierarchical Hole Filling Process
Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos
Hierarchical Hole Filling Process

18. Depth Adaptive Hierarchical Hole-Filling for DIBR-Based 3D Videos

19. Examplar-based Video Inpainting 처리 개요
Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding
Examplar-based Video Inpainting 처리 개요
estimates the motion for each pixel in the image
estimate the motion of each pixel: determine whether a pixel p belongs to (Mc(p) = 1) or (Mc(p) = 0)
inpainting the moving objects according to priority
P(p) = C(p)D(p)
most similar patch with input patch is selected
calculate SSD between vectors of five components (R, G, B, Vx, Vy)
when a patch is filled, the Mc(p) of the filled pixels is updated with the Mc values of the copied pixels
inpainting the stationary background
★The search for the best matching patch: entire images of the sequence => within a motion-compensated window
★template matching => more elaborate neighbor embedding techniques

20. Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding

21. DEPTH IMAGE BASED RENDERING WITH ADVANCED TEXTURE SYNTHESIS

22. Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding

23. Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding
- k-nearest neighbor 중 하나를 선택하는 대신 combine the first K candidates
Locally linear embedding (LLE) : approximating each input data point by a linear combination of its K-nearest neighbors
Non-negative Matrix Factorization (NMF) : searches for two lower dimensional nonnegative matrices whose product gives a good approximation of the input data matrix

24. Examplar-based video Inpainting with Motion-Compensated Neighbor Embedding

25. DEPTH IMAGE-BASED RENDERING WITH SPATIO-TEMPORALLY CONSISTENT TEXTURE SYNTHESIS FOR 3-D VIDEO WITH GLOBAL MOTION
전후 프레임 정보를 사용하여 temporal consistency를 확보하는 알고리즘 제안

26. DEPTH IMAGE-BASED RENDERING WITH SPATIO-TEMPORALLY CONSISTENT TEXTURE SYNTHESIS FOR 3-D VIDEO WITH GLOBAL MOTION