M15672: View synthesis software and assessment of its performance Mateusz Gotfryd Krzysztof Wegner Marek Domański Chair of Multimedia Telecommunications and Microelectronics Poznań University of Technology, Poland July, 21th 2008

Outline Algorithm description Test sequences Used depth maps Experiment description Objective video quality measure Subjective video quality measure Results Summary

Algorithm description Virtual camera can be synthesized from a real camera (the real reference view) and the respective depth map Unfortunately such an approach suffers from occlusion Virtual camera can be synthesized more correctly if two reference views are used

Block diagram View synthesis algorithm

View synthesis algorithm Proposed algorithm is composed of two identical, separate paths Each virtual view is synthesized from one reference view In the end both images of virtual view are merged into one Futher we describe single path in detail (right one)

Virtual view depth map Coordinates of each point in a reference view are transformed into coordinates of a point in the virtual view and virtual view depth map is created In the case when two points from reference view are transformed into the virtual view and have the same coordinates, always the points closer to the camera location are chosen Visibility problem Proper depth values

Virtual view depth map Resultant virtual view depth map has many small black holes on surfaces which have been rotated during the transformation from reference view into a virtual view Median filter is used on virtual view depth map Some regions in view have been uncovered Black regions have unknown depth Uncovered regions

Virtual view synthesis Color information is copied from reference view based on inverse homography matrix Calculated coordinates of corresponding point in reference view are not on pixel gird therefore interpolation is used

Merging of two virtual view images Images from two paths are merged into one Unknown regions from first path are filled with information from second path

Merging of two virtual view images Images from two paths are merged into one Unknown regions from first path are filled with information from second path

Filling holes Missing areas are interpolated from neighboring pixels

Contour correction Aliasing and blurring on the edges of the object are main reasons of „ghosting” effect Unknown regions are outlined by 1 pixel-width

Test sequences Sequences provided by Fraunhofer HHI Book Arrival, Leaving Laptop, Alt Moabit. Only 3 views (2nd, 3rd and 4th) have been used in the experiments

Depth maps Required depth maps were calculated with software provided by Nagoya University

Experiment description Video objective quality measured by PSNR Video subjective quality measured by Mean Opinion Score (MOS) 15 human subjects Rating range from 1 to 10 Collected opinions have been averaged

Results

Single frame subjective quality

View synthesis quality

View synthesis quality

Average synthesis quality

Summary We have presented a new view synthesis software Subjective experiments have been carried out with view-synthesis software provided to MPEG Our proposal received good results in criteria of objective quality measure (PSNR) and in case of subjective quality measure (MOS)