Semi-Global Matching with self-adjusting penalties Laboratory of Photogrammetry National Technical University of Athens Semi-Global Matching with self-adjusting penalties E. Karkalou, C. Stentoumis, G. Karras 7th International Workshop 3D ARCH 3D Virtual Reconstruction and Visualization of Complex Architectures 1-3 March 2017, Nafplion, Greece

Introduction 3D Surface Reconstruction Active (laser and optical) scanning Passive (image-based) approaches competitive to laser scanning in terms of accuracy, cost and flexibility a fundamental procedure is dense image matching: automatic determination of pixel correspondences among images for 3D surface reconstruction by using rectified images -epipolar geometry constraint- a disparity map is produced a core part for the majority of cultural heritage applications

Introduction stereo-matching advantages stereo-matching applications limited number of images – speed fixed geometry (e.g. stereo-camera) stereo-matching applications augmented reality through smart-phones reconstruction from a limited number of images (e.g. historical, aerial) robotics, autonomous navigation, … cost computation dissimilarity measure to each pixel for every value in the disparity range (AD, SSD, NCC, filtered images, rank, census) cost aggregation pixel cost is supported by the cost of neighbouring pixels disparity optimization - WTA (winner-takes-all) - energy function disparity refinement post processing of the final disparity map typical algorithm after [Scharstein & Szeliski, 2002 ]

Semi-Global Matching (SGM) - stereo algorithm for the step of optimization - originally proposed by Hirschmüller (2005, 2008) - global 2D energy function: - 1D-cost approximation in each of 8 directions (paths): - advantages: accuracy, computational efficiency, simplicity

SGM Penalties Imposed on disparity changes between neighbouring pixels up to 1 pixel (P1) or larger (P2) P1 penalizes slightly slanted or curved surfaces; P2 penalizes depth discontinuities Penalty adjustment is needed for every different pair of images or, if a different matching cost method is used, even for the same stereo-pair If parameters have not been properly tuned, the performance of the algorithm may not be as efficient as expected

Contribution  Automatic estimation of SGM penalties, after the computation of simple statistical properties of the DSI (Disparity Space Image) volume, already existing from the previous step of the algorithm Penalties are regarded as being self-adjusted to the particular stereo-pair, in relation to the cost function used No time-consuming tuning required – No ground truth disparity maps or multiple data for training are needed Low computational requirements  Evaluation of method via Middlebury benchmark and EPFL dataset

Self-adjusting penalty values Penalty values influence pixel costs and, therefore, should be related to these It is proposed that penalties are derived from the DSI representation S(x,y,l) of the initial cost  Penalty estimation without user intervention

Final algorithm (SGM-SAP) Initial matching cost (Census transform & Hamming distance or Absolute Differences of intensities) Penalty estimation SGM Disparity selection via WTA Disparity refinement [optional]: sub-pixel interpolation, left-right consistency, median filtering

Middlebury – Version 3 stereo-pairs left image right image true disparity map

Results (Middlebury – Version 3) raw results sub-pixel interpolation median filtering

Results in the Middlebury benchmark } Overall error 22.8% 34th position for quarter-size training images in non-occluded regions, 2 pixel threshold Lower error: Playtable, Vintage Higher error: ArtL, Pipes, PlaytableP  Comparison with the original SGM algorithm error higher by 1.8%

Comparison with original SGM Our method performs better (blue colour) in slightly slanted surfaces Performs less well (red colour) in areas of low texture Differences between the two methods: self-adjusting penalties (SGM-SAP) vs tuning-based penalties (original SGM) Note: Original SGM employs more refinement processes

Results in Middlebury 2006 Comparison of SGM results with and without automatic penalty estimation (using the optimal parameters of a tuning process*) Error of our method over the 21 pairs higher by only 0.87% (11.89% to 11.02%) [Census cost metric] and 2.27% (25.72% to 23.45%) [Absolute Differences]  SGM-SAP is expected to work well for any matching cost function * (Stentoumis et al., 2015)

Results for Herz-Jesu-K7 stereo-pair Registration of the generated point cloud onto the ground truth data (from laser scanning) via ICP: Average distance: 25 mm (~1.1 pixel) Standard deviation: 20 mm (~0.9 pixel)

Results for Herz-Jesu-K7 stereo-pair Registration of generated point cloud onto the ground truth data

3D models (Middlebury – Version 3) Stereo-pair: Motorcycle Ground truth for non-occluded regions SGM-SAP with Left-Right Consistency, subpixel interpolation and median filtering

3D models (Middlebury – Version 3) Stereo-pair: Piano Ground truth for non-occluded regions SGM-SAP with Left-Right Consistency, subpixel interpolation and median filtering

3D models (Middlebury – Version 3) Stereo-pair: Playroom Ground truth for non-occluded regions SGM-SAP with Left-Right Consistency, subpixel interpolation and median filtering

3D models (Middlebury – Version 3) Stereo-pair: Recycle Ground truth for non-occluded regions SGM-SAP with Left-Right Consistency, subpixel interpolation and median filtering

SGM-SAP with Left-Right Consistency, subpixel interpolation and 3D models (Middlebury – Version 3) Stereo-pair: Djembe SGM-SAP with Left-Right Consistency, subpixel interpolation and median filtering

Distances of point clouds Stereo-pair: PlaytableP Ground truth for non-occluded regions SGM-SAP with Left-Right Consistency, subpixel interpolation and median filtering

Distances of point clouds

fused model from two stereo-pairs 3D model (Kapnikarea church, Athens) fused model from two stereo-pairs

3D model (Kapnikarea church, Athens)

Conclusions Presentation of a novel approach (SGM-SAP) aiming at the self-adjustment of penalty values in Semi-Global Matching for any image pair and any matching cost method Automatic estimation of the penalties through a simple process of low computational requirements, relying on the DSI volume (which is already computed in the previous step of the matching process) No tuning of penalties is needed No dataset of “similar” images with corresponding ground truth disparity maps has to be available Evaluation on Middlebury-Version 3 stereo-pairs: results competitive to those from original SGM

Future work Testing with more cost functions and SGM-like approaches (“non-local methods”) Evaluation on the challenging KITTI dataset for autonomous driving Implementation in OpenCV

Thank You… for your attention!