1. Semi-Global Matching with self-adjusting penalties
Laboratory of Photogrammetry
National Technical University of Athens
Semi-Global Matching with self-adjusting penalties
E. Karkalou, C. Stentoumis, G. Karras
7th International Workshop 3D ARCH
3D Virtual Reconstruction and Visualization of Complex Architectures
1-3 March 2017, Nafplion, Greece

2. Introduction 3D Surface Reconstruction Active (laser and optical)
scanning
Passive
(image-based) approaches
competitive to laser scanning in terms of accuracy, cost and flexibility
a fundamental procedure is dense image matching:
automatic determination of pixel correspondences among images
for 3D surface reconstruction
by using rectified images
-epipolar geometry constraint-
a disparity map is produced
a core part for the majority of cultural heritage applications

3. Introduction stereo-matching advantages stereo-matching applications
limited number of images – speed
fixed geometry (e.g. stereo-camera)
stereo-matching applications
augmented reality through smart-phones
reconstruction from a limited number of images (e.g. historical, aerial)
robotics, autonomous navigation, …
cost computation dissimilarity measure to each pixel for every value in the disparity range (AD, SSD, NCC, filtered images, rank, census)
cost aggregation pixel cost is supported by the cost of neighbouring pixels
disparity optimization - WTA (winner-takes-all) - energy function
disparity refinement post processing of the final disparity map
typical algorithm
after [Scharstein & Szeliski, 2002 ]

4. Semi-Global Matching (SGM)
- stereo algorithm for the step of optimization - originally proposed by Hirschmüller (2005, 2008) - global 2D energy function: - 1D-cost approximation in each of 8 directions (paths): - advantages: accuracy, computational efficiency, simplicity

5. SGM Penalties
Imposed on disparity changes between neighbouring pixels up to 1 pixel (P1) or larger (P2)
P1 penalizes slightly slanted or curved surfaces; P2 penalizes depth discontinuities
Penalty adjustment is needed for every different pair of images or, if a different matching cost method is used, even for the same stereo-pair
If parameters have not been properly tuned, the performance of the algorithm may not be as efficient as expected

6. Contribution  Automatic estimation of SGM penalties, after the
computation of simple statistical properties of the DSI (Disparity Space Image) volume, already existing from the previous step of the algorithm
Penalties are regarded as being self-adjusted to the particular stereo-pair, in relation to the cost function used
No time-consuming tuning required – No ground truth disparity maps or multiple data for training are needed
Low computational requirements
 Evaluation of method via Middlebury benchmark and EPFL dataset

7. Self-adjusting penalty values
Penalty values influence pixel costs and, therefore, should be related to these
It is proposed that penalties are derived from the DSI representation S(x,y,l) of the initial cost
 Penalty estimation without user intervention

8. Final algorithm (SGM-SAP)
Initial matching cost
(Census transform & Hamming distance
or Absolute Differences of intensities)
Penalty estimation
SGM
Disparity selection
via WTA
Disparity refinement [optional]:
sub-pixel interpolation, left-right consistency, median filtering

9. Middlebury – Version 3 stereo-pairs
left image
right image
true disparity map

10. Results (Middlebury – Version 3)
raw results
sub-pixel interpolation
median filtering

11. Results in the Middlebury benchmark}
Overall error 22.8%
34th position
for quarter-size training images
in non-occluded regions, 2 pixel threshold
Lower error: Playtable, Vintage
Higher error: ArtL, Pipes, PlaytableP 
Comparison with the
original SGM algorithm
error higher by 1.8%

12. Comparison with original SGM
Our method performs better (blue colour) in slightly slanted surfaces
Performs less well (red colour) in areas of low texture
Differences between the two methods:
self-adjusting penalties (SGM-SAP) vs
tuning-based penalties (original SGM)
Note: Original SGM employs more refinement processes

13. Results in Middlebury 2006
Comparison of SGM results with and without automatic penalty estimation (using the optimal parameters of a tuning process*)
Error of our method over the 21 pairs higher by only
0.87% (11.89% to 11.02%) [Census cost metric] and
2.27% (25.72% to 23.45%) [Absolute Differences]
 SGM-SAP is expected to work well for any matching cost function
* (Stentoumis et al., 2015)

14. Results for Herz-Jesu-K7 stereo-pair
Registration of the generated point cloud onto the ground truth data (from laser scanning) via ICP:
Average distance:
25 mm (~1.1 pixel)
Standard deviation:
20 mm (~0.9 pixel)

15. Results for Herz-Jesu-K7 stereo-pair
Registration of generated point cloud onto the ground truth data

16. 3D models (Middlebury – Version 3)
Stereo-pair: Motorcycle
Ground truth
for non-occluded regions
SGM-SAP
with Left-Right Consistency,
subpixel interpolation and
median filtering

17. 3D models (Middlebury – Version 3)
Stereo-pair: Piano
Ground truth
for non-occluded regions
SGM-SAP
with Left-Right Consistency,
subpixel interpolation and
median filtering

18. 3D models (Middlebury – Version 3)
Stereo-pair: Playroom
Ground truth
for non-occluded regions
SGM-SAP
with Left-Right Consistency,
subpixel interpolation and
median filtering

19. 3D models (Middlebury – Version 3)
Stereo-pair: Recycle
Ground truth
for non-occluded regions
SGM-SAP
with Left-Right Consistency,
subpixel interpolation and
median filtering

20. SGM-SAP with Left-Right Consistency, subpixel interpolation and
3D models (Middlebury – Version 3)
Stereo-pair: Djembe
SGM-SAP with Left-Right Consistency, subpixel interpolation and
median filtering

21. Distances of point clouds
Stereo-pair: PlaytableP
Ground truth
for non-occluded regions
SGM-SAP
with Left-Right Consistency,
subpixel interpolation and
median filtering

22. Distances of point clouds

23. fused model from two stereo-pairs
3D model (Kapnikarea church, Athens)
fused model from two stereo-pairs

24. 3D model (Kapnikarea church, Athens)

25. Conclusions
Presentation of a novel approach (SGM-SAP) aiming at the self-adjustment of penalty values in Semi-Global Matching for any image pair and any matching cost method
Automatic estimation of the penalties through a simple process of low computational requirements, relying on the DSI volume (which is already computed in the previous step of the matching process)
No tuning of penalties is needed
No dataset of “similar” images with corresponding ground truth disparity maps has to be available
Evaluation on Middlebury-Version 3 stereo-pairs:
results competitive to those from original SGM

26. Future work
Testing with more cost functions and SGM-like approaches (“non-local methods”)
Evaluation on the challenging KITTI dataset for autonomous driving
Implementation in OpenCV

27. Thank You…
for your attention!