1 Real-Time Stereo-Matching for Micro Air Vehicles Pascal Dufour Master Thesis Presentation

2 Outline Introduction Stereo-Matching Evaluation Outlook Demo

3 PIXHAWK Computer vision on a MAV to enable autonomous action Current System: Quadrotor COM Express board with Intel Core 2 Duo Integrated GPU

4 Requirements Efficiency: other mission critical processes run in parallel 10 Hz estimated for object avoidance no more than 20% load of one CPU: 20ms at 10Hz Robustness: produce consistent results, dense disparity map

5 Stereo Vision Two cameras allow the triangulation of 3D points Solve the correspondence problem

6 Undistortion Epipolar geometry is based on pinhole cameras Real cameras record distorted images Distortion has to be corrected so epipolar geometry is correct

7 Rectification Epipolar lines align horizontally Search for correspondece becomes search in one dimension

8 Block-Matching Find correspondences by comparing blocks of pixels Fast implementation possible Difficulty: badly textured regions repeating textures occlusions

9 SAD Dissimilarity value: Sum of Absolute Differences Winner-take-all strategy: Disparity with lowest cost wins

10 Sliding window Slide window across epipolar line, only compute differences These differences are columns and can be precomputed for each line Idea: reuse computed data, so matching becomes independent of window width

11 Sliding window When changing line: the columns from the line above can be reused Matching becomes independent of window height

12 Improvements: SSE4 128bit registers Single Instruction, Multiple Data Eight differences with one instruction

13 Improvements: SSE4 minpos( ) returns smallest value and position in a register No branching required

14 Matching on the GPU GPU provides more computation power Implementation with programmable shaders: Compute texture differences Use texture sampling Exploit depth-test

15 Texture Differences

16 Texture Sampling

17 Sampling: Math Sampling can be used to compute the average of a pixel-block

18 Sampling Differences Sampling is applied to the texture differences Each pixel in the sampled texture corresponds to the average of a block of pixel intensity differences

19 Weighted Distribution Sum up dissimilarity values of blocks Weighted distribution: Large area has influence on the matching cost But center has bigger influence

20 Sum of Sampled Textures Sum up texels of sampled textures Nearest neighbor yields bad distribution

21 Sum of Sampled Textures Sum up texels of sampled textures with interpolation Weighted distribution, but not perfectly centered

22 Depth test One pixel per disparity is created, which to show on screen? Set the depth of a pixel to the SAD cost Set the color of the pixel to the disparity The pixel with the lowest depth/cost will is rendered This test is fast: implemented in hardware

23 Post-Processing Robustness can be increased with post-processing Variance in block as confidence Simpler: only variance of a line other possibility: difference between darkest and brightest pixel

24 Evaluation Middlebury’s Stereo-Vision Website allows evaluation of algorithms: percent of pixels at the correct disparity raw disparity map, only evaluate matching disparity of four different images evaluated

25 Evaluation Tsukub a Cones Teddy Venus

26 Evaluation Tsukub a Teddy Venus Cones

27 Evaluation AlgorithmTsukubaVenusTeddyCones SAD: 9x SSD: 9x GPU Percent of pixels at wrong disparity Threshold: 1

28 Evaluation AlgorithmTsukubaVenusTeddyCones SAD: 9x SSD: 9x GPU Percent of pixels at wrong disparity Threshold: 2

29 Performance Undistortion, Rectification, Resizing SAD Matching linearNNC++with SSE GPU ImplementationConfidence Map integrated GPU dedicated GPU C++with SSE CPU load

30 Outlook Use of disparity map: object avoidance possibly SLAM visualization Camera framework with synchronization Realistic testing when quadrotor is finished

31 Acknowledgements Marc Pollefeys Friedrich Fraundorfer Lorenz Meier The whole team

32 DEMO