#? rahul swaminathan (T-Labs) & professor patrick baudisch hci2 hasso-plattner institute determining depth

two subproblems Matching Finding corresponding elements in the two images Reconstruction Establishing 3-D coordinates from the 2-D image correspondences found during matching

a little recap on reconstruction

camera scene lighting graphics light computer

camera scene lighting vision light

computer camera scene lighting light

scene the camera sees a red pixel let’s assume it correctly classifies it as “glass of red wine” screen  but, the red wine could be anywhere along this line

computer two cameras scene lighting light

triangulate the location of the actual glass

wine glass screens

two subproblems Matching Finding corresponding elements in the two images Reconstruction: done Establishing 3-D coordinates from the 2-D image correspondences found during matching

two subproblems Matching: harder Finding corresponding elements in the two images Reconstruction: done Establishing 3-D coordinates from the 2-D image correspondences found during matching

matching structured light

scene Could we replace one camera with a projector? two cameras lighting

structured light :: the process of projecting a known pattern of pixels onto a scene

pattern is disturbed when depth changes

patterns used

gray code 1

Could we achieve the same result with less images?

use (cos)-wave pattern instead of b/w 2

pattern needs processing caveat

Turns out to be a not too hard problem: flood-fill algorithm already provides acceptable solution

continues gradient result from both

Microsoft Kinect 3

Anoto pen 4

matching two cameras

computer two cameras scene lighting light

main approaches 1.pixel/area-based 2.feature-based

problems Camera-related problems - Image noise, differing gain, contrast, etc.. Viewpoint-related problems: - Perspective distortions - Occlusions - Specular reflections

camera positioning baseline

More matching heuristics Always valid: (Epipolar line) Uniqueness Minimum/maximum disparity Sometimes valid: Ordering Local continuity (smoothness)

Area-based matching Finding pixel-to-pixel correspondences For each pixel in the left image, search for the most similar pixel in the right image

Area-based matching Finding pixel-to-pixel correspondences For each pixel in the left image, search for the most similar pixel in the right image Using neighbourhood windows

Area-based matching Similarity measures for two windows SAD (sum of absolute differences) SSD (sum of squared differences) CC (cross-correlation) …

Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

LeftDisparity Map Images courtesy of Point Grey Research Correspondence Using Correlation

Image Normalization Even when the cameras are identical models, there can be differences in gain and sensitivity. The cameras do not see exactly the same surfaces, so their overall light levels can differ. For these reasons and more, it is a good idea to normalize the pixels in each window:

matching features

problems

Scale change Rotation Occlusion Illumination ……

SIFT :: Scale Invariant Feature Transform; transform image data into scale-invariant coordinates relative to local features

result

combining both

2 angles and one side are known  height of the triangle can be computed

wine glass screens

the underlying problem is: compute the intersection of two lines

commonly compute “depth” image

problems

oclusion

end

#11 professor patrick baudisch hci hasso-plattner institute title

:: interactive of the day

#11 professor patrick baudisch hci hasso-plattner institute title

main text is 28 pt Arial, dark gray with highlighted text is good green, and bad orange both in bold face. Include commas etc. in highlighting

36pt text overlay text on 40% black 1 label

1.benefit 1 2.benefit 2 benefits:

:: <text to define it including highlighted text only black text in deck

by Saturday upload storyboards for four tasks to the wiki assignment

E title 2x1min (this is an in-class exercise) in teams of : 1.step 1 2.step 2 Go!