3. Contents Description of the big picture Theoretical background on this work The Algorithm Examples

4. Depth reconstruction process Image acquisition Camera modeling Feature acquisition Image matching Depth determination Interpolation

5. Image acquisition (in aerial photo) Images are vertical (the camera directed downwards) Images are taken from high point Images are taken at very small time intervals Rotation is minimal The height is almost the same

6. Aerial photo

7. Camera modeling To provide function that maps pairs of corresponding points from two stereo images onto scene points one need the camera model: Focal length of the camera Focal length = Zoom Camera location Tilt – No tilt

8. Feature acquisition Pixels grouped into a region contain match more information There exists two techniques of image matching: –Area matching: Compare pixels in both images –Feature based: Corners, junctions, straight or curved segments or other features are extracted from images and then are being matched The algorithm combines the both techniques.

9. Problems in feature based matching – occlusion

10. Feature acquisition problems – different angle (not our case)

11. Image matching Both images are projections of the same 3D object, thus along with camera model one could reconstruct the original 3D object. Common matching constraints: –Epipolarity constraint: –Epipolarity constraint: point on a given epipolar line can match only to the same point on the corresponding epipolar line on the other image. –Uniqueness constraint: –Uniqueness constraint: no two objects in the scene project to the same object in an image –Ordering constraint: –Ordering constraint: projections of two objects along an epipolar line can't swap order

12. Image matching continued Common matching constraints (continued): –Smoothness constraint: –Smoothness constraint: difference of disparities of two nearby objects doesn’t exceed a threshold value. –Orientation constraint: –Orientation constraint: difference of orientations of two matching segments is bounded. –Contrast constraint: –Contrast constraint: matched features has the same contrast sign.

13. Matching techniques Best resemblance approachBest resemblance approach Coarse to fineCoarse to fine Evaluation-propagation approachEvaluation-propagation approach There are more techniques to do it.

14. Best resemblance approach Use a local pixel information to decide a potential match Match decision is made independently of the others (pixels) Main match criteria is a good resemblance between regions of pixels in two stereo images.

15. Coarse to fine approach Hierarchical approach – matching information is passed from coarse to finer level of computation. Features are matched with different level of resolution. The coarse disparities are used to constrain the finer matches.

16. Evaluation propagation approach Uses neighborhood relations between matching features. Potential matches form a correspondence graph whose nodes are pairs of matching segments. Edges of graph join features that doesn’t violate the smoothness and uniqueness constraints.

17. Our algorithm’s technique. The algorithm uses combination of Best resemblance and evaluation- propagation. It builds neighborhood graph. Initial disparity of some features is propagated to other features. Local matches are evaluated with the best resemblance approach.

18. Theoretical Background The algorithm is another implementation of partial Hausdorff distance Pixel based method and segment based method

19. The Algorithm Preprocessing Boot process Evaluate-propagate stage Postprocessing

20. Pixel based method Amit’s thesis Divide both images to grid of search windows Find the disparity of corresponding windows, while they are treated as set of black pixels. If percent of matching pixels is small, repeat the previous step using the previously found disparity as approximated initial disparity. The major disadvantage in our case is the impossibility to obtain objects original height, because the calculated height will be the average per search window!

21. Segment based method There exists a method, which treats both images as set of segments. It’s main disadvantage is Runtime complexity The algorithm will mix both techniques!

22. Preprocessing - 1 Use edge detection on both images. Create non crossing segments. Split wiggly segments Build weakly visibility graph.

24. Split wiggly segment Find the farthest point P where d>= some threshold value. Split the segment at point P. Repeat recursively.

25. Weak visibility

26. Weakly visibility graph Vertices – segments in left image. Edges connects vertices if the corresponding segments are weakly visible. Edge’s weight is the minimal vertical or horizontal distance between the corresponding segments. Graph is built by sweepline technique.

27. Sweepline structure Y - A x i s X 12345678 1 2 3 4 5 6 7 8 9 5 4 3 4 5 6 N L L L L N N N Seg_field X_field

28. Sweepline technique Create events stack Iteratively process the event points e from left to right: Y is the Y coordinate of e (black pixel of segment I) a.If seg_field(Y) <> Nil then I and J (pointed by seg_field) are neighbors, and their distance is X-X_field(Y); If J is already in the Graph then the distance is the minimum b.Seg_filed(Y)=I, x_field(Y)=X

29. Preprocessing - 2 After the graph was built its MST is constructed by PRIM algorithm. Neighboring vertices are visible and close Long and strait segment is chosen as the root. In the right image distance transform is computed and each pixel hold the coordinates of the closest black pixel and distance from it.

30. Why initial disparity ? Wrong values may be obtained if there is no way to minimize the search window for a specific segment.

31. Initial disparity Find the most long and straight segment in the right image. Find it in the left image. Let its corresponding vertex’s initial disparity be its disparity.

32. Evaluate – propagate stage BFS like algorithm is implemented in this stage. Calculation for a vertex in BFS: –Find its disparity using the disparity of its parent vertex –Propagate it’s disparity as the initial disparity to it’s children

33. Segment’s disparity Segment disparity – disparity of the whole segment. Pixel disparity – for pixel of some segment S, distance to the closest black pixel in the right image, under translation t, which is it’s segment’s disparity. Each pixel in left image holds the closest black pixels from the right image

34. Segment’s disparity (2) Each segment is being searched in the window defined by it’s initial disparity and double the maximum allowed disparity For each tested disparities two values are minimized: –D 1 Hausdorf distance of segment –D 2 Hausdorf distance of segment with its MST neighbors 80% of pixels are used during the match process Best disparity is chosen among competing disparities

35. Post processing Pixels may hold more then one close black pixel or none. –If closest pixel is too far it remains unmatched –If it is unmatched it remains unmatched –If it have more then one matched pixel 1)Remove already matched candidates 2)Among the rest choose pixel from segment which has the most matching pixels with the segment of the pixel

36. Runtime The runtime of algorithm is- Which may be represented by Where A n is average number of neighbors L is total number of black pixels W is the size of search window

37. Depth determination When images are rectified (only horizontal displacement of images)

38. Interpolation The problem – feature based approach doesn’t provide depth for the whole scene. There are two methods: – Suppose that there is a continuous function of depth that can be approximated to the sparse depth array achieved in previous steps – Try to fit known geometric models to the sparse depth array.