1. Communication Systems Group Technische Universität Berlin S. Knorr A Geometric Segmentation Approach for the 3D Reconstruction of Dynamic Scenes in 2D Video Sequences Technical University of Berlin Communication Systems Group Director: Prof. Thomas Sikora Sebastian Knorr, Evren Imre*, A. Aydin Alatan*, and Thomas Sikora * Middle East Technical University EEE Department Assoc. Prof. Aydin Alatan

2. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20062 Outline Motivation Overview of proposed solution Feature tracking and motion segmentation Multiview 3D reconstruction Simulation results Future work

3. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20063 Motivation S. Knorr, E. Imre, B. Özkalayci, A. A. Alatan, and T. Sikora, A Modular Scheme for 2D/3D Conversion of TV Broadcast, 3DPVT'06 E. Imre, S. Knorr, A. A. Alatan, and T. Sikora, Prioritized Sequential 3D Reconstruction in Video Sequences of Dynamic Scenes, ICIP'06

4. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20064 Motivation This work: 3D reconstruction of dynamic scenes with independently moving objects (IMOs) using Structure from Motion (SfM) techniques Scenes are captured with a single camera Static background and IMOs are reconstructed independently Main goal: 3D reconstruction of TV broadcast video

5. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20065 Examples

6. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20066 Overview of the Proposed Solution Feature Detection & Tracking Iterative F-Matrix Estimation & Motion Segmentation Initial Structure Computation Prioritized Sequential Structure Estimation

7. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20067 Outline Motivation Overview of proposed solution Feature tracking and motion segmentation Multiview 3D reconstruction Simulation results Future work

8. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20068 Feature tracking and motion segmentation Harris-Corner-Detector for feature selection pyramidal Lucas-Kanade tracker to track features along the whole sequence Geometric Robust Information Criterion (GRIC) for keyframe selection 2D motion model, H (homography), for small baselines vs. 3D motion model, F (epipolar geometry), for large baselines [P.H.S. Torr, A.W. Fitzgibbon and A. Zisserman (ICCV'98)]

9. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 20069 Keyframe Selection 240-frame sequence (TUB-room) with 14 keyframes

10. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200610 Motion segmentation F-matrix estimation for consecutive keyframes (RANSAC and re-RANSAC)  labeling of background and IMO trajectories Increasing the number of features on the IMO (guided-matching) For each independent motion in the sequence, there exists a corresponding F-matrix, F i, which fulfills the epipolar constraint

11. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200611 Outline Motivation Overview of proposed solution Feature tracking and motion segmentation Multiview 3D reconstruction Simulation results Future work

12. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200612 Multiview 3D reconstruction Fast convergence to a reliable estimate: Since the quality of subsequent reconstructions depend on the current (intermediate) structure estimate, errors in the first few frame pairs may cause the entire estimation procedure to collapse. Fast recovery of the scene structure: The number of reconstructed 3-D points should be maximized, while processing a minimum number of frame pairs. Main goals:

13. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200613 Prioritized Sequential Reconstruction 1. Compute the initial reconstruction and camera path (e.g. Pollefeys et al.). 2. Compute the priority metric and order the frame pairs. Given the internal calibration parameters and the feature trajectories: 2-view rec. 1:6 2-view rec. 2:4 2-view rec. 7:9 Add 5, 2-view rec. 1:5 Add 3, 2-view rec. 3:4 Merge sub- reconstruction 3:6 Add 8, 2-view rec. 3:8 Merge sub- reconstruction 7:8 Two-view reconstruction Sub-estimate fusion using 3D-2D correspondences Sub-reconstruction fusion using 3D-3D correspondences Ordered pair list: 1:6, 2:4, 1:5, 3:4, 3:6, 7:9, 3:8, 7:8

14. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200614 Outline Motivation Overview of proposed solution Feature tracking and motion segmentation Multiview 3D reconstruction Simulation results Future work

15. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200615 Simulation Results (1) frame 1 frame 170

16. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200616 Simulation Results (2) frame 1 frame 200

17. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200617 Future Work 2D TV-Display Autostereoscopic Display Shutter-Glases Anaglyph

18. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200618 Future Work initial dense depth maprefined depth mapvirtual right stereo view original left stereo view

19. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200619 Future Work original left stereo viewvirtual right stereo view

20. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200620 Thank you for your attention! Questions? ?

21. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200621 Motion segmentation For each independent motion in the sequence, there exists a corresponding F-matrix, F i, which fulfills the epipolar constraint where x 1 and x 2 are corresponding points in two views. A RANSAC-based F-matrix estimation algorithm identifies the feature pairs belonging to the dominant motion and labels the rest as outliers. Some of the outliers should satisfy the epipolar constraint according to a new F-matrix, which corresponds to the motion of an independent moving object (IMO)

22. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200622 Segmentation Algorithm 1.Compute the F-matrix corresponding to the first and the second key-frame by using a RANSAC-based procedure and label the inliers as background trajectories. 2.Compute the F-matrix on the outliers of step 1 by using again RANSAC and label the inliers as IMO trajectories. 3.Compute the centroid of the inliers of step 2 and check their distances. If the distance is higher than a threshold, reject the feature. 4.Increase the number of features on the IMO in consecutive key- frames with guided-matching. 5.Repeat step 2 to 4 as long as the F-matrix estimation is still reliable and most of the remaining features are spatially close. 6.Proceed to the next key-frame. Estimate the F-matrix between the last and the current key-frame for each motion using the labeled trajectories and classify new trajectories using step 1 to 5. 7.Repeat step 6 for all key-frames.

23. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200623 Priority Metric The priority metric, p, utilized in the algorithm to evaluate the feasibility of a frame pair for reconstruction is defined as where d is the baseline distance between the cameras, n the number of feature matches, a, b and c are the design parameters of the sigmoid function appearing in the second term. The non-linear (sigmoidal) weighting keeps the contribution of the second term within a bound, when there is a relatively small or large number of matching features.

24. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200624 Simulation Results (3) # frame pairs (used/total) # 3D points (recov./total)  reproj. error Cliff45 / 57785890 / 82120.95 Palace25 / 199002771 / 35460.90 TUB- Room 17 / 286804716 / 60950.27

25. Communication Systems GroupS. Knorr Technische Universität Berlin EUSIPCO 200625 Segmentation Results frame 83 frame 1 frame 201 frame 170