1. Reconstruction of a Scene with Multiple Linearly Moving Objects Mei Han and Takeo Kanade CISC 849

2. Factorization Spatial Factorization Rigid body Orthographic Projection Single Body -Tomasi and Kanade Multibody -Costeira and Kanade Scaled Orthographic Projection -Mei Han and Kanade Paraperspective Projection -Poelman and Kanade Perspective Projection Non-rigid body Temporal Factorization -Manor and Irani

3. Road Map Previous works Pros and Cons of this method Feature Point representation Scene Reconstruction Rank Approximations Reconstruction under weak Perspective projection Experiments Conclusion References

4. Previous Works Factorization Method –Tomasi and Kanade - recover the shape of one object and camera motion Multibody Factorization Method – Costeira and Kanade – each object is regarded as one independent structure/motion space. It requires the object to be close and clear, with information available in the form of feature points

5. Pros and Cons No requirement of knowledge of the number of moving objects No requirement for pre-segmentation/motion detection Recover scene structure, camera motion and moving trajectories simultaneously Assumptions Objects moving linearly and with constant speeds

6. Feature Point representation m feature points tracked over n frames Some feature points are static others move linearly with constant speed (static points are treated as points moving with 0 velocity) p ij = s j +iv j where i = 1  n, j=1  m, s j is the point position at frame 0 and v j is the moving velocity

7. Feature Point representation Under Orthographic projection Putting them into the measurement matrix W

8. Feature Point representation By Factorization method

9. Scene Reconstruction Moving World Coordinate system Center of gravity of the points moving with constant linear speed equal to the average of all the velocities v j Transform 3d representation such that origin is at the center of gravity and moving with the average velocity

10. Scene Reconstruction Decomposition W is utmost rank 6  M = 2n£6 and S = 6£m Normalization Required to find the parameters of A where A (6£6) = [A 1 (6£3) A 2 (6£3) ]

11. Scene Reconstruction Imposing linear constraints Simplifications since

12. Scene Reconstruction 8 constraints per frame  8n equations to solve 21 unknowns (why?) of Least squares solution to estimate the unknown elements rank 3 decompositions of  (why?)

13. Scene Reconstruction Shape matrix consists of scene structure and velocities represented in “moving world coordinates” Need to transform representation to a fixed world coordinate system Coordinate System moving at a average velocity of all moving points  static points have same velocity but opposite direction

14. Scene Reconstruction In the presence of noise different static points have different velocities Usually there are more static points that moving objects Get the common velocity using a RANSAC scheme Find the inliers that share common velocity – static points The outliers represent the moving points -> number of moving objects can therefore be detected

15. Degenerate cases Rank-3 case No moving object in the scene  Tomasi and Kanade Factorization method Rank-4 case One or more objects moving in the same/opposite direction Analysis is exactly the same except that instead of a n xi and n yi there would be the x-elements of the i th rotation axis Rank-5 case Velocities of objects lie in a 2 dimensional space Analysis slightly more complicated with both the xand y elements of the i th rotation axis instead of n xi and n yi

16. Scene Reconstruction under Weak Perspective Projection Under weak perspective projection Similar to the previous derivation

17. Experiments and Results 24 images taken 23 feature points were manually tracked over the image sequence Reconstruction using the first 18 frames Arrows indicate the motion trajectories

18. Experiments and Results Airplane flying over a scene with multiple moving cars First 80 fames of 90 frame sequence used for the reconstruction 35 points manually selected in the first frame and automatically tracked in the remaining frames

19. Conclusions Assuming the objects are moving linearly with constant speed, the paper describes a unified geometrical representation incorporating the static scene and the moving objects Using the constraints between the camera motion and the shape matrix, the algorithm provides a reliable reconstruction of the scene

20. Future Directions Noise sensitivity analysis Extension of this work to perspective projection

21. References M. Han and T. Kanade, “Reconstruction of a Scene with Multiple Linearly Moving Objects”, IJCV, 59(3), pp285-300, 2004 M. Han and T. Kanade, “Scene Reconstruction from Multiple Uncalibrated Views”, CMU-RI-TR-00-09, January 2000 M. Han and T. Kanade, “The Factorization Method with Linear Methods”, CMU-RI-TR-99-23, October 1999 M. Han and T. Kanade, “Perspective Factorization Method for Euclidean Reconstruction”, CMU-RI-TR-99-22, August 1999 João Costeira and Takeo Kanade, “A Multi-body Factorization Method for Motion Analysis”, CMU-CS-TR-94-220, September 1994 C. Tomasi and T. Kanade, “Shape and Motion from Image Streams: A Factorization Method - Full Report on the Orthographic Case”, CMU-CS-92-104