1. Computational Vision CSCI 363, Fall 2016 Lecture 27 Structure from motion

2. 2D Motion is just the Beginning
2D image motion contains information about:
Relative depth of surfaces
3D motion of objects
3D structure of objects
Direction of observer motion
Among other things.

3. Structure From Motion
Structure from motion originally studied rigorously by Wallach and O'Connell (1953).
They studied wire-frame objects and examined peoples ability to judge the structure of the objects when moving.
The ability to see a 3D structure from a moving 2D image is known as the Kinetic Depth Effect.
Demo:

4. Inherent Ambiguity
How do we compute a 3D motion from a 2D image motion?
Given the 2D image motion, there are multiple possible 3D motions that could have generated it: ?
Image
plane v
To solve for 3D motion from 2D image information, we must use a constraint.
The Rigidity constraint assumes that the object is rigid.

5. The Rigidity Constraint
To solve for 3D motion from 2D image information, we must use a constraint.
The Rigidity constraint assumes that the object is rigid.
Ullman showed that, for an orthographic projection system, one can compute the 3D structure of an object given:
3 distinct views of
4 non-coplanar points in motion.
If there exists a rigid 3D structure consistent with these views, then it is unique.
The 2D positions of the points generate a set of equations that can be solved for the 3D structure.

6. The problem with noise
For perspective projection, one needs 2 views and 7 points.
If working with velocity fields, you need 5 points and 1 view (for perspective projection).
Problem:
This approach is very sensitive to noise in the velocity estimates.
This approach does not allow interpretation of non-rigid motions.

7. Information about Human ability
The human visual system takes some time to recover the 3D structure: It is not instantaneous.
Humans can cope with significant deviations from rigidity.
The visual system integrates multiple sources of information (e.g. static depth cues) to determine 3D structure.

8. Incremental Rigidity Scheme
The basic idea:
At each instant, generate an estimate of the 3D structure.
The recovery process prefers rigid transformations, but rigidity is not required.
The scheme can tolerate deviations from rigidity.
It should be able to integrate information over an extended viewing period.
It should eventually recover the correct 3D structure (or a close approximation). y x
(x1, y1, z1) -> (x1', y1', ?)
(x2, y2, z2) -> (x2', y2', ?)
(x3, y3, z3) -> (x3', y3', ?)
Initially set z to zero.

9. Maximizing Rigidity
We want to find the 3D model, S'(t), that maximizes the rigidity.
Therefore, we want to find new Z values that minimize the change in structure
Lij
lij
Current model
Model update
Image
Find zi to minimize:
The denominator makes sure that distant points count less, as they are less likely to be rigidly connected to each other.

10. Performance
The incremental rigidity scheme works fairly well for rigid motions. For a rotating object, it can find the 3D shape within about 2-5% error after a few rotations.
Mirror reversals may occur sometimes during the computations. (This may arise from orthographic projection).
The performance is similar to human structure from motion. 0o
90o
180o
360o z
Results of scheme after various rotations of object. x
720o
1440o

11. Properties of Incremental Rigidity Scheme
Veridicality--Usually get a reasonable approximation of the true structure.
Temporal extension--The time required is longer than for the 4 points, 3 views method.
There is some residual non-rigidity in computed 3D structure.
The improvement over time is non-monotonic. There may be some increase in error at times, before it decreases again.
Depth reversals--Sometimes this method exhibits spontaneous depth reversals in the estimated 3D structure.
May converge to a local minimum that's not the most rigid interpretation.
It can be influenced by static depth cues.
It can track moderate amounts of non-rigid motion over time.
In many ways, this scheme behaves similarly to the human system.

12. Psychophysics of Structure from Motion
Psychophysical results show that people can judge structure-from-motion from motion of points with limited lifetimes.
Demo of cylinder: Cylinder Demo
Stimulus: Rotating cylinder composed of dots. Dots have limited lifetimes (e.g. 50, 100, 150 msec)
People can see 3D structure of cylinder for point lifetimes as low as 125 msec.
This is faster than the incremental rigidity scheme can handle.

13. Hildreth extension of Incremental Rigidity
Hildreth and colleagues developed an extension of Ullman's incremental rigidity scheme, that uses velocity estimates instead of measure point positions.

14. Depth Estimate from 2D velocities
We can compare the 2D velocities that would be generated by the estimated depths of each point with the measured 2D velocities.
=>
Estimate depths, Zi, and velocities:
Minimize:

15. Combine with Incremental rigidity
The full model uses the 2D velocity estimates and the rigidity constraint:
Minimize: ED + lER
Where ED is from previous slide.
ER is rigidity error:
Temporal integration: Information from multiple frames is combined using Kalman filters. Velocity computed at one instant is used to predict depths at the next instant.
This tends to smooth out errors.

16. Results Estimated dot positions Surface reconstruction.
Surface reconstruction uses smoothness and position of image points to interpolate a surface between point positions.

17. Cylinder Perceptions
Ramachandran showed that people do not see cylinder structure accurately in some conditions:
Perception
Cylinders with different speeds.
Cylinders with different radii.

18. Model results for illusions
Different speeds
Illusion 2:
Different radii
The model generates structures similar to those perceived by people.