1. SFM under orthographic projection
matrix
2D image
point
3D scene
point
image
offset
Trick
Choose scene origin to be centroid of 3D points
Choose image origins to be centroid of 2D points
Allows us to drop the camera translation:

2. factorization (Tomasi & Kanade)
projection of n features in one image:
projection of n features in m images
W measurement
M motion
S shape
Key Observation: rank(W) <= 3

3. Factorization Factorization Technique
known
solve for
Factorization Technique
W is at most rank 3 (assuming no noise)
We can use singular value decomposition to factor W:
S’ differs from S by a linear transformation A:
Solve for A by enforcing metric constraints on M

4. Metric constraints
Orthographic Camera
Rows of P are orthonormal:
Enforcing “Metric” Constraints
Compute A such that rows of M have these properties
Trick (not in original Tomasi/Kanade paper, but in followup work)
Constraints are linear in AAT :
Solve for G first by writing equations for every Pi in M
Then G = AAT by SVD

5. Results

6. Extensions to factorization methods
Paraperspective [Poelman & Kanade, PAMI 97]
Sequential Factorization [Morita & Kanade, PAMI 97]
Factorization under perspective [Christy & Horaud, PAMI 96] [Sturm & Triggs, ECCV 96]
Factorization with Uncertainty [Anandan & Irani, IJCV 2002]

7. Bundle adjustment

8. CSE 576 (Spring 2005): Computer Vision
Structure from motion
How many points do we need to match?
2 frames:
(R,t): 5 dof + 3n point locations 
4n point measurements 
n  5
k frames:
6(k–1)-1 + 3n  2kn
always want to use many more
Richard Szeliski
CSE 576 (Spring 2005): Computer Vision

9. CSE 576 (Spring 2005): Computer Vision
Bundle Adjustment
What makes this non-linear minimization hard?
many more parameters: potentially slow
poorer conditioning (high correlation)
potentially lots of outliers
Richard Szeliski
CSE 576 (Spring 2005): Computer Vision

10. Lots of parameters: sparsity
Only a few entries in Jacobian are non-zero
Richard Szeliski
CSE 576 (Spring 2005): Computer Vision

11. CSE 576 (Spring 2005): Computer Vision
Robust error models
Outlier rejection
use robust penalty applied to each set of joint measurements
for extremely bad data, use random sampling [RANSAC, Fischler & Bolles, CACM’81]
Richard Szeliski
CSE 576 (Spring 2005): Computer Vision

12. Structure from motion: limitations
Very difficult to reliably estimate metric structure and motion unless:
large (x or y) rotation or
large field of view and depth variation
Camera calibration important for Euclidean reconstructions
Need good feature tracker
Lens distortion
Richard Szeliski
CSE 576 (Spring 2005): Computer Vision

13. Issues in SFM Track lifetime Nonlinear lens distortion
Prior knowledge and scene constraints
Multiple motions

14. every 50th frame of a 800-frame sequence
Track lifetime
every 50th frame of a 800-frame sequence

15. lifetime of 3192 tracks from the previous sequence
Track lifetime
lifetime of 3192 tracks from the previous sequence

16. track length histogram
Track lifetime
track length histogram

17. Nonlinear lens distortion

18. Nonlinear lens distortion
effect of lens distortion

19. Prior knowledge and scene constraints
add a constraint that several lines are parallel

20. Prior knowledge and scene constraints
add a constraint that it is a turntable sequence

21. Applications of Structure from Motion

22. Jurassic park

23. PhotoSynth

24. So far focused on 3D modeling
Multi-Frame Structure from Motion:
Multi-View Stereo
Unknown
camera
viewpoints

25. Next
Recognition
Human can do it well easily but computers can not yet.

26. Today
Recognition

27. Recognition problems What is it? Who is it? What are they doing?
Object detection
Who is it?
Recognizing identity
What are they doing?
Activities
All of these are classification problems
Choose one class from a list of possible candidates

28. How do human do recognition?
We don’t completely know yet
But we have some experimental observations.

29. Observation 1:
Q1: who is she?

30. The “Margaret Thatcher Illusion”, by Peter Thompson
Observation 1:
The “Margaret Thatcher Illusion”, by Peter Thompson
Q1: who is she?
Q2: which picture looks more natural?

31. The “Margaret Thatcher Illusion”, by Peter Thompson
Observation 1:
The “Margaret Thatcher Illusion”, by Peter Thompson
Human process up-side-down images separately

32. Observation 2: Jim Carrey Kevin Costner
High frequency information is not enough

33. Observation 3:

34. Observation 3:
Negative contrast is difficult

35. Observation 4:
Image Warping is OK

36. The list goes on
Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About

37. Face detection How to tell if a face is present?
You can ask people to see what they come up with
How to tell whether a pixel is on face or not?

38. One simple method: skin detection
Skin pixels have a distinctive range of colors
Corresponds to region(s) in RGB color space
for visualization, only R and G components are shown above
Skin classifier
A pixel X = (R,G,B) is skin if it is in the skin region
But how to find this region?

39. Skin detection Learn the skin region from examples Skin classifier
Given X = (R,G,B): how to determine if it is skin or not?
Learn the skin region from examples
Manually label pixels in one or more “training images” as skin or not skin
Plot the training data in RGB space
skin pixels shown in orange, non-skin pixels shown in blue
some skin pixels may be outside the region, non-skin pixels inside. Why?
Q1: under certain lighting conditions, some surfaces have the same RGB color as skin
Q2: ask class, see what they come up with. We’ll talk about this in the next few slides

40. Skin classification techniques
Skin classifier
Given X = (R,G,B): how to determine if it is skin or not?
Nearest neighbor
find labeled pixel closest to X
choose the label for that pixel
Data modeling
fit a model (curve, surface, or volume) to each class
Probabilistic data modeling
fit a probability model to each class

41. Probability Basic probability
X is a random variable
P(X) is the probability that X achieves a certain value or
Conditional probability: P(X | Y)
probability of X given that we already know Y
called a PDF
probability distribution/density function
a 2D PDF is a surface, 3D PDF is a volume
For skin detection, our random variable is multi-dimensional (color, label).
continuous X
discrete X

42. Probabilistic skin classification
Now we can model uncertainty
Each pixel has a probability of being skin or not skin
Skin classifier
Given X = (R,G,B): how to determine if it is skin or not?
Choose interpretation of highest probability
set X to be a skin pixel if and only if
Where do we get and ?

43. Learning conditional PDF’s
We can calculate P(R | skin) from a set of training images
It is simply a histogram over the pixels in the training images
each bin Ri contains the proportion of skin pixels with color Ri
This doesn’t work as well in higher-dimensional spaces. Why not?
Approach: fit parametric PDF functions
common choice is rotated Gaussian
center
covariance
orientation, size defined by eigenvecs, eigenvals

44. Learning conditional PDF’s
We can calculate P(R | skin) from a set of training images
It is simply a histogram over the pixels in the training images
each bin Ri contains the proportion of skin pixels with color Ri
But this isn’t quite what we want
Why not? How to determine if a pixel is skin?
We want P(skin | R) not P(R | skin)
How can we get it?

45. Bayes rule In terms of our problem: The prior: P(skin)
what we measure
(likelihood)
domain knowledge
(prior)
what we want
(posterior)
normalization term
The prior: P(skin)
Could use domain knowledge
P(skin) may be larger if we know the image contains a person
for a portrait, P(skin) may be higher for pixels in the center
Could learn the prior from the training set. How?
P(skin) may be proportion of skin pixels in training set

46. Bayesian estimation likelihood posterior (unnormalized)
Goal is to choose the label (skin or ~skin) that maximizes the posterior
this is called Maximum A Posteriori (MAP) estimation
= minimize probability of misclassification
Suppose the prior is uniform: P(skin) = P(~skin) =
0.5
in this case ,
maximizing the posterior is equivalent to maximizing the likelihood
if and only if
this is called Maximum Likelihood (ML) estimation

47. Skin detection results
Other application: Porn detection

48. General classification
This same procedure applies in more general circumstances
More than two classes
More than one dimension
H. Schneiderman and T.Kanade
Example: face detection
Here, X is an image region
dimension = # pixels
each face can be thought of as a point in a high dimensional space
What’s the problem with high-dim feature points? Cov requires a lot of parameters.
A way to resolve this problem is dimension reduction.
H. Schneiderman, T. Kanade. "A Statistical Method for 3D Object Detection Applied to Faces and Cars". IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2000)