1. 3D From 2D
Image Processing Seminar
Presented by: Eli Arbel

2. Topics Covered Inferring 3D surfaces from 2D contours using symmetry
From a single view
Using symmetry to enhance structure from motion methods
Sequence of images

3. Inferring 3D surfaces from 2D contours using symmetry

4. The challenges
Recover orientations of 3D surfaces projected on 2D image.
Ambiguity - Infinite number of interpretations for a single image
Cannot be done without some assumptions

5. The challenges, cont`d
Still, humans are able to perceive 3D surfaces in a single image…

6. Monocular Cues Shading Texture Contour lines What about conflicts ?
Reflectance properties of the surface, light sources
Texture
Prior knowledge required
Contour lines
Give shape information near the contours only
Conflicts:
Shading Vs. contours – humans prefer contours cues
Perception of shape from Color images is not faster
Methods based on other cues then contour require stronger assumptions
What about conflicts ?

7. Shape from contour – some examples

8. The method
Based on symmetries in the scene to infer surface orientation
Assumes orthographic projection
Also considers interaction between surfaces

9. Symmetries Defined as pointwise correspondence between two curves
Curves of symmetry
Lines of symmetry
Axis of symmetry

10. Symmetries – Cont’d Parallel symmetry: Skew symmetry:
Parallel – correspondence function
Skew – axis straight, line of symmetries at constant angle (not necessarily orthogonal)

11. Qualitative inferences from symmetries
Symmetries can be major source of information for extracting shape from contour
Can give constraints on interpretations of single surfaces
Definition: General Viewpoint
A scene is said to be imaged from general viewpoint if its perceptual properties are preserved under slight
variations of the viewing direction
In particular: straightness, parallelity and curves symmetry

12. Qualitative inferences – Case I
Case I: one skew symmetry covers the entire boundary of the surface
That kind of contour – if bounded by non-limb edges – must be planar under the assumption of general viewpoint
Definition: Limb edges
points on the surface whose normal is orthogonal to the viewing direction
Non-limb edges: wireframes

13. Qualitative inferences – Case I
Proof:
Parallel lines in the image plane must a projection of parallel lines in 3D space. Thus, the symmetry lines are parallel in 3D.
Axis of the skew symmetry must be a projection of a straight line.
The axis of symmetry is obtained by connecting the midpoints of the symmetry lines
From 1,2 and 3 above, the 3D contour must be planar.

14. Qualitative inferences – Case II
Case II: Boundary is covered by two symmetries, at least one of them must be parallel symmetry
Figures belong to case II give us the most information about surface shape

15. Qualitative inferences – Case II
Definition: Zero Gaussian Curvature (ZGC) Surface
Surface in which the product of the minimum and maximum curvature is zero everywhere
Non ZGC
ZGC

16. Qualitative inferences – Case II
If the surface generates one parallel symmetry and one skew symmetry with straight curves of skew symmetry which are also the lines of symmetry for parallel symmetry, then the surface must be ZGC surface.
Assuming general viewpoint and no surface variations that do not produce edges in the image plane
Can be proved…

17. Qualitative inferences – Case III
Case III: Includes all remaining cases.
Contours satisfy specific properties
Figures which have some missing boundaries
All other cases.

18. Recovering Surface Orientations
Reminder: In order to recover a 3D surface from 2D projection, we need to find the orientation (normal) of each point of the surface
Orientations of a surface can be recovered using constraints system derived from symmetries in the image
We will focus on ZGC surfaces.The presence of ZGC surfaces is indicated by observing the properties given above.

19. Some Math… Parametric representation of curves:
S(r) = (x(r), y(r)) in 2D
S(r) = (x(r), y(r), z(r)) in 3D

20. Some More Math... Parametric representation of a surface:
X(u,v) = (x(u,v), y(u,v), z(u,v))
Dini’s surface=X(u,v):
x = cosu*sinv
y = sinu*sinv
z = cosv+log(tan(v/2))+u/50

21. And a little more... Gradient Space:
Given a plane, , its normal N is given by the vector (A, B, C) - the gradient vector.
Plane can be rewritten as , and so its normal
vector: or (p,q,1) where and
(p,q) defines 2D space such that every point at this space corresponds to the normal of a plane in 3D

22. One Last Thing... Definition: Rulings
the lines connecting corresponding points of two curves forming parallel symmetry. (correspondence is determined by the tangent of the curves)
Note: orientations of a ZGC surface does not change along a ruling.
Rulings

23. Curved Shared Boundary Constraint - CSBC
This constraint relates the orientations of two surfaces of opposite sides of an edge X2 X1
Two surfaces: Xi(u,v), i=1,2 N1
Normal vectors: Ni(s)=(pi(s), qi(s), 1), i=1,2 (in p-q space) N2
Intersection curve:
Tangent vector:
Since is on the tangent planes of both X1 and X2, it’s
orthogonal to both N1(s) and N2(s). That is:

24. CSBC – cont’d
A stronger constraint can be obtained if we assume that the intersection curve is planar.
Let N1=(pc,qc,1) be the normal of the plane on which lies on. From this constraint:
we get: X2 X1
Note that can be extracted from the image

25. Inner Surface Constraint - ISC
This constraint restricts the relative orientations of neighboring points within a surface, using the image of the surface’s rulings.
Let X(u,v)=(x(u,v), y(u,v), z(u,v)) be a parametric representation of a surface, and v be along the direction of the minimum curvature (rulings for ZGC surfaces).
ISC intuitive description: as we move along the u parameter (axis of symmetry), the surface orientation should move in p-q plane in a direction orthogonal to the direction of the rulings.
(pi,qi) q p
(pi+1,qi+1) • Ri i
i+1

26. ISC – cont’d
The change in the surface’s orientation when moving from point i to point i+1 in the p-q space is given by the vector (pi+1-pi, qi+1-qi)
Let (xi, yi) be the direction of the ruling Ri.
(pi,qi) q p
(pi+1,qi+1) • Ri
According to ISC, the following equation should hold:
Again, (xi, yi) can be computed directly from the image

27. Orthogonality Constraint - OC
This constraint is derived from the assumption of orthogonality between the axis of parallel symmetry and the lines of symmetry.
For ZGC surfaces, this constrained implies that the cross section must be along the lines of maximum curvature. However, in general, this conflicts with our drive to perceive the cross section as being planar.
Cut along Curves of
maximum curvature ?
Preferred perception
OC perception

28. OC – cont’d Given a ZGC surface:
A is the tangent vector of the axis of symmetry B
B is the tangent vector of the ruling
Let N=(p,q,1) be the normal of the tangent plane at that point. Since A and B are on
the tangent plane of the surface, they can be represented as:
From the orthogonality constraint, we get or:

29. Combining the Constraints
In order to recover 3D surface from its 2D projection, we need to know the orientation at each point in the surface.
For a ZGC surface ,the orientation along the ruling is constant, thus it is sufficient to find the orientation of a single point on the ruling.
These leaves us with computing the orientation of points along the axis of symmetry

30. Combining the Constraints – cont’d
Suppose the surface orientation is to be computed for n points:
CSBC:
Equations: n
ISC:
n-1
OC: n
We have 3n-1 constraints equation producing 2n+2 unknowns.
For n > 3, the system is over constrained

31. Combining the Constraints – cont’d
Because the system is over constrained, it may be impossible to find interpretation for the contours such that all the constraints are obeyed by the surface.
OC and the planarity of the cross section assumed by CSBC are usually in conflict.
However, there are cases where these set of constraints may give a unique answer or even leave one degree of freedom unconstrained.

32. Recoverable Surfaces Circular Cones:
A circular cone in linear straight
homogenous generalized cylinder,
whose cross section is a circle.
These are the only surfaces that have a unique solution to the three constraints.

33. Recoverable Surfaces – cont’d
Cylindrical surfaces
A cylindrical surface is a ZGC
surface for which rulings are
parallel to each other in 3D.
With these kind of surfaces, we have one degree of freedom – qc in CSBC.

34. Recoverable Surfaces – cont’d
General ZGC surfaces
For general ZGC surfaces, the three constraints cannot be satisfied exactly.
In most cases, planarity assumption is stronger then the orthogonality assumption.
The solution is to maximize the orthogonality while keeping CSBC and ISC satisfied exactly.
Again, the degree of freedom is in the orientation of the cross section plane, namely (pc,qc).

35. Estimating (pc,qc)
The method is based on observations of human perception preferences. In particular:
We prefer compact shapes.
We prefer medium slant to very high or very low slant.
We have a large range of uncertainty for the perceived slants.

36. Estimating (pc,qc) – cont’d
An ellipse-fitting process is applied on the top surface to approximate an ellipse to the cross section.

37. Estimating (pc,qc) – cont’d
After the ellipse is fitted, the orientation of the circle that would project as the fitted ellipse is found.
This orientation becomes (pc,qc).
A final update of qc is performed to reflect the bias humans have in orienting the cross section toward 45º.
The above technique was developed as a consequence of experimental results with humans.
It was found that humans estimation of top planes slants is imprecise with an average standard deviation of 8º.
The average of the differences between the algorithm estimation and human estimation are 6º.

38. Testing The Algorithm
The inputs to the program are collection of curves grouped into closed regions using continuity
Each closed region is considered to be a surface
Each curve in the region is checked for parallel symmetry against every other curve in the region.
Two curves are considered to be parallel symmetric if they return low symmetry error which is defined as:

39. Testing The Algorithm – cont’d
Surfaces containing parallel symmetry are treated as curved and others as planar.
For curved surfaces, the curves joining parallel symmetric curves are checked if they are straight, which confirms that the surface is ZGC.
For each surface the orientation of the planar cross section (pc,qc) is computed.
Orientation at each point (pi,qi) on the surface is computed using ISC and CSBC constraints.

40. Some results

41. Some results – cont’d

42. Some results – cont’d

43. Using Bilateral Symmetry To Improve 3D Reconstruction From Image Sequences

44. Introduction Structure from motion methods
Like stereo with single camera
Static scene
Sequence of images, each one is a different view of the object
Reconstruction based on features
Like other methods, sensitive to noise

45. Introduction – cont’d

46. Noisy Projections
Noise can be present in the 2D projections of the scene due to measurements deviations, motion of the camera, etc..
3D mirror symmetry is one of the most common symmetry in our environment
In case the reconstructed object is known to be mirror symmetric, a symmetrization procedure can be applied on the noisy projections to enhance reconstruction of the 3D object

47. The Framework
Apply symmetrization procedure to the 2D data with respect to the 3D symmetry
Reconstruct the object using any structure from motion method
Given the 3D configuration of stage 2, find the closest mirror-symmetric configuration to it

48. 3D symmetrization Consider a 3D configuration of points
If the configuration is from 3D mirror-symmetric object then for every point there exist a point
which is its counterpart under reflection
Definition: Symmetry Distance
Let be a mirror-symmetric configuration derived from The symmetry distance is the quantity:

49. 3D Symmetrization Algorithm
Given a configuration of points in :
1. Divide the points into sets of one or two points. For instance: {P0,P0}, {P1,P3}, {P2,P2}. This defines a matching on the points. •
3. Find the optimal rotation and translation which minimizes the sum of squared distances between the original points and the reflected points •
2. Reflect all points across a randomly chosen mirror plane, obtaining the points .
4. Average each original point with its reflected point
obtaining the point The points are mirror symmetric.
5. Evaluate the Symmetry Distance
6. Minimize the Symmetry Distance by repeating steps 1-5 with all possible divisions of points to sets.

50. Complexity Of Matching
Matching of feature points is of exponential complexity
However, by constraining the search space of all possible matches, complexity can be greatly reduced •
Graph Matching
rank 1
rank 3
rank 4
P0 and P1 need higher order connectivity consideration
Is it Sufficient ?
We might need to consider higher order connectivity as well (maximal…)

51. Complexity Of Matching – cont’d
In the above example the number of possible matchings was reduced to 2
For the class of cyclically connected configurations, the number of possible matchings is reduced to linear • • • • • • • • • • • • • •

52. Complexity Of Matching: Heuristic Approach
The above approach assumes matching is to be found prior to finding the reflection plane
But matching and the reflection plane is related to each other:
Given a matching we can determine the reflection plane
Given the reflection plane, we can constrain the possible matchings

53. Complexity Of Matching: Heuristic Approach – cont’d
The following heuristic can be used:
For every possible pair of points determine the corresponding reflection plane (the plane perpendicular to and passing through the mid point of the segment connecting the two points)
Build a histogram, of all reflection planes
Peaks at the histogram will point at candidates for the optimal reflection plane

54. 2D symmetrization Definition: Projected mirror-symmetry constraint
All Segments connecting pairs of mirror-symmetric points of a mirror-symmetric 3D object, projected on a 2D plane, have the same orientation. • P0 P1 P2 P3 P4

55. 2D Symmetrization – cont’d
Given a 2D configuration of connected points and
Given a matching between the points of the configuration we find a configuration which satisfy:
have the same topology as
satisfy the mirror-symmetry constraint
The Symmetry Distance is minimized

56. Finding the Closest projected Mirror Symmetry
Consider two points and in and an orientation
We want to find two points and such that the segment connecting them is at orientation and the following sum is minimized: • •

57. Finding the Closest projected Mirror Symmetry – cont’d
Suppose we have n 2D points and a matching of these points, we need to find the orientation which minimizes the Symmetry Distance
For a given orientation , we get:
Taking the derivative with respect to and equating to zero:

58. Finding the Closest projected Mirror Symmetry – Examples
Before symmetrization
After symmetrization b. a. a. b.

59. The Reconstruction Process
The enhancement algorithm described above is independent of the reconstruction method
It should be applied only in reconstruction of bilaterally symmetrical objects
Bilateral symmetry can be determined using 2D Symmetry Distance
If the Symmetry Distance is small in all projections we may assume that the 3D configuration is symmetric (deviations may occur due to noise)

60. Some Results Three variations on the algorithm were tested
Only 2D symmetrization was applied prior the reconstruction
Only 3D symmetrization was applied following the reconstruction
Symmetrization both prior and following the reconstruction was applied
Reconstruction was performed on synthetic and real objects

61. Some Results – cont’d Original configuration
Reconstruction without enhancement
2D symmetrization only
3D symmetrization only
Both symmetrizations

62. Some Results – cont’d Original configuration
Reconstruction without enhancement
2D symmetrization only
3D symmetrization only
Both symmetrizations

63. Some Results – cont’d Both 3D 2D No Symmetrization 1.976036 3.192995
Error
40.8
4.3
42.5
% improvement

64. References
Fatih Ulupinar, Ramakant Nevatia, Perception of 3-D Surfaces from 2-D Contours.
Fatih Ulupinar, Ramakant Nevatia, Using symmetries for Analysis of Shape from Contour
Hagit Zabrodsky, Daphna Weinstall, Using Bilateral Symmetry To Improve 3D Reconstruction From Image Sequences