1. www-video.eecs.berkeley.edu/research
Introduction to Structured Light (SL) Systems and SL Based Phase Unwrapping
R. Garcia & A. Zakhor
EECS Department
UC Berkeley
www-video.eecs.berkeley.edu/research

2. Outline Background Structured Light Basics Phase Unwrapping
Our Algorithms
Results
Summary

3. Determining Depth of a Scene
Many applications:
Biomedical
Industrial
Entertainment
Navigation
Methods for Determining Depth
Triangulation
Time of flight
Depth from (de)focus
Other…

4. Triangulation Based Depth Methods
Scene observed from multiple views
Correspondences between views solved
Must know intrinsic and extrinsic parameters for each view
Two categories
Passive: Stereo – multiple cameras
Active: Structured Light – camera with projector or other light source used I J
Projector
Cam I J
Cam
[S. Narasimhan]

5. Traditional Stereo Need 2 or more views of the scene
Scene texture used to identify correspondences across views
Dense stereo remains an active research area in computer vision
Algorithms and new Middlebury vision
Given rectified images, triangulation is simple
Using similar triangles:
[Mirmehdi]

6. Potential Problem with Stereo
Stereo works well on “textured” images

7. Potential Problem with Stereo
Stereo can fail with lack of texture ?

8. Structured Light (SL) SL places “texture” onto the scene
Projector patterns identify each scene region
Classes of patterns
Temporal
Spatial
Other
Survey of patterns [J. Salvi et al., 2004]

9. Temporal Coding
Multiple frames are projected to identify scene regions
Camera pixel’s intensity change used for correspondence
Scene assumed to be static

10. Spatial Coding Encodes unique information into small regions.
Fewer captures Less dense reconstruction

11. Spatial Coding
Recognize this?

12. Other Coding Methods Spacetime Coding Viewpoint Coding
Zhang et al., “Spacetime Stereo: Shape Recovery for Dynamic Scenes”, 2003
Davis et al., “Spacetime Stereo: A Unifying Framework for Depth from Triangulation”, 2003
Viewpoint Coding
Young et al., “Viewpoint-Coded Structured Light”, 2007

13. Structured Light v. Stereo
Advantages of SL:
SL does not require a richly textured scene
Solving correspondences is not expensive
Each scene point scene receives a unique code
Advantages of Stereo:
No interference with observed scene
Only need acquisitions at one time for capture; SL often needs multiple images
Too much texture can be problematic for SL.

14. Outline Background Structured Light Basics Phase Unwrapping
Our Algorithms
Results
Summary

15. Structured Light Geometry
Geometry of SL is simple
Triangulation performed by ray-plane intersection
Intrinsic and extrinsic parameters needed
Calibration matrix for camera and projector
Transformation between coordinate frame of camera and projector

16. Triangulation
Intrinsic and extrinsic parameters
Left Cam
Projector

17. Triangulation
Left Cam
Projector

18. SL for Dynamic Scene Capture
SL capable of capturing dynamic scenes
Want to limit capture time when capturing dynamic scenes
“One-shot” approaches require single capture
Trade-off
Fewer frames -> lower capture resolution
More frames -> more sensitive to scene motion

19. Overview of Phase Shifted Structured Light (SL) Systems
Project patterns to find correspondences between camera and projector
Phase shifted sinusoidal patterns:
Fast capture: 3 shots
Simple to decode
Insensitive to blur
Used in optical metrology
M periods

20. Example: Scene Illuminated with Phase-Shifted Sinusoids
Wrapped
Phase I2 I3
Unwrapped
Phase
M periods in sinusoid 
Need to unwrap phase

21. Outline Background Structured Light Basics Phase Unwrapping
Our Algorithms
Results
Summary

22. What is Phase Unwrapping?
Phase values usually expressed from [-π , π)
Would like to recover original continuous phase measurement

23. Overview of Phase Unwrapping
Unwrapping assumptions:
Single continuous object in scene
Slowly varying depth; discontinuities less than |π|
2D phase unwrapping results in relative phase:
Need absolute phase for triangulation.
Wrapped
Phase
Unwrapped
Phase

24. Creating Point Cloud

25. Stereo-Assisted Phase Unwrapping
Stereo assisted phase unwrapping [Wiese 2007]:
Results in absolute phase
Deals with depth discontinuities
System Setup: Two cameras, single projector
Cam 1 2
Projector A B C D

26. Overview of Phase Unwrapping with two Cameras [Weise et al. 2007]
To resolve the absolute phase for a camera pixel:
Determine wrapped phase for all pixels in first and second camera
Project a ray from pixel in camera 1 with wrapped phase
Project M planes from the projector corresponding to in space
Find the M intersections of the M planes with the ray in (1) in 3D space,
Project onto camera 2
Compare the M phase values at M pixel
locations in camera 2 to the and
choose the closest

27. Stereo Phase Unwrapping [Weise et. al. 2007]
Right
Cam
Left
Cam
Projector

28. Drawbacks of Stereo Phase Unwrapping
Must be run twice.
Possible to incorrectly assign absolute phase values to corresponding points between views. P A B C D
Left Camera
Right Camera

29. Comparison of Stereo Assisted Phase Unwrapping with Merging Stereo and 3D (x,y,t)

30. Temporal Inconsistencies
Consecutive phase images highly correlated
Correlated information not used during phase unwrapping
Results in inconsistent unwrapping

31. 3D Phase Unwrapping Multi-dimensional phase unwrapping
2D: traditional image processing
3D: medical imaging (i.e. MRI)

32. Overview of 3D (x,y,t) Phase Unwrapping
Treat consecutive phase images as volume of phase values
Edges defined between neighboring pixels
Quality assigned to each pixel Inversely proportional to spatio-temporal second derivative
Want to unwrap according to quality of edges

33. 3D Phase Unwrapping (cont.)
Evaluate edges in a 3D volume from highest quality to lowest
Evaluating edge = unwrapping pixels connected to it
Unwrapped pixels connected in chains
Grow the chain by adding pixels/edges in x, y, and t
Chains merged together

34. Outline Background Structured Light Basics Phase Unwrapping
Our Algorithms
Results
Summary

35. Consistent Stereo-Assisted Phase-Unwrapping Methods for Structured Light Systems

36. Merging Stereo & 3D (x,y,t) Phase Unwrapping
Goal: Solve for absolute phase offset of a chain by using stereo matching data.
Approach: use quality to find phase offset probabilistically
M periods in the sinusoidal pattern  M possible phase offsets
Find offset probability for each pixel in the chain, then combine them to find phase offset for the whole chain
How to find phase offset for a pixel P:
Use phase difference between wrapped phase at P and its corresponding M projected pixels to generate likelihood of each of the M phase offset values.

37. Computing Phase Offset for an Entire Chain
Combine offset probabilities for each pixel in the chain to determine phase offset for the whole chain
(a)
(b) +
π-2δ
π-δ
P(k1=0)
P(k1=1) :
P(k1=M-1)
P(k2=0)
P(k2=1)
P(k2=M-1)
-π+δ
+ 0π
+ 2π
Pixels in the same period Pixels in different periods

38. Comparison with 3D (x,y,t) only
Proposed
3D (x,y,t)
Proposed algorithm avoids
unwrapping low quality pixels by
using stereo
Handling multiple disjoint objects

39. Comparison with Stereo Only
Proposed
Consecutive unwrapped phase frames and their difference

40. Comparison of Stereo Assisted Phase Unwrapping with Merging Stereo and 3D (x,y,t)

41. Stereo Phase Unwrapping [Weise et. al. 2007]
Right
Cam
Left
Cam
Projector

42. Proposed Method
Perform unwrapping w.r.t. projector pixels rather than camera pixels
Right
Cam
Left
Cam
Projector

43. Overview of the proposed Method
For each projector pixel with phase , find the corresponding epipolar line in each image.
For each camera, find all points along the epipolar line with phase
Find the 3D point in space resulting from intersection of rays resulting from these points with the plane for the projector:
N1 points in 3D space for the left camera 1  Ai
N2 points in 3D space for the right camera 2  Bi
Find the corresponding pair of Ai and Bi  closest in 3D space
Assign global phase to corresponding pixels of the “best” pair of the two cameras
Camera 1
Camera 2

44. Projector Domain Stereo Method
Right
Cam
Left
Cam
Projector
Top down view of the projector plane corresponding
to the projector column with absolute phase theta

45. Finding corresponding “pair” of points in 3D from the two cameras
Compute pairwise distance for all pairs of points in the two views.
The distance between the 3D locations of corresponding points is small.
Compute possible correspondences for each projector pixel.
Find the correct correspondence labeling for each projector pixel.
Use loopy belief propagation (LBP)
Distance B1 B2 A1 A2 A3
Possible Labels B1 B2 A1
{A1,B1}
{A1,B2} A2
{A2,B1}
{A2,B2} A3
{A3,B1}
{A3,B2}

46. Loopy Belief Propagation Cost Function
Minimize cost function:
where:
Locations of pixels in Cam A & B
Labeling for projector image
3D location of image pixel
Set of projector pixels
2D location of image pixel
Projector pixel
4- connected pixel neighborhood
3D distance cost threshold
2D distance cost threshold

47. Local Phase Unwrapping for Remaining Pixels
Camera Points
Computed
via LBP
Occluded pixels:
Corresponding pair are too far apart in 3D space
Use quality based local unwrapping
Unwrapping order for remaining pixels depends on:
Density of stereo unwrapped points
Local derivatives
Merge pixel density and derivative maps to generate quality map
Unwrap from highest to lowest quality.
Density
Local
Derivative
Merged

48. Results Proposed method results in
Right Camera
Proposed
Proposed method results in
consistent phase results across cameras
Left Camera
Proposed
In a 700 frame sequence, our method:
Has same accuracy in 80% of frames
Has better than or equal accuracy in 96% of frames.
Left Camera
[Weise et. al.]

49. Results: Captured Dynamic Scene

50. Dynamic Point Cloud

51. Advantages of Projector Domain Unwrapping
Only scene points illuminated by the projector can be reconstructed
Unwrapping only needs to be performed once for any # of cameras more consistent and efficient.
Computational complexity scales with projector resolution rather than image resolution.

52. Conclusion Provided introduction to the structured light systems
Presented two phase unwrapping algorithms:
A three-dimensional stereo-assisted phase unwrapping method
A projector-centric stereo-assisted unwrapping method
Results in accurate, consistent phase maps across both views.
Results in accurate 3D point clouds