1. Coded Structured Light Coded Structured Light
Course
Stereoscopy vs. Structured Light
Principle & Survey
Coded Structured Light
Course
Stereoscopy vs. Structured Light
Principle & Survey
(No comment)

2. II/TAP · 3D Perception
The camera behaviour?
Camera Modelling: Identify the set of parameters which determines the geometry and optical characteristics of an approximated camera.
Camera Calibration: Compute the value of the parameters for a given sensor.
retinal
image Z r
co-ordinate
co-ordinate Zi
system (mm.)
system (pixels) Y O r i Yi Xi
Image Centre or
Principal Point X r
Focal distance O r (u
, v )
3D point P
Focal point P f d
World Y
It is also widely known, that the intrinsic parameters model the behaviour of the internal geometry of the camera. Starting from the ideal projection of a 3D point on the image plane, going through lens distortion, till expressing the imaged point from metric co-ordinates to pixels. c
co-ordinate system P X Z u Y w c
Image plane w X c
Retinal plane
Observed
projection Z
Ideal
projection w
Camera O c O
co-ordinate system w K

3. ? Obtaining 3D information?
II/TAP · 3D Perception
Obtaining 3D information? Y M w
3D point
From m, only the optical ray can be computed.
We need more information to determine M. X ? w
optical ray Z w Z C f
Now, let’s spend a bit of time talking about how 3D information can be obtained by computer vision, that is how we can obtain the 3D position of the point M by using calibrated cameras.
Given a single camera we can only infer the orientation of the optical ray which starts from the captured point m and passes through the 3D object point. I m O Y X
Captured Image

4. component of the second point is needed.
II/TAP · 3D Perception
Obtaining 3D information?
We need at least two cameras.
A 3D object point has three unknown co-ordinates.
Each 2D image point gives two equations.
3D point Y M w X w Z w Z’ Z C’ C f’ f m’ Y’
Only a single
component of the second point is needed.
Then, we need at least two cameras in order to cross both rays. In fact as each imaged point has two components, then it gives two equations. So, we need at least to know one of both components of the second point, in order to have the three equations to obtained the 3D point position.
This simplification is quite important in coded structured light. I m O’ O Y I’ X
Captured Image X’
Captured Image

5. 3D Reconstruction is obtained !
II/TAP · 3D Perception
3D Reconstruction is obtained !
Once the correspondence problem is solved, the couples of points are used to compute the position of the 3D object points. Here, we can see a Top view of the scene, and a lateral view from the side of the cylinder.
Note that only the crossing points of the pattern are used to reconstruct the scene. In this case no points have been projected on the edge of the square, that’s why we have obtained two vertexes of the square in the top view.
Top View
Lateral View

6. The Correspondence Problem
II/TAP · 3D Perception
The Correspondence Problem
Given an image point m on an image plane,we have to search for its correspondence point m’ on the other image plane to obtain the co-ordinates of the 3D object point M .
Problems:
Surface occlusions, vanishing points, camera’s scope.
Consequences:
Points without homologues.
Points with multiple homologues.
Solutions:
Matching from the epipolar geometry.
Matching from disparity.
Matching from similarity.
...
However, Stereo Vision has the main problem of the correspondence, that is given an imaged point we have to surely know which is its correspondence point on the other image plane. This problem becomes harder if we take into account that we can capture points without match due to a surface occlusion or due to a projection out of the camera scope, or even we can couple points with a false match due to vanishing features.
There are some principles which can be applied in order to reduce this problem. The most important are based on the epipolar geometry and the disparity gradient (there also exists techniques base on similarity as colour or texture analysis, and on the use of invariants).
In order to explain in more detail the correspondece problem, I will briefly describe the epipolar geometry.

7. The searching space is reduced to a single line.
II/TAP · 3D Perception
The Epipolar Geometry
The searching space is reduced to a single line.
3D point
Epipole
Epipole
2D point
2D point
Epipolar
line
The epipolar geometry defines the relation given by two cameras. Note that, the projection of the focal point of one camera defines an epipole on the other camera. Then, given a 2D image point on an image plane (for instance m) we can affirm that its correspondence point m’ must lie on the epipolar line defined by the intersection of the other image plane I’ and the plane defined by the three points C, C’ and m; that is the plane P. Then, we can reduce the two dimensional search of the correspondence point from the two dimensional space of the image plane to the one dimensional space of the epipolar line.
Epipolar
line
Image plane of the
first camera
Image plane of the
second camera

8. The Epipolar Geometry II/TAP · 3D Perception Area 2 Epipole Area 1
Epipolar lines
Epipolar lines
Correspondence
points
Zoom
Area 2
Zoom
Area 1
In order to test the equations, we have simulated an example based on a stereoscopic system which images 20 3D object points. We have computed the epipolar geometry between both camera models. Note the results obtained: the two epipoles and the epipolar lines. Note also that each point lies on the epipolar line defined by its correpondence points.
Epipolar geometry of Camera 1
Epipolar geometry of Camera 2

9. The Epipolar Geometry: Matching Problems
II/TAP · 3D Perception
The Epipolar Geometry: Matching Problems
Direct order
Scene Geometry:
Unknown Order of Projections.
Points with multiple homologues.
Surface occlusions & Out of camera scope:
Points without homologue.
Inverse order
However, it’s still hard to completely solved the correspondence problem.
Note that points can be imaged in an unknown order along the epipolar line as shown in picture a) and b).
And note also that some 3D points may have a single projection as they could be imaged on a single camera due to surface occlusions as shows picture c), or simply due to a projection outside the camera scope.
These points must be removed from the image plane as they can not be used to infer 3D information but, sometimes it is a hard work. For instance, in the example we can not only used the epipolar line in order to obtained the right association of the q point.
Surface occlusion
Point without homologue

10. II/TAP · 3D Perception
Structured Light
Principle : Based on the relation between a single camera and a light source which projects a known pattern on the measuring scene.
Only the points illuminated by the light source are captured by the camera.
Advantages:
Easy segmentation. Depending on the light source.
Correspondence problem reduced. Depending on the pattern shape.
Able to reconstruct continuous surfaces.
Constraints:
Light dependent.
The problem of points without match can be solved using the concept of structured light. Then, one of both cameras is replaced by a pattern projection system.

11. Structured Light Techniques:
II/TAP · 3D Perception
Structured Light Techniques:
Single dot :
No correspondence problem.
Scanning both axis.
Single slit :
Correspondence problem among points of the same slit => orthogonal translation of the light projector with respect to the slit.
Scanning the axis orthogonal to the slit.
Stripe patterns :
Correspondence problem among slits.
No scanning.
Grid, multiple dots :
Correspondence problem among all the imaged segments.
Initially, structured light was based on the projection of regular patterns, as dots, slits and grids. However, structured light does not solve the problem of multiple matching. Since all the projected points have the same characteristics, we can not identify one from the others.
This problem can be solved if the pattern of light is coded. This principle is the known as coded structured light.

12. Coded Structured Light
II/TAP · 3D Perception
Coded Structured Light
Principle : Based on an “unique” codification of each token of light projected on the scene. When the token is imaged by the camera this decodification permits to know where it comes from and solve the correspondence problem.
In fact, the correspondence problem can be completely solved codifying the projected pattern in such a way that each projected point carries a label indicating from where it comes. Then, analysing the points captured by the camera we can find up from which position on the projector frame it has been projected.
However, we have to take into account the way of coding the pattern to be projected with respect to the scenes that will be analysed.

13. Pattern Projector Frame
II/TAP · 3D Perception
Pattern
Crossing
Points
Image
Crossing
Points
Camera Image Plane
Pattern Projector Frame
Image
segmentation
Searching the
correspondence point
3D Reconstruction
This example is based on the measurement of a human palm. We can see the hand illuminated by the pattern, and the obtained 2D points.

14. Coded Structured Light Classification
II/TAP · 3D Perception
Coded Structured Light Classification
Codification Dependence
Single Axe
Row-coded patterns
(m’y known, m’x unknown)
Column-coded patterns
(m’x known, m’y unknown)
Both Axis
Static
Projection of a set of patterns.
Static scenes.
Dynamic
Projection of single pattern.
Moving scenes.
Temporal Dependence
Depth Dependence
Periodical:
The codification of the tokens
is repeated periodically .
Absolute:
Each token is uniquely encoded.
Projected Light Dependence
The survey done point us to propose the following classifications:
Firstly, a pattern can be classified related to the number of co-ordinates that are coded in the pattern. Then the pattern are classified into single axe or both axe if it is a single co-ordinate or both co-ordinates of the projected point that are coded.
Secondly, the pattern can be classified related to its capability to measure dynamic scenes such as those based on mobile navigation, where only one pattern can be projected, or static scenes, where the sequential projection of several patterns is allowed.
Thirdly, it can be classified related to its capability to measure scenes made by highly saturated colour objects, where basically only binary patterns can be projected; or scenes with a colour content predominantly neutral, where the emission of colour is permitted.
And finally, the capability to measure scenes with a lot of discontinuities of several different depths, where only an absolute codification of the whole pattern is allowed; or scenes predominantly continuous, where very simple periodical patterns can be projected.
Now, I’ll present you the overview, the patterns analysed will be explained in a non chronological order, going from simpler ones to more complicated.
Binary
Grey Levels
Colour

15. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Le Moigne - Waxman 1984
Pattern placed with single vertical displacement from camera.
Vertical slits used as guides to search the dots.
Dots are used to decode the horizontal lines.
Both axis coded / Dynamic / Binary / Absolute
Posdamer - Altschuler 1981
The projection of n patterns are needed to code 2n columns.
Projection of an initial pattern without codification to extract dot positions.
Column Coded / Static / Binary / Absolute
Minou ‘81, Inokuchi ‘84. The same pattern proposal.
Sato ‘86. Better segmentation, positive & negative images.
Mundy ‘87. Sensor implemented in hardware.
Hattori ‘95. Low heat irradiance. Semiconductor Laser.
Muller ‘95. Re-implementation.
For example, in 1984 Le Moigne proposed to add dots on a regular grid. The pattern projector is placed with a single vertical displacement related the camera, then, vertical slits are imaged without deformation and are used as guides to search the dots. Dots are used to decode the horizontal lines which gives the 3D information from their deformation. This pattern is especially indicated to work in dynamic environments as a single patterm is projected.
In 1981 Posdamer and Altschuler proposed the widely known temporal codification of a stripped pattern. Then, the projection of n patterns are needed to codify 2n(raised) columns. The idea proposed by Posdamer has been widely studied. In this case the system can measure only static scenes.
Le Moigne - Waxman 1984
Grid projection with weak codification from dots. Dots are used as landmarks to initiate the matching.
Projector located at a fixed distance along the y axe (vertical displacement).
Posdamer - Altschuler 1981
Projection of n patterns to coded 2n columns.
Projection of an initial pattern without dot codification to extract dot positions.

16. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Carrihill - Hummel 1985
Two shots are needed:
First, constant illuminated pattern.
Second, the linear wedge grey pattern.
Column decodification from image substraction.
Column Coded / Static / Grey / Absolute
Maruyama - Abe 1993
Multiple vertical slits with random cuts.
Matching from slit end points by using epipolar geometry.
Column Coded / Dynamic / Binary / Absolute
The system proposed by Carrihill in 1985 is based on obtaining 3D information from two projections. The first image is captured with a constant illumination projection on the scene. The second one is captured with the wedge pattern shown in the picture. The difference between both grey levels allows us to obtain the 3D information of the object points.
In 1993, Maruyama proposed the projection of multiple vertical slits with random cuts. The matching is obtained from the slit end points by using epipolar geometry. The system is specially recommended in dynamic environments as a single projection is needed. However, it can not be projected in high discontinuous or textured surfaces due to the increase of end points which will difficult the matching.
Carrihill - Hummel 1985
Linear wedge grey pattern.
Two images needed. First whole illuminated.
Column coded from grey level substraction.
Maruyama - Abe 1993
Multiple vertical slits coded by random cuts.
Matching from slit end points along epipolar lines.
Improve : Intelligent distribution of the cuts in order to reduce false matching.

17. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Morita - Yakima - Sakata 1988
Initial projection of a whole illuminated dot matrix to extract dot position.
Window coded pattern.
Column Coded / Static / Binary / Absolute
Lavoie ‘96. A grid pattern with random binary dots in the cross-points.
Vuylsteke - Oosterlinck 1990
Chess-board pattern projection with coded squares.
Window coded pattern.
Column Coded / Dynamic / Binary / Absolute
Pajdla ‘95 : Re-implementation.
Ito ‘95 : A three grey level checkerboard pattern.
In 1988, Morita proposed the interested codification of the pattern by windows. As in the Posdamer’s pattern, an initial whole illuminated matrix is projected to know the position of the captured dots on the image plane. Then, a coded dot matrix is projected. Each window with a fixed size represents a binary number which gives us information of the position of the same pixel on the projector frame.
The same idea of dot codification was used by Lavoie in In this case random binary dots were placed in the vertexes of a grid pattern.
Another window coded pattern was proposed by Vuylsteke in In this case a chess-board pattern is projected. A codification bit is located at each chess-board dot. The association of 6 coded bits gives the column index needed to obtain 3D information. In 1995 Pajdla re-implemented the same pattern proposed by Vuylsteke, and in the same year Ito proposed another dot codification from a three grey level checkerboard pattern.
Morita - Yakima - Sakata 1988
Binary dot matrix projection.
Initial projection of a whole illuminated matrix.
Window coded.
Vuylsteke - Oosterlinck 1990
A single regular chess-board pattern projection.
Every square has a codification bit.
Window size of 2 x 3 bits Þ 64 columns.

18. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Boyer - Kak 1987
Multiple coloured vertical slits.
Codification from slit colour sequence.
Column Coded / Dynamic / Colour / Absolute
Monks ‘93 : Utilisation of the same pattern for speech interpretation.
Chen ‘97 : Unique codification and colour improvement.
Tajima - Iwakawa 1990
Projection of a rainbow pattern.
Monochromatic cameras: Two images were captured using two different camera filters.
Column Coded / Static / Colour / Absolute
Smutny ‘96 : Re-implementation.
Geng ‘96 : Single RGB grabber.
In 1987, Boyer and Kak proposed to illuminate the scene by the single projection of a stripped pattern. The pattern is made by a sequence of vertical slits in red, green and blue. The pattern is split in sub-patterns where each sub-pattern exists only once. In 1992, Monks has used the same coloured pattern for speech interpretation.
Three years later, Tajima proposed the projection of a rainbow pattern. Initially, Tajima used monochromatic cameras, as a result of which two images were captured using different image filters. Recently, Smutny and Geng have re-implemented the pattern proposed by Tajima using a single RGB camera.
Boyer - Kak 1987
Multiple coloured slits.
Codification from slit colour sequence.
Tajima - Iwakawa 1990
Hue wedge pattern of the whole colour spectrum.
Two monochromatic images grabbed with different colour filter.

19. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Boyer - Kak Slit based projection
Single axis codification.
Column coded by groups of 4 slits.
Problem of matching.
Improve:
Both axis codification.
Project some additional information in order to identify if two slits are neighbours or not. Þ Grid pattern projection determine the neighbourhood by slit continuity.

20. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Wust - Capson 1991
Periodical pattern made by shifting and merging red, green and blue sinusoidal intensity patterns.
Column Coded / Dynamic / Colour / Periodical
Hung ‘93. Single grey level sinusoidal pattern.
Griffin - Narasimhan - Yee 1992
Mathematical study to obtain the largest codification matrix from a fixed number of colours.
Dot position coded by the colour of its four neighbours.
Both axis coded / Static / Colour / Absolute
Davies ‘96 : Re-implementation.
In 1991, Wust and Capson proposed the projection of a periodical coloured pattern made by the merging of three sinusoidal intensity patterns. The pattern is made by merging a Red sinus, with a Green and Blue sinus, each one shifted 90º. In this case the pattern is constraint to measure surface without high depth discontinuities.
Finally in 1992, Griffin carried out a mathematical study about which should be the largest size allowed for a coded matrix dot pattern. A dot position is coded with the colour emitted by itself and the colour of its four neighbours. In the whole matrix there may not be two different dot positions with the same code.
Wust - Capson 1991
Projection of a coloured pattern made from Red, Green and Blue shifted sinusoidal intensity patterns.
Griffin - Narasimhan - Yee 1992
Mathematical study to obtain the largest codification matrix from a fixed basis.
Dot codification from its four neighbours.

21. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Griffin et al Dot matrix projection
Both axis codification.
Dot position coded by the colour of itself and its four neighbours.
(C E N W S)
Horizontal and vertical neighbours coded by using the same colours.
Depending on the scene, sometimes its difficult to determine the neighbours.
Improve: C E N S W
Connect the dots with slits in order identify the neighbours uniquely.
Horizontal and Vertical axis coded independently Þ Projector placed without positional constraint.

22. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
Posdamer ‘81
Moigne ‘84
Carrihill ‘ 85
Boyer ‘87
Morita ‘88
Vuylsteke ‘90
Tajima ‘ 90
Wust ‘ 91
Griffin ‘ 92
Maruyama ‘93
Summary
Single Axis
Codification
Both Axis
Static
Temporal
Dynamic
Binary
Light
This summary shows the described techniques in a chronological order.
Most of the proposing techniques are based on single axe codification, this is due to the fact that a single co-ordinate of each projecting point is needed in order to infer 3D information. However, if both co-ordinates can be coded, then the results might be more accurate.
Considering the temporal dependence, at the beginning most of the patterns were static. However, as a result of the technology and the research done they have become more dynamic, allowing it to measure static and moving scenes.
The same idea can be used in order to explain the way why the patterns has progress from binary to coloured ones. Obviously, the main reason is the simplification of the pattern codification, as more information can be projected with coloured patterns than binary ones. However, coloured patterns might not be used to measure high saturated colour surfaces.
Considering depth dependence, nearly all the patterns are based on an absolute codification which allows the pattern to be used without depth constraints.
Grey
Colour
Periodical
Depth
Absolute

23. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
What is it required?
How can it be done?
Easy Segmented
Projection of a Grid
Measurement of
Dynamic Scenes
Pattern coded in a
single projection
Accurate Results
Both axes codification
Without positional
constraint with respect
to the camera
Vertical and Horizontal
slits coded
independently
Use of
colour
Each slit with its two
neighbours exits only
once in the pattern
Without depth
dependence

24. Coded Structured Light Techniques
II/TAP · 3D Perception
Coded Structured Light Techniques
As an example, we have used:
p = 3 P = {1,2,3}
Slit colour sequence:
Horizontal 1 = red, 2 = green, 3 = blue.
Vertical = magenta, 2 = cyan, 3 = yellow.
Colours well-spaced in the chromatic space.
Maximum 29 slits with p = 3 Þ Grid of 29 x 29 slits.
Slits are constantly spaced in a 512 x 512 image.
Resolution increase :
· Increasing the p basis.
· Reducing the space between slits.

25. A way of obtaining a sequence of uniquely encoded slits:
II/TAP · 3D Perception
A way of obtaining a sequence of uniquely encoded slits:
Given P={1,2,...,p} set of colours.
We want to determine S={s1,s2,...sn} sequence of coloured slits.
Node: {ijk} Î
Number of nodes: p3 nodes.
Transition {ijk} Ù {rst} Þ j = r, k = s
Example: p = 2
Path: (111),(112),(122),(222),(221),(212),(121),(211).
Slit colour sequence:111,2,2,2,1,2,1,1 Þ Maximum 10 slits.
111
112
121
212
122
211
221
222
In this slide we can see an example of path finding and the sequence of numbers that are obtained. Note that each number with its two neighbours exists only once. We have implemented the Griffin algorithm, and this algorithm has been used using a p basis equal two three obtaining a sequence up to 29 numbers. .

26. System Testing II/TAP · 3D Perception Camera Image Plane
3D Reconstruction
In this slide we can see an example of path finding and the sequence of numbers that are obtained. Note that each number with its two neighbours exists only once. We have implemented the Griffin algorithm, and this algorithm has been used using a p basis equal two three obtaining a sequence up to 29 numbers. .
Pattern Projector Frame
Smooth surface Reconstruction

27. System Testing

28. Accuracy Results Average discrepancy values Xw: 0.728 mm. (0.31 %)
II/TAP · 3D Perception
Accuracy Results
Real Scene
Reconstructed Scene w P 1 c O Y Z X d 2 u ˆ
Average discrepancy values
Xw: mm. (0.31 %)
Yw: mm. (0.27 %)
Zw: mm. (0.20 %)
In this slide we can see an example of path finding and the sequence of numbers that are obtained. Note that each number with its two neighbours exists only once. We have implemented the Griffin algorithm, and this algorithm has been used using a p basis equal two three obtaining a sequence up to 29 numbers. .
Maximum discrepancy values
Xw: mm. (0.50 %)
Yw: mm. (0.62 %)
Zw: mm. (0.29 %)

29. II/TAP · 3D Perception
Conclusions
Structured light has been presented as a technique which permits to reduce the correspondence problem. However, structured light can not solve the problem of multiple matching.
Coded structured light has been presented as a technique to solve completely the matching.
Proposal of new coded structured light challenges.
The pattern coded with respect to both axis which permits to obtain more accurate results.
The pattern is coded by a single projection which enables to obtain 3D information with a single shot, allowing the system to measure moving scenes.
Vertical and horizontal slits are coded independently which permits the projector to be placed without positional constraints with respect to the camera.