Pattern Projection Techniques Computer Vision and Robotics University of G iro na.

Slides:



Advertisements
Similar presentations
Shapelets Correlated with Surface Normals Produce Surfaces Peter Kovesi School of Computer Science & Software Engineering The University of Western Australia.
Advertisements

QR Code Recognition Based On Image Processing
電腦視覺 Computer and Robot Vision I
High-Resolution Three- Dimensional Sensing of Fast Deforming Objects Philip Fong Florian Buron Stanford University This work supported by:
Structured Light principles Figure from M. Levoy, Stanford Computer Graphics Lab.
www-video.eecs.berkeley.edu/research
--- some recent progress Bo Fu University of Kentucky.
(thanks: slides Prof. S. Narasimhan, CMU, Marc Pollefeys, UNC)
Prénom Nom Document Analysis: Document Image Processing Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.
Computer Vision and Robotics Group
Gratuitous Picture US Naval Artillery Rangefinder from World War I (1918)!!
Stereo Many slides adapted from Steve Seitz. Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image Where does the.
Image Segmentation Image segmentation (segmentace obrazu) –division or separation of the image into segments (connected regions) of similar properties.
« Uncalibrated Vision based on Structured Light » Joaquim Salvi 2 David Fofi 1 El Mustapha Mouaddib 3 3 CREA EA 3299 Université de Picardie Jules Verne.
Stereo.
December 5, 2013Computer Vision Lecture 20: Hidden Markov Models/Depth 1 Stereo Vision Due to the limited resolution of images, increasing the baseline.
Active Calibration of Cameras: Theory and Implementation Anup Basu Sung Huh CPSC 643 Individual Presentation II March 4 th,
Last Time Pinhole camera model, projection
CS6670: Computer Vision Noah Snavely Lecture 17: Stereo
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2005 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Stereoscopic Light Stripe Scanning: Interference Rejection, Error Minimization and Calibration By: Geoffrey Taylor Lindsay Kleeman Presented by: Ali Agha.
Multiple Human Objects Tracking in Crowded Scenes Yao-Te Tsai, Huang-Chia Shih, and Chung-Lin Huang Dept. of EE, NTHU International Conference on Pattern.
Introduction to Computer Vision 3D Vision Topic 9 Stereo Vision (I) CMPSCI 591A/691A CMPSCI 570/670.
The plan for today Camera matrix
3D from multiple views : Rendering and Image Processing Alexei Efros …with a lot of slides stolen from Steve Seitz and Jianbo Shi.
Presented by: Ali Agha March 02, Outline Sterevision overview Motivation & Contribution Structured light & method overview Related work Disparity.
CSE473/573 – Stereo Correspondence
Introduction of the intrinsic image. Intrinsic Images The method of Finlayson & Hordley ( 2001 ) Two assumptions 1. the camera ’ s sensors are sufficiently.
3-D Computer Vision Using Structured Light Prepared by Burak Borhan.
Announcements PS3 Due Thursday PS4 Available today, due 4/17. Quiz 2 4/24.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Computer Vision Spring ,-685 Instructor: S. Narasimhan WH 5409 T-R 10:30am – 11:50am Lecture #15.
Introduction --Classification Shape ContourRegion Structural Syntactic Graph Tree Model-driven Data-driven Perimeter Compactness Eccentricity.
Camera Calibration & Stereo Reconstruction Jinxiang Chai.
1 Intelligent Robotics Research Centre (IRRC) Department of Electrical and Computer Systems Engineering Monash University, Australia Visual Perception.
Image Formation. Input - Digital Images Intensity Images – encoding of light intensity Range Images – encoding of shape and distance They are both a 2-D.
Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T Computer Vision Tahun: 2010.
Visual Perception PhD Program in Information Technologies Description: Obtention of 3D Information. Study of the problem of triangulation, camera calibration.
Recap from Monday Image Warping – Coordinate transforms – Linear transforms expressed in matrix form – Inverse transforms useful when synthesizing images.
December 4, 2014Computer Vision Lecture 22: Depth 1 Stereo Vision Comparing the similar triangles PMC l and p l LC l, we get: Similarly, for PNC r and.
Digital Image Processing CCS331 Relationships of Pixel 1.
Stereo Readings Szeliski, Chapter 11 (through 11.5) Single image stereogram, by Niklas EenNiklas Een.
Stereo Many slides adapted from Steve Seitz.
Project 2 code & artifact due Friday Midterm out tomorrow (check your ), due next Fri Announcements TexPoint fonts used in EMF. Read the TexPoint.
Stereo Many slides adapted from Steve Seitz. Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image image 1image.
Computer Vision, Robert Pless
Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.
Introduction --Classification Shape ContourRegion Structural Syntactic Graph Tree Model-driven Data-driven Perimeter Compactness Eccentricity.
Computer Vision Lecture #10 Hossam Abdelmunim 1 & Aly A. Farag 2 1 Computer & Systems Engineering Department, Ain Shams University, Cairo, Egypt 2 Electerical.
Edge Detection and Geometric Primitive Extraction Jinxiang Chai.
Lecture 16: Stereo CS4670 / 5670: Computer Vision Noah Snavely Single image stereogram, by Niklas EenNiklas Een.
Autonomous Robots Vision © Manfred Huber 2014.
CSE 185 Introduction to Computer Vision Feature Matching.
Robust and Accurate Surface Measurement Using Structured Light IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 57, NO. 6, JUNE 2008 Rongqian.
Lecture 9 Feature Extraction and Motion Estimation Slides by: Michael Black Clark F. Olson Jean Ponce.
Project 2 due today Project 3 out today Announcements TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA.
John Morris Stereo Vision (continued) Iolanthe returns to the Waitemata Harbour.
Machine Vision Edge Detection Techniques ENT 273 Lecture 6 Hema C.R.
Robotics Chapter 6 – Machine Vision Dr. Amit Goradia.
Instructor: Mircea Nicolescu Lecture 10 CS 485 / 685 Computer Vision.
Detection of nerves in Ultrasound Images using edge detection techniques NIRANJAN TALLAPALLY.
Stereo CS4670 / 5670: Computer Vision Noah Snavely Single image stereogram, by Niklas EenNiklas Een.
Coded Structured Light Coded Structured Light
José Manuel Iñesta José Martínez Sotoca Mateo Buendía
CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.
Common Classification Tasks
Range Imaging Through Triangulation
Chapter 11: Stereopsis Stereopsis: Fusing the pictures taken by two cameras and exploiting the difference (or disparity) between them to obtain the depth.
Stereo vision Many slides adapted from Steve Seitz.
Presentation transcript:

Pattern Projection Techniques Computer Vision and Robotics University of G iro na

2 Presentation outline 1.- A Survey of Pattern Projection Techniques –Introduction and some classifications –Pattern Projection Techniques –Experimental results –Conclusions & guidelines 2.- An Optimised one-shot technique –Optimal pattern –A new coding strategy –Implementation design –Experimental results –Conclusions

3 Formació Presentation outline 1.- A Survey of Pattern Projection Techniques –Introduction and some classifications –Pattern Projection Techniques Time Multiplexing Spatial Codification Direct Codification –Experimental results –Conclusions & guidelines Conclusions Experiments Classification Introduction

4 Conclusions Experiments Classification Introduction Shape acquisition Encoding/ decoding The challenge of Computer Vision INTERPRETATION KNOWLEDGE BASE DATA BASE Interpretation Level SCENE DESCRIPTION FUSIONTRACKING SHAPE IDENTIFICATION LOCALIZATION SCENE ANALYSIS SEGMENTATION SHAPE ACQUISITION FEATURE EXTRACTION MOVEMENT DETECTION TEXTURE ANALYSIS IMAGE RESTORATION EDGE RESTORATION Description Level Image Processing High Level FILTERING EDGE THINNING EDGE DETECTION COLOUR COMPENSATION THRESHOLDING GRADIENTS RE-HISTOGRAMATION A/D COLOUR SEPARATION SENSOR Image Acquisition Level Image Processing Low Level

5 Experiments Classification Introduction Shape acquisition Encoding/ decoding Shape Acquisition Techniques Shape acquisition techniques Contact Non-destructiveDestructive CMMJointed arms Slicing Non-contact Reflective Transmissive Non-optical Optical... Microwave radarSonar Passive Active StereoShadingSilhouettesTextureMotion Shape from X Interferometry (Coded) Structured light MoireHolography Source: Brian Curless Conclusions

6 Sub-pixel matching Experiments Classification Introduction Shape acquisition Stereovision Encoding/ decoding Introduction: passive stereovision Correspondence problem geometric constraints  search along epipolar lines 3D reconstruction of matched pairs by triangulation

7 Conclusions Sub-pixel matching Experiments Classification Introduction Shape acquisition Stereovision Encoding/ decoding Introduction: passive stereovision Arrange correspondence points Ordered Projections Projections bad order Occlusions Points without homologue Inverse order Direct order Surface occlusionPoint without homologue

8 Conclusions Sub-pixel matching Experiments Classification Introduction Shape acquisition Stereovision Encoding/ decoding Introduction: passive stereovision Y Z X I M f m I’ m’ X’ Y’ Z’ f’ O O’ C C’ Z w X w Y w Captured Image We need at least two cameras. A 3D object point has three unknown co-ordinates. Each 2D image point gives two equations. Only a single component of the second point is needed.

9 Conclusions Sub-pixel matching Experiments Classification Introduction Shape acquisition Stereovision Encoding/ decoding Introduction: active stereo (coded structured light) One of the cameras is replaced by a light emitter Correspondence problem is solved by searching the pattern in the camera image (pattern decoding)

10 Experiments Classification Introduction Shape acquisition Encoding/ decoding Coded Structured Light The correspondence problem is reduced : The matching between the projected pattern and the captured one can be uniquely solved codifying the pattern. – Stripe patterns : No scanning. Correspondence problem among slits. – Grid, multiple dots : No scanning. Correspondence problem among all the imaged features (points, dots, segments,…). – Single dot : No correspondence problem. Scanning both axis. – Single slit : No correspondence problem. Scanning the axis orthogonal to the slit. Conclusions

11 Experiments Classification Introduction Shape acquisition Encoding/ decoding Pattern encoding/decoding (I) A pattern is encoded when after projecting it onto a surface, a set of regions of the observed projection can be easily matched with the original pattern. Example: pattern with two-encoded-columns The process of matching an image region with its corresponding pattern region is known as pattern decoding  similar to searching correspondences Decoding a projected pattern allows a large set of correspondences to be easily found thanks to the a priori knowledge of the light pattern Pixels in red and yellow are directly matched with the pattern columns Codification using colors Object scene Conclusions

12 Experiments Classification Introduction Shape acquisition Encoding/ decoding Pattern encoding/decoding (II) Two ways of encoding the correspondences: single and double axis codification  it determines how the triangulation is calculated Decoding the pattern means locating points in the camera image whose corresponding point in the projector pattern is a priori known Double-axis encoding Triangulation by line-to-plane intersection Triangulation by line-to-line intersection Single-axis encoding Conclusions Encoded Axis

13 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Coded structured light patterns: a classification proposal AXIS CODIFICATION Single Axis - Row-coded patterns - Column-coded patterns Both Axis Static Scenes - Projection of a set of patterns. Moving Scenes - Projection of a unique pattern. SCENE APPLICABILITY CODING STRATEGY Periodical - The codification of the tokens is repeated periodically. Absolute - Each token is uniquely encoded Binary Grey Levels Colour PIXEL DEPTH Conclusions

14 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Coded structured light patterns: a classification proposal Conclusions

15 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing The time-multiplexing paradigm consists of projecting a series of light patterns so that every encoded point is identified with the sequence of intensities that receives The most common structure of the patterns is a sequence of stripes increasing its length by the time  single-axis encoding Advantages: –high resolution  a lot of 3D points –High accuracy (order of  m) –Robustness against colorful objects (using binary patterns) Drawbacks: –Static objects only –Large number of patterns  High computing time Pattern 1 Pattern 2 Pattern 3 Projected over time Example: 3 binary- encoded patterns which allows the measuring surface to be divided in 8 sub-regions Conclusions

16 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing: Binary codes (I) Pattern 1 Pattern 2 Pattern 3 Projected over time Every encoded point is identified by the sequence of intensities that receives n patterns must be projected in order to encode 2 n stripes Example: 7 binary patterns proposed by Posdamer & Altschuler … Codeword of this píxel:  identifies the corresponding pattern stripe Conclusions

17 Formació Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing: binary codes (II) Coding redundancy: every edge between adjacent stripes can be decoded by the sequence at its left or at its right Conclusions

18 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing: n-ary codes (I) Using a 4-ary code, 3 patterns are used to encode 64 stripes (Horn & Kiryati) n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours)  increases the basis of the code The number of patterns, the number of grey levels or colours and the number of encoded stripes are strongly related  fixing two of these parameters the reamaining one is obtained Using a binary code, 6 patterns are necessary to encode 64 stripes Conclusions

19 Conclusions Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing: n-ary codes (II) n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours)

20 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing: Gray code + Phase shifting (I) Gühring’s line-shift technique A sequence of binary patterns (Gray encoded) are projected in order to divide the object in regions An additional periodical pattern is projected The periodical pattern is projected several times by shifting it in one direction in order to increase the resolution of the system  similar to a laser scanner Example: three binary patterns divide the object in 8 regions Without the binary patterns we would not be able to distinguish among all the projected slits Every slit always falls in the same region Conclusions

21 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification A periodical pattern is shifted and projected several times in order to increase the resolution of the measurements The Gray encoded patterns permit to differentiate among all the periods of the shifted pattern Time-multiplexing: Gray code + Phase shifting (II) Conclusions

22 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Formació Time-multiplexing: hybrid methods (I) In order to decode an illuminated point it is necessary to observe not only the sequence of intensities received by such a point but also the intensities of few (normally 2) adjacent points The number of projected patterns reduces thanks to the spatial information that is taken into account The redundancy on the binary codification is eliminated Hall-Holt and Rusinkiewicz technique: 4 patterns with 111 binary stripes Edges encoding: 4x2 bits (every adjacent stripe is a bit) Pattern 1 Pattern 2 Pattern 3 Pattern 4 Edge codeword: Conclusions

23 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Time-multiplexing: hybrid methods (II) Conclusions

24 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial Codification Spatial codification paradigm encodes a set of points with the information contained in a neighborhood (called window) around them The codification is condensed in a unique pattern instead of multiplexing it along time The size of the neighborhood (window size) is proportional to the number of encoded points and inversely proportional to the number of used colors The aim of these techniques is to obtain a one-shot measurement system  moving objects can be measured Advantages: ― Moving objects supported ― Possibility to condense the codification into a unique pattern Drawbacks: ― Discontinuities on the object surface can produce erroneous window decoding (occlusions problem) ― The higher the number of used colours, the more difficult to correctly identify them when measuring non- neutral coloured surfaces Conclusions

25 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: non-formal codification (I) The first group of techniques that appeared used codification schemes with no mathematical formulation. Drawbacks: - the codification is not optimal and often produces ambiguities since different regions of the pattern are identical - the structure of the pattern is too complex for a good image processing Maruyama and Abe  complex structure based on slits containing random cuts Durdle et al.  periodic pattern Conclusions

26 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: non-formal codification (II) Conclusions

27 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: De Bruijn sequences (I) A De Bruijn sequence (or pseudorrandom sequence) of order m over an alphabet of n symbols is a circular string of length n m that contains every substring of length m exactly once (in this case the windows are unidimensional) m=4 (window size) n=2 (alphabet symbols) Formulation: Given P={1,2,...,p} set of colours. We want to determine S={s 1,s 2,...s n } sequence of coloured slits. Node: {ijk}  Number of nodes: p 3 nodes. Transition {ijk} Ù {rst}  j = r, k = s The problem is reduced to obtain the path which visits all the nodes of the graph only once (a simple variation of the Salesman’s problem). –Backtracking based solution. –Deterministic and optimally solved by Griffin. m = 3 (window size) n = p (alphabet symbols) Conclusions

28 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: De Bruijn sequences (II) Example: p = 2 Path: (111),(112),(122),(222),(221),(212),(121),(211). Slit colour sequence:111,2,2,2,1,2,1,1  Maximum 10 slits. ‘1’  Red ‘2’  Green m = 3 n = 2 Conclusions

29 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: De Bruijn sequences (III) The De Bruijn sequences are used to define coloured slit patterns (single axis codification) or grid patterns (double axis codification) In order to decode a certain slit it is only necessary to identify one of the windows in which it belongs to Zhang et al.: 125 slits encoded with a De Bruijn sequence of 5 colors and window size of 3 slits Salvi et al.: grid of 29  29 where a De Bruijn sequence of 3 colors and window size of 3 slits is used to encode the vertical and horizontal slits Conclusions

30 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: De Bruijn sequences (IV) Conclusions

31 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: M-arrays (I) An m-array is the bidimensional extension of a De Bruijn sequence. Every window of w  h units appears only once. The window size is related with the size of the m-array and the number of symbols used Morano et al. M-array represented with an array of coloured dots Example: binary m- array of size 4  6 and window size of 2  2 M-array proposed by Vuylsteke et al. Represented with shape primitives Conclusions

32 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Spatial codification: M-arrays (II) Conclusions

33 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Direct Codification Every encoded pixel is identified by its own intensity/colour Since the codification is usually condensed in a unique pattern, the spectrum of intensities/colours used is very large Additional reference patterns must be projected in order to differentiate among all the projected intensities/colours: Ambient lighting (black pattern) Full illuminated (white pattern) … Advantages: –Reduced number of patterns –High resolution can be teorically achieved (all points are coded) Drawbacks: –Very noisy in front of reflective properties of the objects, non-linearities in the camera spectral response and projector spectrum  non-standard light emitters are required in order to project single wave-lengths –Low accuracy (order of 1 mm) Conclusions

34 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Direct codification: grey levels (I) Every encoded point of the pattern is identified by its intensity level Carrihill and Hummel Intensity Ratio Sensor: fade from black to white Requirements to obtain high resolution Every slit is identified by its own intensity Every slit must be projected using a single wave-length Cameras with large depth-per-pixel (about 11 bits) must be used in order to differentiate all the projected intensities Conclusions

35 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Direct codification: grey levels (II) Conclusions

36 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Direct codification: Colour (I) Every encoded point of the pattern is identified by its colour Tajima and Iwakawa rainbow pattern (the rainbow is generated with a source of white light passing through a crystal prism) T. Sato patterns capable of cancelling the object colour by projecting three shifted patterns (it can be implemented with an LCD projector if few colours are projected  drawback: the pattern becomes periodic in order to maintain a good resolution) Conclusions

37 Experiments Introduction Classification Time- multiplexing Spatial codification Direct codification Direct codification: Colour (II) Conclusions

38 Introduction Experiments Experimental results Time- multiplexing Posdamer (128 stripes)Horn (64 stripes)Gühring (113 slits) Spatial codification De Bruijn (64 slits)Salvi (29x29 slits)Morano (45x45 dot array) Direct codification Sato (64 slits) GühringDe Bruijn Classification Conclusions

39 Experiments Introduction Conclusions Guidelines Classification Conclusions Types of techniques  Time-multiplexing Highest resolution High accuracy Easy implementation Large number of patterns Only motionless objects Spatial codification A unique pattern is required Can measure moving objects Lower resolution than time- multiplexing More complex decoding stage Occlusions problem Direct codification High resolution Few patterns Very sensitive to image noise  cameras with large depth-per-pixel required Sensitive to limited bandwith of LCD projectors  special projector devices are usually required Only motionless objects

40 Experiments Introduction Classification Guidelines RequirementsBest technique High accuracy Highest resolution Static objects No matter the number of patterns Phase shift + Gray code  Gühring’s line-shift technique High accuracy High resolution Static objects Minimum number of patterns N-ary pattern  Horn & Kiryati Caspi et al. High accuracy Good resolution Moving objects Optimised Bruijn pattern  Salvi and Pagés. Conclusions Guidelines

41 Conclusions Experiments Optimisation Introduction Presentation outline 2.- An Optimised one-shot technique –Typical one-shot patterns –Optimal pattern –A new coding strategy –Implementation design –Experimental results –Conclusions design Coding strategy One-shot patterns

42 Conclusions Experiments Optimisation Introduction design Coding strategy One-shot patterns Typical one-shot patterns Stripe patternGrid pattern De Bruijn codification M-array codification Array of dots Array of shape primitives Multi-slit pattern Checkerboard pattern Image by Li Zhang

43 De Bruijn patterns Stripe patterns High resolution and good accuracy Multi-slit patterns High accuracy and good resolution Grid patterns Low resolution M-array patterns Array of dots Low resolution and inaccurate sub-pixel localisation of the dots Shape primitives Low accuracy and difficult sub-pixel accuracy Checkerboard Low resolution Conclusions Experiments Optimisation Introduction design Coding strategy One-shot patterns Best one-shot patterns De Bruijn patterns Stripe patterns High resolution and good accuracy Multi-slit patterns High accuracy and good resolution Grid patterns Low resolution M-array patterns Array of dots Low resolution and inaccurate sub-pixel localisation of the dots Shape primitives Low accuracy and difficult sub-pixel localisation of shape primitives Checkerboard Low resolution

44 Conclusions One-shot patterns Experiments Introduction design Coding strategy Optimisation Optimising De Bruijn patterns: interesting features OptimisationAdvantages Maximising resolution Larger number of correspondences in a single shot Maximising accuracy Better quality in the 3D reconstruction Minimising the window size More robustness against discontinuities in the object surface Minimising the number of colours More robustness against non-neutral coloured objects and noise

45 Conclusions Experiments Introduction design Coding strategy Optimisation One-shot patterns Optimisation: maximising resolution Stripe patterns: achieve maximum resolution since they are formed by adjacent bands of pixels. Multi-slit patterns: the maximum resolution is not achieved since black gaps are introduced between the slits Ideal pattern: the one which downsamples the projector resolution so that all the projected pixels are perceived in the camera image  in fact only columns or rows must be identified in order to triangulate 3D points.

46 Conclusions Experiments Introduction design Coding strategy Optimisation One-shot patterns Optimisation: maximising accuracy Ideal pattern: multi-slit patterns allow intensity peaks to be detected with precise sub-pixel accuracy Multi-slit patternStripe pattern Edge detection between stripes: need to find edges in the three RGB channels  the location of the edges does not coincide Figure by Jens Gühring RGB channels intensity profile of a row of the image

47 Conclusions Experiments Introduction design Coding strategy Optimisation One-shot patterns Optimisation: minimising the window size and the number of colours MaximisingMinimising Number of colours Larger resolution Smaller distance in the Hue space  more difficult colour identification Smaller resolution Larger distance in the Hue space  Easier colour identification Window size Larger resolution Danger to violate the local smoothness assumption Smaller resolution More robustness against surface discontinuities If both the number of colours and the window size are minimised the resolution decreases

48 Conclusions One-shot patterns Experiments Introduction design Coding strategy Optimisation Optimising De Bruijn patterns: summary OptimisationBest pattern Maximising resolution Stripe pattern Maximising accuracy Multi-slit pattern Minimising the window size and the number of colours preserving a good resolution Stripe pattern + multi-slit pattern

49 Conclusions Experiments Optimisation Introduction design One-shot patterns A new hybrid pattern Coding strategy intensity Horizontal scanline Combination of the advantages of stripe patterns and multi-slit patterns Stripe pattern in the RGB space Multi-slit pattern in the Luminance channel intensity Horizontal scanline intensity Horizontal scanline Projected luminance profile Perceived luminance profile

50 Conclusions Experiments Optimisation Introduction design One-shot patterns The new coding strategy Coding strategy Given n different values of Hue and a window size of length m A pattern with 2n m stripes is defined with a square Luminance profile with alternating full-illuminated and half- illuminated stripes The pattern is divided in n periods so that all the full-illuminated stripes of every period share the same Hue The half-illuminated stripes of every period are coloured according to a De Bruijn sequence of order m-1 and the same n Hue values Example: n=4, m=3 128 stripes 32 stripes 16 coded stripes … …

51 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Implementation design of the coded structured light technique Colour selection and calibration Colours A M, A m Geometric calibration K 1, P c, P p RGB channel alignment calibration g H r, g H b Peak compensation calibration offsets RGB alignment Image rectification Peak detection Peak decoding Peak compensation Colour rectification 3D reconstruction Offline processes Online processes Radial distortion removal Most of the processes of this schema are valid for any one-shot technique based on colour structured light

52 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Offline processing: RGB channel misalignment calibration The grid cross-points coordinates are located with sub-pixel accuracy in each RGB channel. Two homographies g H r and g H b are calculated in order to reduce the misalignment between the red and the blue channel with respect to the green. RGB image red channel green channel blue channel red grid green grid blue grid It should be a white grid Homographies

53 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Offiline processing: geometric calibration - Camera: Pinhole + Radial lens distortion - Projector: like a reverse camera

54 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Offline processing: colour selection (I) Example: selection of n=3 colours Hue 1) Select n equi-spaced Hue levels 2) Project the n Hue values with different Luminance values and interpolate their response curve. Each RGB channel has different sensitivity  the intensity perceived is in function of the projected Hue emitted Luminance perceived Luminance 3) Given the maximum and minimum levels of luminances that we want to perceive  the corresponding projecting luminances for every Hue can be chosed

55 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Offline processing: colour selection (II) The projector-camera system introduces a strong colour cross-talk which makes difficult the colour identification The cross-talk can be reduced with a calibration procedure: a pure red, green and blue patterns are projected over a colour- neutral panel. a linear mapping is calculated between the colour instructions and the perceived ones  a cross-talk matrix is obtained this matrix approximates the behaviour of the projector-camera system  This matrix will be used online to reduce the cross-talk. The highest cross-talk exists between the green and the red channel

56 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Offline processing: peak compensation calibration Every peak of luminosity is affected by the colours of the surrounding stripes  the detected peak position is affected by an offset Peak compensation calibration: the pattern containing both full and half- illuminated stripes is projected and then the pattern only containing the full-illuminated stripes is also projected Maxima peak sub-pixel position - offsets The same process is made with the minima peaks Maxima peak sub-pixel position

57 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Online processing Colours A M, A m K 1, P c, P p g H r, g H b offsets RGB alignment Image rectification Peak detection Peak decoding Peak compensation Colour rectification 3D reconstruction Online processes Radial distortion removal Stripes location: binarised 2nd derivative of the luminance channel Radial distortion removal RGB alignment Image rectification Peak detection Peak decoding Colour rectification Peak decoding 1  number of period De Bruijn: Position of the stripe inside the period Sub-pixel location of each positive and negative luminance peak: Blais and Rioux detector Colour rectification by using the inverse of the calibrated cross-talk matrix Peak compensation 3D reconstruction

58 Conclusions Optimisation Introduction Experiments design One-shot patterns Coding strategy Experimental results (I)

59 Conclusions Optimisation Introduction Experiments design One-shot patterns Coding strategy Experimental results (II)

60 Conclusions Experiments Optimisation Introduction design One-shot patterns Coding strategy Conclusions A new coding strategy for one-shot patterns has been presented. The new pattern combines the high resolution of stripe- patterns and the high accuracy of multi-slit patterns. A pattern with 2n m stripes is achieved with only n levels of Hue and a window property equal to m. (ex. 2*4 3 = 128 stripes) Given the same parameters, stripe-patterns encoded with classical De Bruijn sequences have a resolution of n m (ex. 4 3 = 64 stripes) The resolution has been doubled. The pattern has a sinusoidal profile in the luminance channel so that maximum and minimum peaks can be accurately detected with classical peak detectors. n = Number of colours (4), m = Window size (3)

61 Published Material Journals –J. Pagés, J. Salvi, J. Forest. Optimised De Bruijn patterns for one-shot shape acquisition. Image and Vision Computing. In revision process. –J. Salvi, J. Pagés, J. Batlle. Pattern Codification Strategies in Structured Light Systems. Pattern Recognition 37(4), pp , April –J. Batlle, E. Mouaddib and J. Salvi. A Survey: Recent Progress in Coded Structured Light as a Technique to Solve the Correspondence Problem. Pattern Recogniton 31(7), pp , July –J. Salvi, J. Batlle and E. Mouaddib. A Robust-Coded Pattern Projection for Dynamic 3D Scene Measurement. Pattern Recognition Letters 19, pp , September Recent Conferences –J. Pagès, J. Salvi and J. Forest. Optimised De Bruijn Patterns for One-Shot Shape Acquisition. IEEE Int. Conf. on Pattern Recognition, ICPR 2004, Cambridge, Great Britain, –J. Pagès, J. Salvi and C. Matabosch. Implementation of a robust coded structured light technique for dynamic 3D measurements. IEEE Int. Conf. on Image Processing, ICIP 2003, Barcelona, Spain, Septembre –J. Pagès, J. Salvi, R. García and C. Matabosch. Overview of coded light projection techniques for automatic 3D profiling. IEEE Int. Conf. on Robotics and Automation, ICRA 2003, Taipei, Taiwan, May More Information :

Computer Vision and Robotics Group Institute of Informatics and Applications THE END