Depth from Structured Light II: Error Analysis Michael H. Rosenthal April 19th, 2000

Outline Review of Structured Light Projection Error in Ideal Model 4-19-00 Outline Review of Structured Light Projection Error in Ideal Model Lighting and Surface Property Errors Stripe Assignment Errors Motion Errors Structured Light in Practice

Structured Light Projection 4-19-00 Structured Light Projection Camera and projector are calibrated Light stripes are projected onto scene Depth for each pixel is measured by intersecting pixel ray with stripe plane

Binary Stripe Encoding 4-19-00 Binary Stripe Encoding Pixels are “labeled” using binary encoding: light = 1, dark = 0 High-order bits encode broad regions, low-order bits encode fine location

4-19-00 Error in Ideal Model

Error in Ideal Model Pixels and stripes have finite size 4-19-00 Error in Ideal Model Pixels and stripes have finite size All visible surfaces within the pixel/stripe intersection volume are sampled and lumped together

4-19-00 Error in Ideal Model Size of sampled volume depends upon stripe width, pixel width and projector orientation Corollary: depth resolution is controlled by the same factors

Error from Lighting and Reflection 4-19-00 Error from Lighting and Reflection

Lighting and Reflection 4-19-00 Lighting and Reflection Prior assumptions: Bright Lambertian surface Path from projector to surface to camera Real world Dark or shiny materials (apples, metal, cloth) Things in your way Specular highlights interfere with structured light patterns

Lighting and Reflection 4-19-00 Lighting and Reflection Preventive Methods: Capture images with projector fully on and fully off Use difference to measure intensity range for each pixel Threshold difference to find to find trouble spots Images from structured light projected into a simple scene

Lighting and Reflection 4-19-00 Lighting and Reflection Minimum intensity image may have bright spots - specular highlights Maximum intensity image may have dark spots - shadows or dark objects

Lighting and Reflection 4-19-00 Lighting and Reflection Difference image shows range of intensity between on and off Narrow range corresponds to non-ideal behavior Threshold to identify trouble spots Exclude trouble spots from final depth image

Lighting and Reflection 4-19-00 Lighting and Reflection Other practical solutions: Lambertian coatings and sprays (talc) High dynamic range cameras Variable exposure times

Error in Stripe Assignment 4-19-00 Error in Stripe Assignment

Stripe Assignment Errors 4-19-00 Stripe Assignment Errors Stripe encoding requires binary decision for each pixel What happens if a pixel is misassigned to the wrong stripe (i.e. gets a 0 rather than a 1 when illuminated)? Images from structured light projected into a simple scene

Stripe Assignment Errors 4-19-00 Stripe Assignment Errors Caused by random noise, low contrast, motion, edge uncertainty, etc. Magnitude depends upon significance of bit! 1st bit yields error N/2, 2nd bit yields error N/4, kth bit yields error N/2k Binary stripe encoding (phase shifted method)

Stripe Assignment Errors 4-19-00 Stripe Assignment Errors Assume that probability of misassignment is p Expected error from kth bit is p*N/2k Total expected error E = p*N/2 + p * N/4 +… + p * N/2k + … + p*N/2logN E = p*N*(1- 1/2logN) For large N, E ~ p*N For < 1% expected error, we must have < 1% chance of misassignment

4-19-00 Error from Motion

Moving object in a structured light system 4-19-00 Motion Errors What errors can motion cause in a structured light system? Two components of motion: Along a pixel ray Pixel to pixel Moving object in a structured light system

Moving object in a structured light system 4-19-00 Pixel to Pixel Motion Similar to stripe assignment error - “random” bits are flipped in affected pixels Moving object in a structured light system

Motion along a pixel ray traverses projected stripes over time 4-19-00 Motion on a Pixel Ray Motion along a pixel ray causes different stripes to be reflected over time Final pixel code is composed of bits from different positions Motion along a pixel ray traverses projected stripes over time

4-19-00 Motion on a Pixel Ray Magnitude of error depends upon initial position and velocity Very large errors near edges in broad stripes (high order bits)

4-19-00 Motion on a Pixel Ray Absolute error relative to final object position - note the chaotic dependence upon position and velocity

Structured Light in Practice 4-19-00 Structured Light in Practice Office of the Future uses it for surface measurement Augmented Reality Biopsy project has tested it for skin surface measurement and for internal surgery Videos!

Conclusions Four major classes of error in structured light: 4-19-00 Conclusions Four major classes of error in structured light: Error intrinsic to ideal system - affects spatial resolution Lighting and reflection error - some regions of the scene will be unmeasurable Stripe assignment error - need reliable methods for binary categorization Motion error - yields large, unpredictable errors Despite these problems, structured light is useful and widely available