Real-time Structured Light Depth Extraction Kurtis Keller & Jeremy Ackerman Department of Computer Science University of North Carolina Chapel Hill, North Carolina, USA EI2000

Divisions of talk  Application and reason for research  Comparisons of various, relevant depth extraction systems (structured light, etc..)  First design with large digital projector  Second design with real time custom projector  Improvements...

High speed structured light depth extraction for Augmented reality surgical procedures

Application: Medical l Virtual / Augmented Reality view from single incision. l This information can be rendered with textures to surgeon’s viewpoint. l Must be real-time, small, able to image shiny/curved surfaces.

Comparison of various, relevant Depth Extraction Systems l Stereo laparoscopes l Traditional viewing of images l Automated depth extraction poor l Structured light depth Extraction l Too slow with standard projector and camera l Can be used on curved surfaces l Laser scan l Too slow for real time use l Miniaturization

Surface Texture and Z depth scanning methods for real-time medical use

Structured light (Binary encoded) Start with coarse projected texture, then increase number of bars for finer resolution

First System

First Design: Structured Light with large digital projector

First System v Not Real time v Utilized off the shelf (modified) components l TI micromirror projector Custom lens to flex scope to laparoscope Custom lens to flex scope to laparoscope v 3D scans in lab and animal cadaver tests v Bulky

First System ( in operation) v Close to Real time (60Hz frame imaging, 5Hz updates) v Custom components l Projector Lens Custom lens to flex scope to laparoscope Custom lens to flex scope to laparoscope v Off the shelf componets l TI Micromirror projector l Laparoscope Pulnics progressive scan camera Pulnics progressive scan camera

Second System (real time)

Second System v Almost Real time (~10 Hz) v Utilized custom components l Displaytech ferro-reflective LCD engine in custom projector l Custom lens to flex scope to laparoscope v Fits in with standard surgical environment l Custom projector snap onto standard medical laparoscope l Light source 2 meters away from patient by fiber light Projector about the size of an orange, fitting to surgeons hand and light weight Projector about the size of an orange, fitting to surgeons hand and light weight l All exposed surfaces surgical steel and autoclaveable.

High-speed Projector v Miniature size (50mm Dia x 75mm long) v Custom components l Displaytech 640x Hz. engine l Custom lens to laparoscope

High-speed Miniature Projector Attached to Laparoscope Attached to Laparoscope Exploded view Exploded view

v Real time v High-speed components l Custom, 180Hz projector l 180+ Hz 512x512 pixel camera l High speed graphics processor

Hardware Improvements l Color additive scanning by synchronized color wheel l Single tube Stereo Laparoscope l Reduction of Specular rejection problems l Increased light level bit depth l Image processing/rejection Color Wheel

FUTURE RESEARCH in HIGH SPEED STRUCTURED LIGHT l Obtain Ferro reflective (or other) displays which are not hampered by crystal migration l If possible, purchase graphics processor and high speed camera from same vendor l Full projection imaging is being performed, not micro internals that we emphasized. l Virtual Teleconferencing l Inspection l Surveillance