Capstone Design Implementation of Depth Sensor Based on Structured Infrared Patterns June 11, 2013 School of Information and Communication Engineering, Inha University Song Myung Ho

Capstone Design 2 Calculation of depth using Structured Infrared Patterns Introduction Operation Principle Capture 2 pictures for standard distance Match Images and Induce a disparity map Calculate Depth from disparity map Used Hardware Information Camera module Board Test : Image capture to Calculation depth Test environment Object height Result of disparity Contents

Capstone Design 3 Introduction Depth information In these days, interest on virtual reality and need of Human- machine interaction grow bigger. Depth information is used both of them. And also using other areas. More detail, Depth information is used to make stereopsis, Unmanned Vehicle, U- health, and so on. Application I want to implement Depth measuring system to inform machine about object length inexpensively and quickly. Especially, assembling machine of facility automation would need assembled object’s 3D size.

Capstone Design 4 Principle [Source : ‘How the kinect work’, Derek Hoiem, University of illinois] [Source : US B2, ‘Depth mapping using projected patterns’] Projector : emit infrared light on random speckle pattern Sensor : this is camera. Capture patter image

Capstone Design 5 Principle IR projector Pattern : Random Speckle ( Unique pattern ) [Source : Kinect Hacking 103: Looking at Kinect IR Patterns] Disparity : Numbers of Shifted Pixel

Capstone Design 6 Calculate Depth Depth : dx : disparity d : distance between an (ref.) object and a camera F : focal length Alpha : approximately L’/L, where L’ is distance between camera and projector, L is distance between camera and object [Source : US B2, ‘Depth mapping using projected patterns’] Matching : Search Equal Dot Pattern and Record Disparity. Block matching algorithm : SAD (Sum of Absolute Differences) Block matching algorithm is faster other matching algorithm. [ref. : E. G. Richardson, Iain (2003). H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia. Chichester: John Wiley & Sons Ltd..]

Capstone Design 7 Calculate Disparity map Ref. Image Cmp. Image 1. Get two windows 2. Sum each pixel of Absolute level difference on two windows 3. Move the window of compared image by 1 4. Repeat step 2 ( limit : setting range in same row) 5. We know equal pattern by minimum value of summation 6. Record the shifted range of same pattern 7. Move the window of reference image by 1 pixel. Repeat step 1~6. ( if all pixel have disparity value, then all is done. [Image Source : Effilux]

Capstone Design Camera Module 8 -. Aptina MT9M111 CMOS [Source : Aptina] -. Sensor : Aptina 1/3” CMOS Sensor MT9M active pixels : 1280 * 1024 pixels -. pixel size : 3.6um * 3.6um -. focal length : 16mm -. Data interface : 8-bit parallel -. Control interface : I2C -. Maximum input clock : 54Mhz Hardware Information RDB1768, CodeRed : using MCU – LPC ARM cortex M3 ( 32bit ) - Operation Clock : 100Mhz - Internal RAM : 64KB ( and no external memory ) - Flash : 512KB - Using peripheral : UART / I2C - GPIO Operation Clock : 25Mhz

Capstone Design 9 Test Environment 1 st object ( height : 52mm) 2 nd object (height : 17mm) Arrange to ceiling

Capstone Design 10 Test Environment Install Camera and IR projector (Base distance : 110mm, Focal length : 16mm ) Object and Sensors ( distance to ceiling : 1944mm)

Capstone Design 11 Result of images Reference Image Comparing Image Real disparity map Normalized disparity map visibly

Capstone Design 12 Calculation : depth Disparity : image size – output 128x 80 pixel / x4 zoom ( vision range : 256 x 160 pixels ) => pixel size : 7.2um x 7.2um Base distance : 110mm Focal length : 16mm Distance between Sensor chip to ceiling : 1944 mm Distance between Sensor chip to lense : 34mm 1 st object depth : 52mm 2 nd object depth : 17mm dx : camera pixel size * disparity pixels = disparity distance d : distance between an object and a camera F : focal length Alpha : approximately L’/L, where L’ is distance between camera and projector, L is distance between camera and object

Capstone Design 13 Calculation : depth Disparity : (24-5)/16 (5-5)/16 (63-5)/16 = = 0 = dx = * number of shifted d = 1944 F = 16 Alpha = 110/1944 = …. (rad) tangent Alpha = dZ = *Real Disparity dZ : (ref.) = 18 = 0 = 56 (mm) Error : (18-17)/17 (56-52)/52 = = = 5.88% = 7.69% ↑ Depth resolution

Capstone Design 14 Calculation : depth 1.height : 8.3mm 2.height : 16.6mm 3.height : 24.9mm 4.height : 33.2mm

Capstone Design 15 Result : depth accuracy Calculate real disparity as floating point, accuracy is 90.69%, but this is not real depth because real disparity is integer

Capstone Design 16 Result : depth accuracy This is real depth. Error is far bigger.

Capstone Design 17 Calculation : depth 1.height : 16.6mm 2.height : 32.45mm 4.height : 45.5mm 3.height : 66.4mm

Capstone Design 18 Result : depth accuracy Depth resolution is about , Stair 1,2,3 are nearby multiple of depth resolution but stair4. error rate of stair 1,2,3 is small but stair 4 is big. So, we know depth resolution is the most important factor of accuracy. ( we can obtain detail resolution using more distant focal length and smaller pixel size )

Capstone Design 19 References US Patent : Method and system for object reconstruction(2010) US Patent : Depth mapping using projected patterns(2012) Derek Hoiem(2012), How the kinect works Stefano Mattoccia(2012), Stereo Vision : Algorithms and applications Gary Rost Bradski(2009), Learning OpenCV Jihong Liu, Chengyuan Wang (2009), An algorithm for image binarization based on adaptive threshold

Capstone Design 20 Contact Information H.P : Thanks for listening