1. 3D Stereo Reconstruction using iPhone Devices Final Presentation 24/12/2012 1 Performed By: Ron Slossberg Omer Shaked Supervised By: Aaron Wetzler

2. Project’s Goal Building a self-contained mobile 3D reconstruction system using two iPhone devices 2

3. Background – Pinhole Camera Model The basic camera model Transformation from 3D to 2D coordinates: Distortion also taken into account 3

4. Background – Stereo Vision Combine images from two cameras to generate depth image Relative cameras’ positions in physical space (R, T) and image planes space (F) retrieved by stereo calibration process 4

5. Background – Stereo Correspondence OpenCV offers a number of algorithms for stereo correspondence We chose to use two algorithms that offer a good compromise between efficiency and quality: – Block Matching – Semi Global Block Matching 5

6. Background – Stereo Correspondence (Cont.) Block Matching: – Looks at blocks of pixels along the epipolar lines and finds matches according to cross correlation. – Example of typical result: 6

7. Background – Stereo Correspondence (Cont.) Semi Global Block Matching: – Adds on to the normal block matching algorithm by introducing global consistency constraints – The constraints are introduced by aggregating matching costs along several independent, one- dimensional paths across the image – Example of typical result: 7

8. Background – Reconstructed Scene The matching algorithm produces a disparity map which is a gray scale image where every color corresponds to a certain disparity and thus a certain depth Using a reprojection matrix on the disparity map we obtain a point in space corresponding to each pixel We can render these points as a 3d mesh using the original picture colors for each vertex. 8

9. Programming Environment iPhone app programming – Objective-C programming – Model-View-Controller design pattern Main implemented features: – Displaying and controlling the views – Inter-device communication and time synchronization – Persistent storage of data 9

10. Programming Environment (cont.) OpenCV libraries – C++ open-source code – Implement all the required algorithms for performing the calibration and reconstruction processes – Handle the interface with the iPhone’s camera Main implemented features: – Integrating openCV functions into our iPhone app – Correct data flow into and out of every openCV function 10

11. Programming Environment (cont.) OpenGL ES libraries – A lightweight version of the open-source OpenGL libraries, which includes an iOS API – Implement the framework for rendering 2D and 3D computer graphics Main implemented features: – Displaying the reconstructed images as an interactive 3D surface 11

12. Software High-Level Design Generating a Bluetooth session between the devices 12 Main MenuConnect Devices

13. Software High-Level Design Setting the right parameters for stereo calibration and reconstruction 13 Main MenuSettings

14. Software High-Level Design Performing stereo calibration for the devices using a chessboard pattern 14 Main MenuCalibration

15. Software High-Level Design Performing 3D stereo reconstruction of the images captured by the devices 15 Main MenuReconstruction

16. Software High-Level Design Interactive 3D color display of the images Disparity map images shown within table 16 Main MenuPhoto Album3D Image Display

17. 17

18. Calibration Process Flow 18 Bluetooth session created Calibration Capture Send Capture Indication Capture Image Extract Corners Send Capture Order Exchange image corners data Initial State Message delay calculated Validating message delay Initializing camera Wait Message Delay

19. Calibration Process Flow (Cont.) 19 Calibrate Save Parameters Compute Intrinsic Parameters Performed separately at each device Compute Extrinsic Stereo Parameters Process was separated to increase accuracy

20. Reconstruction Process Flow 20 Bluetooth Session Created Reconstruction Initial State Load parameters Reconstruction Load parameters Message delay calculated Calibration Performed Compute undistorted rectified bitmap Validating message delay Initializing camera Choose Reconstruction Algorithm Compute undistorted rectified bitmap Choose Reconstruction Algorithm

21. Reconstruction Process Flow (Cont.) 21 Save Disparity Image Compute Disparity by stereo correspondence Capture Send Capture Indication Capture Image Send Capture Order Send and Receive Image Send Images Wait Message Delay Capture Image Remap Images to get Rectified Images Send and Receive Image Compute Disparity by stereo correspondence Save Disparity Image

22. Implementation Issues Simultaneous Photo Capture – Problem: need devices to capture images at the same time to gain good results – Solution: implemented RTT calculation algorithm that ignores anomalous results and performs update phases during operation – Other Solutions: Web Service, GPS – Advantage: messages traverse only short distance, no dependency on GPS signals – Main Disadvantage: GPS achieves better accuracy 22

23. Implementation Issues (Cont.) Inter-Device Communication – Problem: need to pass messages and data between the two devices – Solution: Bluetooth communication – Other Solutions: Wi-Fi – Advantages: existing easy-to-use framework, simple protocol with low overhead – Main Disadvantage: smaller BW (affects the duration of the reconstruction process) 23

24. Summary Very challenging and enjoyable project Introduction with both computer vision and mobile app development Final outcome is a stable, user-friendly app providing live results Helpful documentation for future usage – Detailed webpage and demo movie Finally – many thanks to Aaron for his guidance and support throughout this project ! 24

25. References Computer Vision course, Spring 2010, University of Illinois – http://www.cs.illinois.edu/~dhoiem/courses/vision_spring10/lectures/ http://www.cs.illinois.edu/~dhoiem/courses/vision_spring10/lectures/ Developing Apps for iOS, Paul Hegarty, Stanford fall 2010 course (available on iTunes) Multiple View Geometry in Computer Vision course, University of North Carolina – http://www.cs.unc.edu/~marc/mvg/slides.html http://www.cs.unc.edu/~marc/mvg/slides.html Modeling the Pinhole camera, course lecture, University of Central Florida Computer Vision tutorial, GIP lab Learning OpenCV, Gary Bradski & Adrian Kaehler Stereo Vision using the OpenCV library by Sebastian Droppelmann, Moos Hueting, Sander Latour and Martijn van der Veen 25

26. The End 26