For Internal Use Only. © CT T IN EM. All rights reserved. 3D Reconstruction Using Aerial Images A Dense Structure from Motion pipeline Ramakrishna Vedantam CTT IN, Bangalore

For Internal Use Only. © CT T IN EM. All rights reserved. Project Goal Volume Estimation of mine dumps Infrastructure development monitoring Augmented Reality Page 2 3D capture of ground structures using aerial imagery

For Internal Use Only. © CT T IN EM. All rights reserved. 3D from Images : Stereo? Page 3

For Internal Use Only. © CT T IN EM. All rights reserved. Stereo 3D information can be ascertained if an object is visible from two views separated by a baseline This helps us to estimate the depth of the scene Page 4

For Internal Use Only. © CT T IN EM. All rights reserved. Disparity/ Depth Image Page 5 Stereo Input Images Disparity / Depth Image

For Internal Use Only. © CT T IN EM. All rights reserved. Multi View Stereo (MVS) Images from multiple views at short baselines used. Give Better Precision and reduce matching ambiguity Case for Multi View Stereo Disparity baseline, focal length and matching. Page 6 Camera Model Needed !

For Internal Use Only. © CT T IN EM. All rights reserved. Calibration of a Camera Model Internal parameters Focal length, pixel aspect ratio etc External camera parameters Rotation and Translation in global frame of reference Page 7 Calibration: finding the internal parameters of the camera

For Internal Use Only. © CT T IN EM. All rights reserved. STRUCTURE FROM MOTION Page 8

For Internal Use Only. © CT T IN EM. All rights reserved. Structure from Motion (SFM) Finding the complete 3D object model and complete camera parameters from a collection of images taken from various view- points. Involves  Stereo Initialization  Triangulation  Bundle Adjustment. Page 9

For Internal Use Only. © CT T IN EM. All rights reserved. Bundle Adjustment Stereo Initialization: Finding relation between features in two initial scenes. Bundle Adjustment: Iteratively minimizing reprojection error while adding more cameras and views. Page 10 Computationally Expensive ! Initialization is Key

For Internal Use Only. © CT T IN EM. All rights reserved. SFM: Reconstruction Page 11 SFM: 2 imagesSFM: 5 imagesSFM: 20 images Clearly, not suitable for dense reconstruction.

For Internal Use Only. © CT T IN EM. All rights reserved. SFM -> Multi-View Stereo Pipeline SFM Typically involves matching of sparse features and triangulation of those features. Generates Camera Parameters. Multi-View Stereo Patch based “every pixel” methods used to estimate the disparity/ depth for the whole of a scene. Uses Camera Parameters to give dense depth estimates. Page 12 SFM to MVS pipeline gives dense reconstructions !

For Internal Use Only. © CT T IN EM. All rights reserved. Accurate, Dense and Robust MVS  Extract features  Get a sparse set of initial matches  Iteratively expand matches to nearby locations  Use visibility constraints to filter out false matches Page 13

For Internal Use Only. © CT T IN EM. All rights reserved. The Missing Link Page 14 Multi View Stereo SFM Images Where do the Images come from ?

For Internal Use Only. © CT T IN EM. All rights reserved. LOCALIZING THE CAMERA Page 15

For Internal Use Only. © CT T IN EM. All rights reserved. PTAM: Parallel Tracking and Mapping Page 16 Stereo Initialization Tracking Mapping PTAM: Key frame selection

For Internal Use Only. © CT T IN EM. All rights reserved. PTAM Tracking and mapping are done in parallel allowing more features to be added to map as they are detected. Bundle Adjustment is done after every few frames. Enforces a pose change and time heuristic to select key frames. Page 17

For Internal Use Only. © CT T IN EM. All rights reserved. KeyFrames Page 18

For Internal Use Only. © CT T IN EM. All rights reserved. PTAM – Pose Page 19

For Internal Use Only. © CT T IN EM. All rights reserved. PTAM -> SFM -> MVS Block Results Page 20 CUP_60 dataset

For Internal Use Only. © CT T IN EM. All rights reserved. PTAM -> SFM -> MVS Block Results Page 21 Olympic Coke CAN

For Internal Use Only. © CT T IN EM. All rights reserved. PTAM -> SFM -> MVS Block Results Page 22 Olympic Coke CAN + Pen

For Internal Use Only. © CT T IN EM. All rights reserved. System Block Diagram – So Far Page 23 Multi View Stereo SFM Keyframes PTAM Bundler PMVS-2 3 stage dense reconstruction pipeline

For Internal Use Only. © CT T IN EM. All rights reserved. Volume Estimation 3D reconstructions stored as point clouds, a set of points in space with color information. From a point cloud, planar features are segmented out. Remaining points are clustered. User views clusters and gives the reference ground truth data and the cluster whose volume is to be estimated. Page 24

For Internal Use Only. © CT T IN EM. All rights reserved. Segmentation and Filtering Page 25

For Internal Use Only. © CT T IN EM. All rights reserved. Volume Estimation After segmenting the point cloud, the volume is estimated by finding the convex hull of the 3-D point cloud. Page 26

For Internal Use Only. © CT T IN EM. All rights reserved. Volume Estimation Page 27 Original Point cloud Clusters

For Internal Use Only. © CT T IN EM. All rights reserved. Volume Estimation - Dataset Ground Truth data : 16.2 cm distance between pens Height of Cylinder : 12.9 cm Radius of Cylinder : 2.9 cm Volume of Cylinder : Page 28

For Internal Use Only. © CT T IN EM. All rights reserved. Volume Estimation - Dataset Volume for PTAM dataset: cu cm Image Resolution: 640 x 480 Accuracy : ground truth is 85.4 % of volume Number of Images: 102 Volume for DSLR dataset: cu cm Image Resolution: 1920x1480 Accuracy : ground truth is 81.4 % of volume Number of Images: 30 Page 29

For Internal Use Only. © CT T IN EM. All rights reserved. Volume Accuracy The multi view stereo algorithm gives 98.7% of points 1.25 mm of the reconstruction for reference datasets. Cameras parameters are noisy, affecting volume accuracy. Pose information given by the IMU can improve camera parameters. Clustering done without a-priori shape information, if given, outliers can be filtered out and geometric consistency enforced. Page 30

For Internal Use Only. © CT T IN EM. All rights reserved. Scope for Improvement 1.Use sensor data from IMU to estimate camera pose 2. Make it a real time, live dense reconstruction system 3. Improve accuracy of volume estimation 4. Plan the flight of the UAV doing the reconstruction 5.Making the reconstruction interactive Page 31

For Internal Use Only. © CT T IN EM. All rights reserved. Related work Dense Reconstruction on the fly (TU Graz) :  Real time reconstruction  User interaction with live reconstruction  Successfully adapted to UAV Dense Tracking and Mapping (Imperial College, UK):  Real time dense reconstruction using GPU  Superior Tracking performance, blur resistant Live dense reconstruction from Monocular Camera (IC) :  Real time monocular dense reconstruction  Sparse Tracking Page 32

For Internal Use Only. © CT T IN EM. All rights reserved. THANK YOU ! Page 33