Depth estimation and Plane detection for Image Analysis and Understanding Cheng Hanni Huang Jingjing Song Mengting Xiao Ying

Project Description Depth estimation is the set of techniques and algorithms aiming to obtain the distance of each 3D point to the image plane. The goal of plane detection is to identify pixels in a 2D image which locates on a 2D plane. Depth estimation and Plane detection are for 3D reconstruction, scene understanding and robotics etc.

Overview Literature investigation Website establishment Depth estimation Plane Detection Conclusion

Literature investigation Learning Depth from Single Monocular Images In Proc. of NIPS.2005 Depth Map Prediction from Single Image using Multi Scale Deep Network. in Proc. of NIPS, 2014. Accurate 3D Ground Plane Estimation from a Single Image, In Proc. of ICRA 2009 Detecting planes and estimating their orientation from a single image, In Proc. of BMVC 2011.

Website Establishment Django is a free and open-source web framework, written in Python, which follows the model-view-template (MVT) architectural pattern. Our templates include demo.html and show_pic.html The core code of depth-estimation and plane-detection are under the folder ‘homework (app)’,we call the two functions in views.py. If I run the django project on my computer, then the url “http://localhost:8000/demo/” can leads to our demo.

Depth estimation Methods：Multi-Scale Deep Network Global Coarse-Scale Network: contains five feature extraction layers of convolution and max-pooling,followed by two fully connected layers.to predict the overall depth map structure using a global view of the scene.

Depth estimation Methods：Multi-Scale Deep Network Local Fine-Scale Network:The fine-scale network stack consists of convolutional layers only, along with one pooling stage for the first layer edge features. To edit the coarse prediction it receives to align with local detais such as object ande wall edges.

Depth estimation Scale-Invariant Error

Plane Detection Methods SLIC Algorithm to over-segment a picture (TPAMI 2012) For each 2D point in the picture, add the information of depth map to reconstruct a 3D point. Get the adjacent-matrix according to the segmented clusters. Go through the adjacent matrix using BFS, judge if two adjacent matrix are coplanar.(dimensionality reduction using SVD) Compute cosine of each plane

Plane Detection Experiment Results a) Source image b) Depth map c) Plane with avg color(ICRA 2009) d) Plane with box (demo video)

Plane Detection Analysis Solution The method in ICRA 2009 divide a picture into several irregular parts, draw a bounding box according to the top-left and bottom-right point of a part can’t get the precise result. The method depends on many parameters,(sample number, cosine threshold), one certain parameter cannot fit all picture. Solution Train a number of pictures to choose the right parameters which can make the training error smallest. Extract other features (texture/gradient) from one picture to get better result. But time is limited, I… don’t try above methods.()

Conclusion We use Django Framework written in python to establish our website, so that our front-end page can connect with our picture-processing codes. CNN can be applied to depth estimation. We write a python script to get the output depth map from the pre-trained caffe model. Plane detection includes three steps. Reconstruct 3D points using depth information Over-segment the picture using SLIC algorithm Treat the coplanar parts of the segmented clusters as a plane.

Links http://10.13.49.144:8000/demo/ (Not always available) Demo: Project website: http://itec.hust.edu.cn/~chenghn/project/Depth_Estimation/index.html Source code: https://github.com/honey0920/depth-estimation-plane-detection Course website: http://lbmedia.ece.ucsb.edu/member/uweb/Teaching/website/index.htm

Thanks.