Segmentation Based Environment Modeling Using a Single Image SeungTaek Ryoo Department of Software, HanShin University, Korea stryoo@hs.ac.kr 2004. 9. 29

Previous Works 3D Reconstruction Planar Objects Single View Metrology (Criminisi1999) Piecewise Planar Objects from Single Images (Sturm1999) Automatic 3D Modeling of Architecture (Dick2000) Image Based Modeling and Photo-Editing (Oh2001) Criminis1999 Sturm1999 Oh2001 Dick2000

(Regular Image, OCC Map) Objective Objective Realistic & Real-time View Rendering with Full-View Panorama Image-Segmentation Based Environment Modeling Environment Mapping Desired Image Input Image (Regular Image, OCC Map) Segmented Image Image Segmentation Depth Calculation Depth Image Environment Model Environment Modeling The Structure of Segmentation Based Modeling

Image Segmentation Segmentation indoor scene consists of floor and ceiling. outdoor scene consists of ground and sky. the environment map is divided into a floor (ground), ceiling (sky), and surrounding objects. In general, An indoor scene consists of a floor and ceiling and the outdoor scene consists of ground and sky. From such features, the environment map is divided into a floor (ground), ceiling (sky), and surrounding objects. This figure shows the segmented image using the region based segmentation method.

Depth Calculation (1/4) Assumption View direction of the camera must be parallel to the ground The floor (ground) and the ceiling must consist of a plane that is parallel to the XZ plane in the world coordinate. The sky must consist of a hemi-sphere with an infinite radius The surrounding objects must be perpendicular to the ground Image plane From this segmented image, We can acquire the depth image using the following conditions. View direction of the camera must be parallel to the ground The floor (ground) and the ceiling must consist of a plane that is parallel to the XZ plane in the world coordinate. The sky must consist of a hemi-sphere with an infinite radius The surrounding objects must be perpendicular to the ground o WC m Z R

Depth Calculation (2/4) Extracting the depth value from Regular Image The depth value can be obtained by testing for the intersection points between the floor and the ray, and between the Image plane and the ray The equation can be obtained with the law of the triangle Now, We’ll examine the issue of how to extract the depth value from a regular image This figure shows the world coordinates and image plane, r is the distance between the origin and the image plane And then setting the floor that parallel to the horizontal planes, d is the height value of the floor The depth value can be obtained by testing for the intersection points between the floor and the ray, and between the Image plane and the ray. we can obtain the Z value with the law of the triangle

Depth Calculation (3/4) Extracting the depth value from OCC map depth value of the ceiling (floor) by testing the relationship b/w XZ cylinder and ray, and b/w XY cylinder and ray r b d Z R Y = d XY Cylinder XZ Cylinder Next, We’ll examine how to extract the depth value using the OCC Map, We can also obtain depth value of the ceiling (floor) by testing the relationship between XZ cylinder and ray, and between XY cylinder and ray We can obtain the same equation as before the depth value of the surrounding objects can be obtained by setting the depth value of the floor on the objects surrounding the floor.

Depth Calculation (4/4) How to get the depth value of the surrounding objects Set the depth value of the floor(ceiling) close to the surrounding objects. Some other assumptions should be made. the relationship between the surrounding objects. the relationship between the floor(ceiling) and the surrounding objects. the relationship between the image plane and the surrounding objects. V U o Color Image Segmented Image Depth Image

Environment Modeling Rectangular Grid Triangulation Feature Points 3D Environment model can be made from a rectangular grid modeling method and a triangulation method To make triangulated model, we need a preprocessing step that extracts feature points In this study, the feature points can be extracted from color image and segmented image using edge detection methods From these feature points, we make triangulated model using delaunay triangulation method

Texture mapping & Point sample rendering This figure shows the image of environment model using texture mapping and point sample rendering As shown in middle one, the stretching effect of the 3D model occurs by the occlusion Also, holes occurs by the lack of the information when the point sample rendering method is applied To resolve these problem, we must construct the environment model using a multiple number of images. Wireframe Texture Mapping Point Sample Rendering Texture mapping : the stretching effect of the 3D model occurs by the occlusion Point Sample Rendering : holes occurs by the lack of the information

Regular Image Regular Image Desired Image Segmentation Environment Mapping Now, We’ll talk about the process of the environment mapping and environment modeling from a regular image This Color figure show the hallway of the building. we can obtain segmented image using region based segmentation method, And then, We can get the depth image using the depth calculation equation. And then, We can build the 3D Environment model using depth image through rectangular grid modeling methods This figure shows the image of the environment model using Texture mapping methods Depth Calculation Environment Modeling

Regular Image Gyeongbokgung (Royal Palaces) Seokguram (Stone Cave)

OCC Map (Indoor Scene) Segmentation Depth Calculation Environment Map (Octahedral Map) Orthogonal Cross Cylinder Environment Mapping Segmentation This Figure shows the process of environment mapping from OCC map and environment modeling from depth image. Depth Calculation Environment Modeling

OCC Map (Outdoor Scene) Orthogonal Cross Cylinder Segmentation Based Environment Model Segmentation Desired Image Depth Calculation

Result (Art Gallery) Environment Map (sidestep) Segmented Based Environment Model (sidestep)

Conclusion & Future Works can be used for environment modeling by extracting the depth value by the image segmentation navigate through the virtual environment when the viewpoint has been changed easy to implement on the environment map Future Works Multiple Resolution the problem of the blurring-image effect caused by the fixed resolution Multiple Images (Registration & Integration) the problem of the stretching effect of the 3D model caused by the occlusion Generalization to apply any type of images only images whose viewpoint and the floor are perpendicular may be used.