1. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 1 Zhilin Li & Kourosh Khoshelham Dept of Land Surveying & Geo-Informatics Hong Kong Polytechnic University Integration of Multi-source Data for Automated Building Extraction

2. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 2 Automated building extraction: Problems Diversity of buildings: Buildings in a variety of shapes and sizes; Loss of information: Buildings are three- dimensional objects but the third dimension is lost in 2D images.

3. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 3 Automated building extraction: Solutions To minimise information loss High percentage of overlap between images Multi-source images Multi-viewpoint images To compensate for the loss Additional data sources, e.g. 2-D map data and height data Knowledge, e.g. building models

4. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 4 Our methodology Solution 1: Building extraction based single image Solution 2: integration of image data and height data (DSM) Solution 3: integration of 2D ground plans, image and height data (DSM) Model ImageDSMGround plan

5. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 5 Image + model Image + height + model Image + height + 2D plan +model Process

6. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 6 Approach 1 From image to edge From edge to line From line to grouped lines From grouped lines to building (matched with building model selected from library)

7. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 7 The process Locate the building on image (drawing a box); Select a model (of the building type) from model library (parametric models) Model matched with buildings on image (represented by lines which are extracted from image) Parameters for the model is then computed

8. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 8 Recursive split-and-merge line fitting process edge detection (e.g. by Canny operator) From image to edge to line

9. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 9 From image lines to grouped lines: Perceptual grouping of image lines Perceptual grouping based on two relations that are viewpoint invariant: proximity parallelism Significance of each relation defined using a number of linguistic fuzzy rules Example of a fuzzy rule for proximity relation: If DISTANCE is Close and LENGTH is Short then ProximityDecision is ProbablyConnected. Example of a fuzzy rule for parallelism relation: If ANGLE is Small and LENGTH is Short then ParallelismDecision is ProbablyParallel.

10. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 10 Construction of the library of building models

11. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 11 Model selection and matching Model selection is interactive. A human operator selects a model from the library. Once the model is selected, it will be automatically matched with the grouped image lines. For the matching, corresponding model and image lines must be selected. The model is searched for the model lines that carry the same relations as in the grouped image lines.

12. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 12 Matching procedure The model is parameterized. Initial model parameters are estimated. Model lines are transformed to image space. Distance between the endpoints of every model line and the corresponding image lines are measured. Corrections to model parameters are estimated that minimize the sum of squared distances.

13. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 13

14. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 14 fusion of image and height data for automated building extraction By making use of height data these two steps can be automated. In this work a split-and-merge method is developed to fuse image and height data, and automate the model selection. Localization of the building in the image is also carried out automatically by applying a top hat transform to height data.

15. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 15 Integration of image data and height data (DSM) Image data used to extract building planes using segmentation Height data used to find building locations Height data used to assist segmentation Also automated model matching

16. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 16 Building edges from image by segmentation to partition an image into a finite set of homogenous regions that correspond to surfaces in object space

17. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 17 Detection of building candidates from height data (DSM) Opening: flattens sharp peak and height blobs Such areas can be retrieved by subtracting the opened DSM from the original DSM The process is called morphological top hat transform

18. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 18 Problems with image segmentation Over-segmentation occurs when the segmentation method detects more than one region on a single surface. Under-segmentation occurs when the segmentation method fails to sufficiently separate two or more surfaces.

19. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 19 A split-and-merge process for refining the initial segmentation by making use of height data Height points are projected from the DSM to image regions. An LMS (least median of squares) estimator is used to fit planar faces to the height points in every image region. Initial segmentation results are refined using the computed planar faces.

20. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 20 Split process Original image Over-grown region Two planes found in height data Over-grown region split

21. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 21 Merge process Every two neighboring regions whose planar faces are coplanar are merged.

22. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 22 An example of the performance of the split-and-merge process Roof planes are identified by checking the height of the region-planes over the DTM. Roof planes are attributed as flat-roof or slanted-roof according to their parameters.

23. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 23 Automated model selection Number of roof planes are used as an index to select a class of models from the library. Attributes of the roof planes are used to further limit the number of selected models.

24. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 24 Model selection is automated. Only a 2D roof model is matched with the image. Library of parametric models

25. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 25 2D parameterization of the roof models Since the parametric forms of the roof planes are known, only a 2D model of the roof is matched with the image lines to compute the boundary of the roof.

26. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 26 Matching a 2D roof model with image lines

27. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 27 Approach 3: integration of 2D plans with image and height data for automated building extraction Integration of 2D ground plans leads to the reconstruction of generic polyhedral models that cover a wider range of buildings. While building detection using top hat transform may include objects other than buildings, using 2D ground plans and height data leads to a more reliable detection.

28. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 28 Overview of the approach

29. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 29 Overview of the approach Buildings are localized in the image using ground plans and DEM. Assuming that walls are vertical, their parametric forms are derived from the ground plans. The parametric forms of roof planes are derived from the integration of image and height data using the split-and-merge technique. Model faces are intersected and assembled to form a generic polyhedral model using a plane patch reconstruction method

30. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 30 Experiments and results -Image data consists of an orthoimage in four channels with 0.5m ground resolution. -Height data consists of DSM and DTM with 1m resolution obtained from a laser scanner. -Ground plans are manually digitized from the orthoimage.

31. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 31 Color (RGB) aerial orthoimage Color infrared (CIR) orthoimage Image data and the computed NDVI map Computed NDVI mapGray-scale image (for segmentation)

32. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 32 Lidar DSM (first echo) Lidar DTM and detected building candidates Detected building candidates Height data Lidar DSM (last echo)

33. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 33 10 successful reconstruction (parametric models); 3 failure in the split-and-merge process; 1 failure in the model matching process Results models reconstructed using split-and-merge process and model matching Front view Rear view

34. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 34 Result Integration of 2D ground plans manually digitized from orthoimages in the plane patch reconstruction method The use of ground plans for the localization of buildings leads to an improvement in the results of the split-and-merge process B12 Plane patch reconstruction method

35. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 35 Rear view 11 successful reconstruction (generic polyhedral models); 2 failure in the split- and-merge process; 1 failure in the plane patch reconstruction method Front view Completeness evaluation split-and-merge process and plane patch reconstruction

36. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 36 Other tests Accuracy efficiency

37. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 37 Conclusions The combination of the proposed approaches in one system will provide the user with a great flexibility to choose the most suitable method based on the available data.

38. Workshop on Earth Observation for Urban Planning and Management, 20th November 2006, HK 38 End of presentation