1. Allen Hanson Chris Jaynes Mauricio Marengoni Edward Riseman Howard Schultz Frank Stolle University of Massachusetts 3D Building Reconstruction With: Keith Hoepfner and Heddi Plumb Ascona 2001

2. Talk Outline  Introduction: RADIUS and Ascender I  Overview of Ascender II  Geometric Reconstruction of Rooftops  Experiments  Conclusions Lab URL: http://vis-www.cs.umass.edu APGD URL: http://vis-www.cs.umass.edu/projects/ascender/ascender.html

3. Research Goal(s) EO, Digital Elevation Maps IFSAR, Multi/Hyperspectral 3D Geometric Site Model Reconstruction With multiple objects, Using multiple strategies, From multiple images

4. RADIUS: Ascender I System for Automatic Site Modelling Multiple ImagesGeometric Models F 2D polygon detection l line extraction l corner detection l perceptual grouping F epipolar matching F multi-image triangulation l geometric constraints l precise photogrammetry F extrusion to ground plane F projective texture mapping F CVIU 72(2) 1998

5. Ascender Works  Extensive evaluation Good detection but high false alarms Good geometric accuracy Ground TruthAscender Results Detection Rates

6. Ascender Doesn’t Work  Delivered to NEL (NIMA)  Applied to classified imagery  Performance not as expected High failure rate (buildings not detected) High false positive rate  WHY? Imagery didn’t conform to design constraints! No Buildings Detected!

7. Some Observations  IU Algorithms work within correct context constrained contexts constrained object classes  Domain knowledge provides constraints from domain, partial results, strategies  Many strategies, only correct ones used selective application correct parameters fuse results from individual strategies into complete reconstruction

8. Key to Success A system which: applies the right strategy at the right time in the right place with the right constraints

9. Ascender II Overview

10. Dealing with Context: Ascender II  Goals 3D Geometric Site Model Reconstruction Focus: More complex building structures Context sensitive control strategies for applying algorithms  Multiple strategies  Multiple images  Multiple sensors  Knowledge Representation, Control and Inferencing  layered Bayesian networks EO, Digital Elevation Maps IFSAR, Multi/Hyperspectral

11. Observations Buildings = Location + Geometry Reconstruction = Localization + 3D Recovery 3D Recovery is a two-stage process Where are the buildings?What is their structure?

12. Ascender II Overview CP: Control Policy  IU Algorithms work within correct context.  Domain knowledge provides constraints.  Choose good strategy from many alternatives. Basic Principles

13. Ascender II Components  Visual Subsystem Library of IU Algorithms Geometric Database (data, models) Display/user interface through RCDE  Knowledge Base Control system Belief networks in Hugin Reasoning mechanisms over regions of discourse  Regions of Discourse Subset of available data Image regions, a model, partial interpretation

14. Knowledge Representation Possible Values H 1 ={j,..,k} H1H1 Variable of interest Schema/Belief Node Evidence Policy H3H3 H4H4 H2H2 First level of classification hierarchy Second level (object subclass) Combination of belief networks and visual schemas, implemented in Hugin. Encodes: Domain Knowledge Acquired Site Knowledge Control Mechanism Allows both diagnostic and causal inference. Hierarchical topology allows simpler networks which reduces propagation time and controls complexity. Evidence policy defines how IU strategies gather relevant evidence according to context.

15. Algorithms to Evidence  Define the context in which it can run currently straightforward context only disallows an algorithm too simple in the long run  Method for deriving certainty value needed to update knowledge network Given an IU algorithm....

16. Algorithms  Algorithms operate on regions of discourse Ascender I Surface Fits Line Extraction and Count Junction Count Average Junction Contrast Region Height Region Height to Width ratio Edge Terminals ……………...

17. 3D Geometric Rooftop Reconstruction

18. Fusing DEM and Optical Data  Registered Optical and Elevation  Reconstruct Rooftops (and buildings) Optical Image (Fort Hood)Elevation Data (Fort Hood)

19. Technical Overview Registered Optical and DEM Parameterized Model Library Site Model Visualization Model Indexing Elevation Surface and Histogram Model Fit: Optimization

20. Surface Primitive Library 51 Models in 8 Parameterized Classes Gable

21. Ascender + Terrest Results

22. Experiments Fort Hood Avenches Fort Benning

23. Experiments  Approach Site model from Ascender I with no filtering and loose polygon acceptance criteria => high detection rate but lots of errors (false positives) Classify Ascender I regions into {building, parking lot, open field, unknown} Classify buildings into {multi-level building, single-level building} Then into {flat, peak, round, flat-peak, other} Site model from Ascender II after classification/reconstruction

24. Degraded Ascender I Model Fort Hood 22 True Positive Buildings 1 Incorrect Model (A) (multi-level reconstructed as single level) No Knowledge and Loose Polygon Acceptance Criteria A 20 False Positive Buildings

25. Bayes Nets for Experiments Planar Fit Evidence Policy: none Multi-Level Single Building T Junctions Good Bad Yes No Region Building Open Field Parking Lot Complex Unknown Evidence Policy: none Height Evidence Policy: multi-EO: triangulateHeight() DEM: medianHeight() Low Medium High Planar Fit Good Bad Evidence Policy: DEM: robustFit(plane) Width Line Count L Junctions Number T Junc. Contrast T Junc. Evidence Policy: EO: junctions(T) <5 >5 None <50 >50 Zero Evidence Policy: EO: junction_count(T) EO: junction_contr(T) First level of classification hierarchy Second level (object subclass) Level 1 Level 2 Next Slide

26. Level 3 of Network Rooftop Flat Peak Round Flat-peak Other Evidence Policy: none Plana r Fit Good Bad Evidence Policy: DEM: robustFit(plane) Level 3 Middle > Corner Yes No Side Line Yes No Center Line Yes No Shadow Format Prism Round Flat Multi- level Yes No Evidence Policy: none Planar Fit A Good Bad Evidence Policy: DEM: robustFit(plane) Level 3 Height A > Height B Yes No Center Line Ratio Yes No Planar Fit B Good Bad Evidence Policy: DEM: robustFit(plane) Multiple calls, one for each section

27. Visual Subsystem: Algorithms and Evidence Policies Stereo-Optical DEM DEM Median Height Multi-Optical Epipolar Align. Region-Edge Triangulate Hgt. USGS DTMDTM at region center Height P{} xF{x} P{} Height FEvidence policies associate algorithms with belief. FFunction P maps request for evidence from knowledge base to algorithm. FF(x) maps algorithm result to belief value. FExample Algorithms: l Parallel Lines l T junctions l DEM-Median-Height l Region-Edge-Triangulate-Height l Planar Fit Error

28. Experiments F Detection rates per Object Class (Semantic Accuracy) F Three-Dimensional Accuracy l For each DataSet l For each Object Class (buildings) F How do network priors influence performance? l Uniform (no knowledge) versus Estimated Priors l Adjustment of Priors for each dataset versus uniform F Three Datasets l Fort Benning MOUT l Fort Hood Facility l Avenches Boat Yard

29. Priors and Operators Used Network Priors by Dataset Operators Applied by Dataset Skip F.Hood

30. Ascender II Classification Context Sensitive Strategies 43 Regions, 4 Misclassified, 1 Unknown 88% Accuracy

31. Ascender II Reconstruction 3 False Positives 23 True Positive Buildings 0 Incorrect Models (single, multi level) 3 New Functional Areas

32. Site Reconstruction: Fort Hood

33. Reconstruction Accuracy: Fort Hood Detection Rates by Object Class Median Accuracy Measures, Building Classes Skip Avenches

34. ISPRS Avenches Dataset F Two Overlapping Optical Images (60% sideward) F Corresponding DEM F Calibration provided by ETH F 200x220 meter “Boatyard” F Ground Sample Dist: ~ 0.13 meters F Reference model provided by ETH

35. Regions and Classifications: Avenches Parking Lot Peak Bldg. Peak Bldg. Peak Bldg. Peak Bldg. Open Field Flat Bldg. Flat Bldg. Flat Bldg. Flat Bldg. Flat Bldg. Peak Bldg.

36. Site Reconstruction: Avenches Skip Benning

37. Fort Benning MOUT Site F Two overlapping downlooking optical images F DEM computed from image pair using Terrest F Camera resection (SRI) F Reference Model (SRI) F GSD, optical view, 0.31 meters Optical DEM

38. Original ImageAscender I Polygons Polygons from SAR + Final Polygons MOUT Input Polygons

39. MOUT Site: Classification Object Class TrueFalseFalse Positive Positive Negative Building (General) 19 1 0 Peak Roof 14 1 0 Flat Roof 5 0 0 Single Car 1 1 2 Flat roof building detected as peak. Low walled entrance (open field) Detected as single vehicle. Missed detection of two vehicles. Overall classification accuracy: 91% Recovery of 4 attached buildings.

40. Model Indexing: MOUT Region 8 Constructed Surface Mesh Corresponding Gaussian Sphere Plane Flat Peak Peak (15) Peak (35).939.819.742.601 Correlation Scores

41. MOUT Site: Accuracy Detection Rates, Full Evaluation Detection Rates, FOA Regions Only

42. MOUT Site: Reconstruction

43. MOUT Reconstruction Skip Kodak

44. Examples on Three Image Chips Hospital FineArts Industrial N

45. Building Localization OpticalLines2D Polygons Building FP Hypotheses + Final Building Location Hypotheses + Classifier: Trees Buildings Ground Building FP Hypotheses ElevationBlobs Buildings (st. lines) Trees (no st. lines) v Building FP Hypotheses Registration FP = Footprint

46. Building Localization: Polygons Group Focus (optional) Graph Polys Geometric Recovery Classification Canny EdgesGrouped LinesHypotheses Building Footprint On Range Data Essentially Ascender I 2D Polygon Detection Algorithm with additional external inputs providing additional constraints.

47. Building Localization: Blobs Bald Earth Model Thresholded using bald earth model Filtered using Long Lines and Region Size First Threshold Lines from Optical Range Data

48. Localization Results Hospital Industrial Fine Arts

49. Final Building Models

50. Conclusions  Vision works when properly constrained and focused  Still a lot of work to do on: Basic Algorithms Knowledge Representations Representing and Using Context and Constraints Inferencing and Causal Reasoning System Architectures

51. Conclusions, Part 2 No shortage of interesting research problems!

52. Lab URL: http://vis-www.cs.umass.edu

53. 3D Geometric Rooftop Reconstruction

54. Model-Based Indexing FUse DEM and focus-of-attention to recognize rooftop shape. FBuilding rooftops can be modeled as the union of a small set of parameterized surfaces. FVariety arises from different combinations and parameterizations. FIndex into a library of rooftop models. FProvide initial parameters for final reconstruction (robust fitting) FRegions classified as single-building are then passed to the Model- Indexing algorithm for recognition. l Library of models is filtered based on the distribution at the single-building node.

55. DEM + Optical 2: Complex Roofs  Registered Optical and Elevation  Reconstruct Rooftops (and buildings) Optical Image (Fort Hood)Elevation Data (Fort Hood)

56. Technical Overview Registered Optical and DEM Parameterized Model Library Site Model Visualization Model Indexing Elevation Surface and Histogram Model Fit: Optimization

57. Model-Based Indexing FUse DEM and focus-of-attention to recognize rooftop shape. FBuilding rooftops can be modeled as the union of a small set of parameterized surfaces. FVariety arises from different combinations and parameterizations. FIndex into a library of rooftop models. FProvide initial parameters for final reconstruction (robust fitting) FRegions classified as single-building are then passed to the Model- Indexing algorithm for recognition. l Library of models is filtered based on the distribution at the single-building node.

58. Visual Subsystem: Algorithms and Evidence Policies FEvidence policies associate algorithms with belief. FFunction P maps request for evidence from knowledge base to algorithm. FF(x) maps algorithm result to belief value. FExample Algorithms: l Parallel Lines l T junctions l DEM-Median-Height l Region-Edge-Triangulate-Height l Planar Fit Error

59. Experimental Method F Use “reference model” of each site as basis for evaluation F Centerline Distance to measure the difference between two arbitrary polygons/wireframes l tessellate target and reference, take average of minimum distance for all points between target and reference, and reference to target. l measures: size, shape, location error F Detection: C-dist(T,R) < 0.2*cube-root(volume(T)) F Planimetric, Altimetric, and Intervertex error F Voxel Overlap F Completeness = 100 x (True Positives)/(True Positives + False Negatives) F Correctness = 100 x (True Positives)/(True Positives + False Positives)

60. Algorithm Library  Library contains both lightweight and heavyweight processes  Algorithms expected to perform only under constrained conditions  Algorithms contain descriptions of context under which they can be applied