1. 1 Howard Schultz, Edward M. Riseman, Frank R. Stolle Computer Science Department University of Massachusetts, USA Dong-Min Woo School of Electrical Engineering Myongji University, South Korea Error Detection and DEM Fusion Using Self-Consistency 7th IEEE International Conference on Computer Vision September 20-27, 1999 Kerkrya, Greece

2. 2 Long-Term Objectives l Generate 3D terrain models from multiple, overlapping images (including video sequences) Accurate - Photogrammetric applications Robust with respect to: – Widely spaced cameras – Oblique viewing – Occlusions – Non-lambertian surface patches Automatic Efficient

3. 3 Long-Term Objectives l Terrain models include an estimate of geospatial uncertainty Detect unreliable elevation estimates associated with blunders, occlusions, shadows, false matches,... Estimate the RMS elevation errors

4. 4 Environmental Monitoring l Wide-angle video: 1 meter per pixels covers a 3/4 km swath l Zoom Video: 10 cm pixels covers a 75 meter swath l GPS, IMU & laser altimetery continuously recordedWide-angleZoom

5. Reducing the forest to a simple model of poles and circles Biomass Estimation from Counting Trees

6. Counting Trees in 3D

7. 7 Terrain Reconstruction from a Oblique Views tilt: 34ºtilt: 53º

8. 8 Real World Problems l Need reliable estimates of accuracy l Almost impossible to get sufficient ground truth l Even 1 blunder in 1,00,000 is problematic

9. 9 l Work in object space to enable the fusion of multiple DEMs generated from multiple image pairs l Use Laclerc’s Self-Consistency measure to detect unreliable elevation estimates General Approach

10. 10 l Elevation estimates result from two types of correspondences True correspondences, characterized by small, normally distributed errors that result from –Surface micro structure –Geometric misalignment –Optical distortion False correspondences (outliers), characterized by large errors resulting from –random, unrealistic texture matches - Large effects Small effects

11. 11 l We use the UMass Terrain reconstruction system Terrest, which is an implementation of a hierarchical, texture matching algorithm l Terrest produces a set of pixel correspondences, which are stored in a disparity map D RT R denotes the reference image T denotes the target image l The pixels (i,j) in R and (i+D(i,j)) in T view the same surface spot l The process is not symmetric with respect to the reference and target images, D AB  D BA

12. 12 Correspondences Computed DEM True DEM Error l The computed DEM is the sum of the true surface structure and an error term

13. 13 l Two ways to compute a DEM from 2 images (A and B). A is the Reference and B is the Target B is the Reference and A is the Target

14. 14 l The intra-frame difference Z AB -Z BA =  AB -  BA Depends only on the computed DEMs l Taking the standard deviation of both sides  (Z AB -Z BA ) =  (  AB -  BA ) l The distribution (Z AB -Z BA ) provides a means to separate reliable from unreliable elevation estimates

15. 15 l If are normally distributed, except for a small number of outliers, and computed  AB,  BA dependent  AB,  BA independent 0 <  < 1 uncertainty geospatial intra-frame standard deviation l  describes the amount of statistical independence l  depends on surface geometry, viewing geometry, sensor type, optics,...

16. 16 l The tails of the distribution are dominated by unreliable points. l We need a method to estimate  (Z AB -Z BA ) when the distribution is polluted by unreliable points l Fit the histogram of (Z AB -Z BA ) to a Gaussian plus a constant The numbers h max, dz 0, , h 0 are parameters of the fit

17. 17 l Consider Z AB and Z BA to be unreliable if |Z AB -Z BA | > n  l n is a threshold l Small values of n pass more points which are less self-consistent l Larger values of n pass fewer points which are more self-consistent l The threshold can be set based on consistency or the number of points passed

18. 18 l A simple algorithm to estimate the optimal DEM Accumulate elevations that have an intra-frame difference less than the threshold. Keep Z AB and Z BA if  Z AB -Z BA  n  Compute the mean surface Z Go back and add in the elevations close to the mean surface, keep Z AB if  Z-Z AB  n  re-compute the mean surface Z

19. 19 Example 4 Views

20. 20

21. 21 Z 12 Z 13 Z 14 Z 23 Z 24 Z 34 Z 21 Z 31 Z 41 Z 32 Z 42 Z 43 4 views  12 image pairs  12 DEMs

22. 22 Z 12 -Z 21 Z 13 -Z 31 Z 14 -Z 41 Z 23 -Z 32 Z 24 -Z 42 Z 34 -Z 43 4 images  6 intra-frame differences Z AB -Z BA -1.0 0.0 +1.0

23. 23  h max dz 0

24. 24 Z 12 -Z 21 Z 13 -Z 31 Z 14 -Z 41 Z 23 -Z 32 Z 24 -Z 42 Z 34 -Z 43 Intra-frame differences after removing unreliable elevations Z AB -Z BA -1.0 0.0 +1.0

25. 25 Rendered View No. of consistent points 0 1 2 3 4 5 6 7 8 9 10 11 12 157 0 17 41 84 296 782 2087 5997 19139 74096 288430 926778 DEM

26. 26

27. 27 DEM Ortho-image Tree Counting l Group 1: for every bump in the DEM looked for a tree in the ortho-image l Group 2: for every tree in the Ortho-image looked for a bump in the DEM l 95% agreement Another Example

28. 28 Verification Using Photo-realistic Simulation l Comprehensive analysis requires ground truth, which is impossible to collect l Instead use photo-realistic synthetic images l Enables analysis from any view point l Allows for changes in lighting and surface texture

29. 29 l Start with a previously generated DEM and ortho-image (pseudo ground truth) l Define the viewing geometry l Use a photo-realistic rendering program to generate synthetic images of the pseudo ground truth l Recover the DEM and ortho-image and compare to the pseudo ground truth

30. 30 XC,YC,ZC)XC,YC,ZC) X Y Z X Y Z X Y Z World Coordinate System Camera Coordinate System    f

31. 31 Nadir views Oblique views

32. 32 Original image 400  400 region 400  400 region synthetic view

33. 33 l Self-consistency and geospatial error statistics as a function of viewing geometry base-to-height ratio (b/h) incidence angle (  ) B/h  A  B  (Z AB –Z BA ) % Inliers 2  cutoff  (Z * –Z AB )  (Z * –Z BA ) 0.277 0  15  0.45118991.90 0.332601 0.244706 0.213685 0.293 15  30  0.48681392.50 0.344480 0.330056 0.260698 0.575 15  -15  0.31155391.36 0.163137 0.213822 0.131443 0.868 -15  30  0.20350389.40 0.157535 0.194275 0.152326 1.230 30  -30  0.16771384.24 0.155302 0.188295 0.155993  (Z * –Z) –

34. 34 Reliable Point Mask  A = -30°  B = +30°  A = 0°  B = +15° No. of Reliable Points DEM RMS error: 17cm Elevation range: 762.7 - 885.7m GSD:35cm  2 consistent point:99.54%

35. 35 Future Directions l Develop models that predict the geospatial uncertainty  (  ) from the distribution of self-consistency (Z AB -Z BA ) l Use the DEM fusion techniques to generate terrain models from digital video sequences

36. 36

37. 37

38. 38

39. 39

40. 40 Left MosaicRight Mosaic 3D Rendering