1. Digital Imaging and Remote Sensing Laboratory Automatic Tie-Point and Wire-frame Generation From Oblique Aerial Imagery Seth Weith-Glushko Advisor: Carl Salvaggio Research Proposal November 7, 2003

2. Digital Imaging and Remote Sensing Laboratory Table of Contents The Problem Specific Aims Proposed Solution A description of the individual transforms and algorithms used to make up the proposed algorithms Methods Timetable Budget References

3. Digital Imaging and Remote Sensing Laboratory The Problem Using photogrammetric techniques, tie matching algorithms and bundle adjustment algorithms, it is possible to create a 3D model using a series of images of an object These images must be around an axis (for objects) or ortho-rectified (for landscapes) for the algorithms to work We want to be able to use oblique imagery for input into the algorithms By using oblique imagery, more pixels are available to describe the features of a 3D object than would be in an ortho-rectified image

4. Digital Imaging and Remote Sensing Laboratory Specific Aims Current algorithms only work on ortho-graphic (for tie-points) and axial imagery (for wire-frame generation) The goal is to draw from both photogrammetry and computer graphics to develop two algorithms which can work in unison A tie-point algorithm that works on oblique imagery A bundle adjustment algorithm that can use oblique imagery In the future, this algorithm would become part of a suite of algorithms that could generate an accurate 3D model of scene independent of the type of imagery used

5. Digital Imaging and Remote Sensing Laboratory Proposed Solution The algorithm below relies heavily on the use of INS (Inertial Navigational System) data The input would be a series of oblique images made around a common area The output would be a matrix of matching points and a file using a common 3D format Image Processing Ortho-rectification Transform Input: A series of oblique images around a common area Histogram processing Converts an oblique image into an ortho-rectified image

6. Digital Imaging and Remote Sensing Laboratory Proposed Solution Definition of Points Point matching algorithm Geometric Transform Use the Laplacian of Gaussian (LoG) filter to find points of high frequency (i.e. edges) Use point distance comparison, point scale comparison and point angle comparison to find matching points Using matched points, generate a geometric transformation and use registration as indicator of “goodness” of points

7. Digital Imaging and Remote Sensing Laboratory Proposed Solution Inverse Geometric Transform Inverse Orthorectification Transform Bundle Adjustment Algorithm Perform an inverse geometric transformation using previous transformation matrix Convert the orthorectified image back into its original oblique form Using pairs of matched points, define points in 3D space Output: Matrix containing matched pairs between images

8. Digital Imaging and Remote Sensing Laboratory Proposed Solution Interface with 3D System Using software libraries and defined 3D points, generate an output file Output: 3D wire-frame mesh of a scene

9. Digital Imaging and Remote Sensing Laboratory Ortho-rectification Transform The ortho-rectification transform converts an oblique image into an ortho-rectified (flat) image by means of a linear equation There are four unknowns. We can use the fiducial points of an image as the four points we need to solve for the constants

10. Digital Imaging and Remote Sensing Laboratory Image Processing Image processing needs to be performed because images with dissimilar digital count affect the point generation operator Histogram matching is performed to minimize this dissimilarity Images courtesy C. Salvaggio

11. Digital Imaging and Remote Sensing Laboratory Definition of Points To define points the Laplacian of Gaussian operator is used Walli found that if there is an edge in an image, a thresholded filtered image would show a point at that edge Images courtesy K. Walli

12. Digital Imaging and Remote Sensing Laboratory Point Matching Algorithms To match defined points, three algorithms are primarily used: pixel distance match, scale match, angle match Another matching criteria is LoG maxima similarity 25%

13. Digital Imaging and Remote Sensing Laboratory Pixel Distance Match This algorithm works by comparing distances between pixels in a matrix 1 2 3 4 Distance= 1 2 3 1 2 3 # Same Elements: 2 3 2

14. Digital Imaging and Remote Sensing Laboratory Scale Match This algorithm works by comparing ratios of distances between pixels in a matrix 0 1 2 3 4 1 2 3 4 0 1 2 3 4 5 6 7 1 2 3 4 Distance= Compare Ratios= Ratios of distances from like points is equal! 1 2 3 1 2 3

15. Digital Imaging and Remote Sensing Laboratory Angle Match This algorithm works by comparing the angle formed by the triangle of 3 pixels in a matrix 1 2 3 4 1 2 3 1 2 3 Vertice 1 2 3 Point Set 1 Point Set 2 Vertice 1 2 3

16. Using the matched pixels, solutions to the affine polynomial problem are found Using the affine polynomial, one ortho-rectified image is geometrically transformed so that it can register with another ortho-rectified image. Quality metrics are performed to determine whether the registration is good. Hence, the matched points are good. Digital Imaging and Remote Sensing Laboratory Geometric Transformation

17. Digital Imaging and Remote Sensing Laboratory Inverse Transformations The matched points are put through an inverse geometric transformation and an inverse ortho- rectification transform to return the points to their original oblique pixel form A matrix of matched points is output

18. Digital Imaging and Remote Sensing Laboratory Bundle Adjustment Algorithm Bundle adjustment algorithms allow the mapping of 2D points into 3D space using more than two images around a common point The algorithm estimates the underlying plane geometry of a scene Images courtesy M. Pollefeys

19. Digital Imaging and Remote Sensing Laboratory 3D Library Interfacing Using these 3D points generated from the bundle adjustment algorithm, a triangle mesh is created which forms the structure of the wire-frame scene Also, a texture map is generated from bundle adjustment. This map is overlaid on the mesh The full model is saved to a generic 3D format

20. Digital Imaging and Remote Sensing Laboratory Methods Using a programming environment, engineering code will be developed to determine the feasibility of this algorithm If it is feasible, quality metrics will be applied to determine effectiveness Visual analysis, absolute mean variance, and deviation from a polynomial model (RMSDE) can be used to check tie- point generation Visual analysis and post-photogrammetric analysis can be used to check wire-frame generation

21. Digital Imaging and Remote Sensing Laboratory Timetable September 1, 2003 – November 15, 2003 Search for previous research, background knowledge November 15, 2003 – April 1, 2004 Development of algorithm and engineering code April 1, 2004 – May 15, 2004 Complete paper, poster and presentation

22. Digital Imaging and Remote Sensing Laboratory Budget No money will be required for this project as the investigator has all of the resources he currently requires 2 credits will be required for both Winter and Spring Quarters Due to the nature of the contract, most of the work performed will be done on a pay basis Flexibility in the experimenter’s schedule was required

23. Digital Imaging and Remote Sensing Laboratory References Honkavaara, Eija and Anton Hogholen. “Automatic Tie Point Extraction in Aerial Triangulation.” International Society for Photogrammetry and Remote Sensing, 16th Congress, Vienna, July 1996. 337-342. Moffitt, Francis H. and Edward M. Mikhail. Photogrammetry. 3rd Ed. New York: Harper and Row, 1980. Pollefeys, M. “3D Geometry from Images.” 15 Oct. 2003. Walli, Karl C. “Multisensor Image Registration Utilizing the LOG Filter and FWT.” Diss. Rochester Institute of Technology, 2003. Wolf, Paul R. Elements of Photogrammetry. 2nd Ed. New York: McGraw-Hill, 1983.

24. Digital Imaging and Remote Sensing Laboratory Questions Are there any?