1. IGARSS 2011, Vancouver, Canada
Stabilization and Georegistration of Aerial Video Over Mountain Terrain by Means of LIDAR
IGARSS 2011, Vancouver, Canada
July 24-29, 2011
Mark Pritt, PhD
Lockheed Martin
Gaithersburg, Maryland
Kevin LaTourette
Lockheed Martin
Goodyear, Arizona

2. Problem: Georegistration
Georegistration is the assignment of 3-D geographic coordinates to the pixels of an image.
It is required for many geospatial applications:
Fusion of imagery with other sensor data
Alignment of imagery with GIS and map graphics
Accurate 3-D geolocation
Inaccurate georegistration can be a major problem:
Misaligned GIS
Correctly aligned

3. Solution
Our solution is image registration to a high-resolution digital elevation model (DEM):
A DEM post spacing of 1 or 2 meters yields good results.
It also works with 10-meter post spacing.
Works with terrain data derived from many sources:
LIDAR: BuckEye, ALIRT, Commercial
Stereo Photogrammetry: Socet Set® DSM
SAR: Stereo and Interferometry
USGS DEMs

4. Methods
Create predicted images from the DEM, illumination conditions, sensor model estimates and actual images.
Register the images while refining the sensor model.
Iterate.
Aerial Video Sensor
Image Plane
Scene
Occlusion
Illumination
Shadow
Predicted Images

5. Methods (cont) Predicted Image from DEM
Predicted Image from Aerial Image
Registration Tie Point Detections
The algorithm identifies tie points between the predicted and the actual images by means of NCC (normalized cross correlation) with RANSAC outlier removal.

6. Methods (cont)
The algorithm uses the refined sensor model as the initial guess for the next video frame:
The refined sensor model enables georegistration.
Exterior orientation: Platform position and rotation angles
Interior orientation: Focal length, pixel aspect ratio, principal point and radial distortion
Initial Camera
Estimate camera model
Use camera focal length & platform GPS if avail.
Register
Predict images from DEM and camera
Register images with NCC
Refine
Compose registration fcn & camera
LS fit for better cam estimate
Iterate
Next Frame
Register to previous frame
Compose with cam of prev. frame for init. cam estimate
Iterate for each video frame
Finish
Trajectory
Propagate geo data from DEM
Resample images for orthomosaic

7. Example 1: Aerial Motion Imagery
Inputs:
Aerial Motion Imagery over Arizona, U.S.
1/3 Arc-second USGS DEM
Area: 64 km2
Post Spacing: 10 m
16 Mpix, 3.3 fps, panchromatic

8. Problem: Too shaky to find moving objects
Example 1 (cont)
Problem: Too shaky to find moving objects
Zoomed to full resolution (1 m)

9. Example 1: Results Outputs: Sensor camera models
Images georegistered to DEM
Platform trajectory

10. Example 1 Results (cont)
ATV Vehicle
Human
Pickup Truck
Video is now stabilized, and as a result, moving objects are easily detected.

11. Example 2: Oblique Motion Imagery
Inputs:
Oblique Motion Imagery Over Arizona, U.S.
LIDAR DEM
Area: 24 km2
Post Spacing: 1 m
16 Mpix, 3.4 fps, pan

12. Stabilized Video Inset
Example 2: Results
Map coordinates
Stabilized Video Inset
Orthorectified Video
Background LIDAR DEM
Aligned Map Graphics
Target Tracking

13. Example 2 Results (cont)
How fast does the algorithm converge?
The initial error is high, but it decreases after only several iterations.
Subsequent frames have better initial sensor model estimates and require only 2 iterations.
IMAGE 1
Camera Iteration 1 2 3
Num tie points:
319
318
282
RMSE:
17.4
4.8
2.9
Mean Δx:
1.4
-0.7
0.1
Mean Δy:
-3.8
-0.1
Sigma Δx:
15.8 4
2.5
Sigma Δy: 6
2.6
1.5
IMAGE 591
Camera Iteration 1 2 3
Num tie points
681
687
RMSE
2.7
0.6
0.3
Mean Δx
Mean Δy
0.9
Sigma Δx
2.1
0.5
Sigma Δy
0.2
0.1

14. Aerial Video Over Arizona, U.S.
Example 3: Aerial Video
Inputs:
LIDAR DEM
Area: 24 km2
Post Spacing: 1 m
Aerial Video Over Arizona, U.S.
720 x 480 Color 30 fps

15. Background Image Draped Over DEM
Example 3: Results
Map coordinates
Orthorectified Video
Background Image Draped Over DEM
Aligned Map Graphics

16. Example 3 Results (cont)
Map Graphics Stay Aligned with Features in Video

17. Example 4: Thermal Infrared Video
Inputs:
Commercial LIDAR DEM
MWIR Video Over White Tank Mountains in Arizona
Post Spacing: 2 m
1 Mpix, 3.3 fps

18. Video Mosaic Georegistered and Draped Over Mountains in Google Earth
Example 4: Results
Video Mosaic Georegistered and Draped Over Mountains in Google Earth
BackgroundLIDAR DEM
Video Mosaic
Inset: Original Video with Map Graphics Overlay

19. Demo
Click picture to play video

20. Conclusion
We have introduced a new method for aerial video georegistration and stabilization.
It registers images to high-resolution DEMs by:
Generating predicted images from the DEM and sensor model;
Registering these predicted images to the actual images;
Correcting the sensor model estimates with the registration results.
Processing speed is 1 sec per 16-Mpix image on a PC.
Absolute geospatial accuracy is about 1-2 meters.
We are developing a rigorous error propagation model to quantify the accuracy.
Applications:
Video stabilization and mosacs
Cross-sensor registration
Alignment with GIS map graphics