1. Landing a UAV on a Runway Using Image Registration Andrew Miller, Don Harper, Mubarak Shah University of Central Florida ICRA 2008

2. Overview ► System for landing a UAV on a runway  Small RC airplane  Only sensor is a fixed, forward-looking camera  Finds the runway using SIFT registration  Linear control system ► Experiments  Microsoft Flight Simulator (no flight model)  Partial implementation on a real UAV

3. Human operator for high level control Laptop Computer for Vision Processing and Control Algorithms Ground Station Block Diagram RC Plane with Camera 30 frames per second 720x480 pixels RGB Downsample to 360x240 Remote Control (72 Mhz) Analog Video (NTSC 900 Mhz)

4. Main Steps 1. Locate the runway in each video frame 2. Estimate the attitude of the UAV 3. Steer the UAV towards the runway maintaining the correct glideslope

5. 1. Locate the Runway ► Base point and vanishing point (location and orientation) Base point Vanishing point

6. Planar Homography ► The 3x3 planar homography matrix projects every point in the reference frame to the corresponding point in the incoming video frame Reference Frame Warped Reference Frame Test Frame

7. Find the Homography using SIFT and RANSAC ► SIFT Feature Matching  200-500 feature points, 100-200 matches  Chosen greedily, least ambiguous first ► Planar homography between correspondences ► RANSAC to discard outliers

8. Stack of Reference Frames ► Prepare a reference frames from a video  Annotate the runway and vanishing point  Sample the frames (more samples at lower altitudes)  Terrain features are important (not just the runway)

9. Stack of Reference Frames

10. Using the Stack ► Keep track of the current index  Highest number of SIFT matches = most similar viewpoint ► Only need to compare adjacent frames Previous Closest Match SIFT Matching

11. 2. Estimate the UAV Attitude ► 6 Degrees of Freedom  Pitch, Bank, Heading, Elevation, Distance, Course ► Strategy  Ignore Distance  Find Pitch and Bank from the horizon line (x-axis)  Find Elevation, Heading, Course from the runway

12. Intuitive Geometry ► Relationship between runway appearance and UAV attitude  This is how human pilots land visually Too HighOn TargetToo Far Right

13. Formal Geometry ► 3D Projection  C = Internal Calibration  R = External Calibration ► Small Angle Approximation  Assume the UAV is flying smooth and level

14. 2. Estimate the UAV Attitude ► Recover the orientation parameters ► Vanishing point of the runway ► Beginning of the runway

15. Find the Horizon ► Horizon estimation algorithm by Ettinger, et al. ► Based on Differing Color Distributions ► Used to recover two (pitch / bank) Correct HorizonWrong Horizon

16. 3. Control the UAV ► Cascaded Linear Feedback Controller  Two separate chains  Two gains ► Proportional ► Integral ► Intuitive  If UAV is too far right, steer left  If UAV is too high, pitch down  Bank angle is derivative of heading, heading is derivative of course  Pitch is derivative of elevation PI 1 PI 2 PI 3 PI 2 Course Heading Bank Elevation Pitch

17. Autopilot GUI

18. Algorithm Performance ► Multiple stages  Control loops run at 50 Hz ► Integrates smoothly even while input stays same  Horizon detection runs at 10 Hz ► Pitch and bank are the most sensitive  Runway detection runs at 2 Hz ► Elevation and course are the least sensitive

19. Simulator Results ► Microsoft Flight Simulator ► Simulator-in-the-loop (separate computer)  ICRA08_1140_VI_fi.mp4 ICRA08_1140_VI_fi.mp4  Horizon 2007 09 Sep 11 Tue 08.13pm.avi Horizon 2007 09 Sep 11 Tue 08.13pm.avi Horizon 2007 09 Sep 11 Tue 08.13pm.avi

20. Simulator Results ► Error from earth curvature

21. Actual UAV Experiments ► Only using partial implementation  Horizon stabilization  Road following (no runway available)  Only brief periods of autonomous control

22. Horizon Stabilization - Results

23. Conclusions ► Successful but imprecise landings ► Performance is applicable to Cessna  Slower and more stable than actual UAVs ► Assumption of linear system is not applicable near the runway  This is why the aircraft oscillates before landing ► Future work  Incorporate flight model into controller design