1. Vision Based Motion Estimation for UAV Landing
Cory Sharp, Omid Shakernia
Department of EECS
University of California at Berkeley

2. Outline Motivation Vision based ego-motion estimation
Evaluation of motion estimates
Vision system hardware/software
Landing target design/tracking
Active camera control
Flight videos

3. Goal: Autonomous UAV landing on a ship’s flight deck
Motivation
Goal: Autonomous UAV landing on a ship’s flight deck
Challenges
Hostile operating environments
high winds, pitching flight deck, ground effect
UAV undergoing changing nonlinear dynamics
Why the vision sensor?
Passive sensor (for stealth)
Gives relative UAV motion to flight deck
U.S. Navy photo

4. Objective for Vision Based Landing

5. Preliminaries Motion of camera relative to inertial frame is given by
Given fixed point on landing pad , its coordinates
in camera frame are given by
Body angular and linear velocities are given by
The coordinates of a point in the camera frame satisfy:

6. Camera Model Calibrated pinhole camera Perspective Projection:
The image of a point is
denoted by
Notice: where:
Important identity:

7. Planar Essential Constraint
All points on landing pad satisfy
Image correspondences satisfy
the planar essential constraint
Current camera
position
planar constraint
Desired camera
position
Feature points on landing pad

8. Planar differential essential constraint
For optical flow, image points satisfy the
differential planar essential constraint:
We have developed a technique to uniquely estimate differential planar essential matrix B
We have developed an algorithm to decompose B into its motion and structure parameters

9. Differential Motion Estimation
Differential planar essential constraint on B is written as:
vector of image velocities
matrix of image points
vector of entries of B
With at least 4 image points, F has rank 8
Using linear least squares, B is recovered up to ker(F):
the entries of are the LLSE of B
the entries of span ker(F):
observation:
Observation:
Solve uniquely for B using

10. Vision Performance Evaluation
Performance evaluation of vision sensor
under varying levels of noise is image correspondences and optical flow
under different camera distances to landing plane
under different camera motions relative to landing plane
Observations
planar algorithm performs better under measurement noise than general 8-point algorithm
motion estimates improve as camera approaches landing pad

11. Noise Sensitivity Discrete Case Differential Case Observation:
Planar case is more robust to noise than general case

12. Depth Sensitivity Discrete Case Differential Case Observations:5 10 15 1 2 3 4
Differential case- depth variation dependency: noise level .5 pixels
translation mean error (degrees)
ratio (zmax - zmin)/zmin
Planar
8-Point
0.02
0.04
0.06
0.08
0.1
0.12
Differential case- depth variation dependency: noise level .5 pixels
rotation mean error (degrees)
Discrete Case Differential Case
Observations:
Motion estimates improve as camera approaches landing pad
Translation estimates more sensitive to noise than rotation

13. Motion Sensitivity Discrete Case Differential Case Observations:
X-X
X-Y
X-Z
Y-X
Y-Y
Y-Z
Z-X
Z-Y
Z-Z 5 10 15
Discrete case- translation axis dependency: noise level 3 pixels
translation mean error (degrees)
translation-rotation axes
8 point
Planar
0.02
0.04
0.06
0.08
Discrete case rotation axis dependency: noise level 3 pixels
rotation mean error
Discrete Case Differential Case
Observations:
Worst estimates when translation parallel to surface normal

14. Vehicle Control Language
Vision in Control Loop
Camera
Pan/Tilt Control
Feature Point
Correspondence
Motion
Estimation
Image Processing,
Corner Finding
Helicopter State
RS-232
Control Strategy
Vehicle Control Language
Navigation Computer
Vision Computer

15. UAV Testbed

16. Vision System Hardware
Ampro embedded PC Little Board P5/x
Low power Pentium 233MHz, running LINUX
440 MB flashdisk HD, robust to body vibration
Runs motion estimation algorithm
Controls PTZ camera
Motion estimation algorithms
Written and optimized in C++ using LAPACK
Give motion estimates at 30 Hz

17. Vision System Software

18. Nonlinear Motion Estimation
Minimize reprojection error using Newton-Raphson
Gaussian elimination to solve for dlamda
Iteratively, this drives dbeta to 0

19. Pan/Tilt Camera Control
Feature tracking issues:
Leave the field of view
Pan/tilt increases the
range of motion of the UAV
Pan/tilt control
drive all feature points to the
center of the image

20. Coordinate Frames

21. Flight Test Results

22. Vision Based State Estimate, RMS Error
Position error to within 5cm
Rotation error to within 5deg

23. Vision Ground Station

24. Flight Video

25. Pitching Landing Deck Landing deck to simulate motion of a ship at sea
6 electrically actuated cylindrical shafts
Motion Parameters:
sea state (freq, amp of waves)
ship speed
direction into waves
Stiffened Aluminum construction
Dimensions: 8’ x 6’

26. Moving Landing Pad

27. Hovering Above Deck

28. Landing on Deck

29. Papers Published
A Vision System for Landing an Unmanned Aerial Vehicle
Cory Sharp, Omid Shakernia, Shankar Sastry
Submitted: ICRA 2001
Landing an Unmanned Air Vehicle: Vision based motion estimation and nonlinear control
Omid Shakernia, Yi Ma, T. John Koo, Shankar Sastry,
Asian Journal of Control, Vol. 1, No. 3, Sept. 1999
Vision guided landing of an Unmanned Air Vehicle,
Omid Shakernia, Yi Ma, Joao Hespanha, Shankar Sastry,
IEEE Conf. on Decision and Control, Phoenix, Arizona, Dec. 1999