1. 1 Vision Based Following of Locally Linear Structures using an Unmanned Aerial Vehicle Sivakumar Rathinam, Zu Whan Kim, Raja Sengupta Center for Collaborative Control of Unmanned Vehicles University of California, Berkeley

2. 2 Motivation  Aim: Enable UAV use for infrastructure monitoring Traffic monitoring, aqueduct inspection, pipeline monitoring ….

3. 3 Our Technology  Keep the vehicle over the structure using with vision in the loop  Complement GPS waypoint navigation Waypoint navigation to get the vehicle over the structure Lock it on using vision for accurate imaging

4. 4 Forest Fire Monitoring Inter-Office Cargo Delivery Motivation  Unmanned Aerial Vehicles for Traffic Surveillance The Ohio Department of Transportation, The Florida Department of Transportation, The Georgia Department of Transportation  Lane changes, Average inter-vehicle distances, Heavy vehicle counts, Accidents, Vehicle trajectories, Type of vehicles etc.  The road should be in view.  Coifman et. al, Surface Transportation Surveillance from Unmanned Aerial Vehicles “The turning radius of the fixed wing UAV is such that changing directions at waypoints can take some time and space until the vehicle regains its course. When traversing roadway links of lengths less than 400 ft, large portions of the links went unobserved.”

5. 5 Motivation Hanshin Expressway, Japan 1995 Alaska pipeline  The visual feedback compensates GPS inaccuracies and tracks the structure even it is shifted from the assumed location.

6. 6 Generalization: Vision Based Following of Locally Linear Structures (Closed Loop on the California Aqueduct, June 2005)

7. 7  The average error of the position of the vehicle from the curve was 10 meters over a length of 700 meters of the canal.  Algorithm ran at 5 Hz Results Tracking the California Aqueduct

8. 8 Current UAV Platform Configuration  Wing-Mounted Camera allowing for vision-based control, surveillance, and obstacle avoidance  Ground-to-Air UHF Antenna for ground operator interface  GPS Antenna for navigation  802.11b Antenna for A-2-A comm.  Payload Tray for on-board computations and devices  Payload Switch Access Door for enabling / disabling on-board devices

9. 9 Current Payload Configuration  Off-the-shelf PC-104 with custom Vibration Isolation  Orinoco 802.11b Card and Amplifier for A-2-A comm.  Analog Video Transmitter for surveillance purposes  Printed Circuit Board for Power and Signal Distribution among devices.  Umbilical Cord Mass Disconnect for single point attachment of electronics to aircraft.  Keyboard, Mouse, Monitor Mass Disconnect for access to PC-104 through trap door while on the ground.

10. 10 Problem  Follow a given curved structure based on visual feedback. Overhead View

11. 11 Following a Structure using Visual Feedback 1.Structure detection a.Learn the structure from a one example b.Real time structure detection of the structure c.Curve fitting 2.Tracking a.Transformation of image to ground coordinates b.Control the vehicle to follow the structure  Hardware in the loop setup and evaluation  Experiments

12. 12 Basic Detection Idea Locally linear: Structure should look approximately linear in each image

13. 13 1a. Learning the Structure from One Example Rectify image -Finding the vanishing point

14. 14 1a. Learning the Structure from One Example mean variance Road Template Mean intensity will show high variation at the boundary The variance in between the boundary points should be low Done off-line Can be automated or manual

15. 15 1b. Real Time Detection in each Image Road Template For every 4 th horizontal scan line pick several boundary hypotheses -Each hypothesis is a pair of features (high local intensity gradient) -Score each hypothesis for match quality with learnt profile -Keep the best three hypotheses for each scan line

16. 16 1b. Real Time Detection in each Image Road Template For every 4 th horizontal scan line pick several boundary hypotheses -Each hypothesis is a pair of features (high local intensity gradient) -Score each hypothesis for match quality with learnt profile -Keep the best three hypotheses for each scan line  Corr(I h ’(p),L)

17. 17 1c. Curve Fitting Road Template RANSAC for Curve Fitting Pick four scan lines at random and four center hypotheses i.e., one from each line Fit a cubic spline Score the cubic spline Pick the spline with the best score Set of supporting scan line matches

18. 18 Cal Road Detection on MLB Video (No Control) Generic corridor detection by one- dimensional learning Roads Aqueducts Perimeters Pipelines Power Lines

19. 19 2a. Transforming Image to Ground Coordinates  Height is measured by the pressure sensors.  Use accelerometers and the gyros in the avionics package to calculate the transformation Roll and pitch  Internal calibration parameters Coordinates

20. 20 2b. Controlling the Vehicle to Follow the Structure  Find a connecting contour that joins the current position to the desired curve and follow that path Position and slope at the origin and the look ahead distance (Soatto 2000)

21. 21 Literature review – Vision Based Road Following Systems  VITS (1988) Tracked roads at 13 miles/hr  Dickmanns (1992) Tracked roads in autobahn at speeds up to 62 miles/hr  Taylor et.al (1999) Tracked roads at speeds up to 75 miles/hr  Eric Frew et.al (2003) Unmanned Aerial Vehicle Tracked roads at around 44 miles/hr

22. 22 More dangerous stuff…… Obstacle Avoidance  Experiment flown on a Sig Rascal airframe with a Piccolo avionics package and vision processing on an onboard PC104.  An 8.5 foot diameter balloon was used as the obstacle (distance currently calculated using GPS).

23. 23 Flight Demonstration

24. 24 Related Work  Vision-based obstacle avoidance has been studied primarily in the context of mobile ground robots. Lenser ’03, Ohya ’00, Lorigo ‘97,  Vision based navigation of UAVs Saripalli ’02, Shakernia ’02, Furst ’98 – Landing with known markings Sinopoli ’01, Doherty ‘00 – Visual landmark navigation (terrain avoidance) for helicopter Ettinger ’02, Pipitone ’01, Kim ’03 – Pose estimation for aircraft  Obstacle/Collision Avoidance for UAVs Mitchell ‘01 – Aircraft avoiding known aircraft Sigurd ’03 – Aircraft with magnetic sensors Sastry ‘03 – Helicopters avoiding known helicopters/obstacles How ’02 – MILP for Obstacle Avoidance  Vision based obstacle avoidance Barrows ’03 – Biomimetic reactive control

25. 25 Related Research  Ground robots Fixed baseline stereo – JPL, many others Monocular map construction – Lenser (CMU), Kim (Berkeley) Cooperative stereo - CMU  Optical Flow Helicopter ground following – Srinivasan/Chahl (Australia) Corridor following - USC helicopter Micro UAV obstacle avoidance – Centeye  UAV depth map construction Lidar – CMU Helicopter Project, Sastry (Berkeley Helicopter Project). Vision + high precision IMU – Bhanu (joint with Honeywell)  Stereo Vision GT Helicopter

26. 26 Requires Depth Typically use Stereo Vision  Given the image coordinates of a feature in one image if one can find the image coordinates of the feature in the other image (feature matching), and if one knows the rotation and translation of the two image planes then one knows the world coordinates of the feature (Ego-motion Estimation)

27. 27 Problem with Depth Estimation by Stereo Vision ZZ+Z+ Z-Z- 0 z Increased accuracy requires increased camera separation

28. 28 Accurate Depth Estimation is a Problem  Range error due to pixel errors is.

29. 29 Approach  UAVs flying at low altitudes must autonomously avoid obstacles  Strategy Segment the image into sky and non-sky  Non-sky in the middle  OBSTACLE Strategy 1  Aim at the sky Strategy 2  If it looms faster than a threshold and is in the middle  AVOID Else do NOTHING

30. 30 Tailored to………………..  For most UAV applications (>50 m), the obstacles of concern will be large objects such as towers, buildings or large trees  For these cases, the problem of obstacle detection is different from that of ground vehicles in environments cluttered with many obstacles. VS

31. 31 Segmentation at Moffet Airfield  Results for multiple regions found (only largest regions shown, dark blue represents all small regions)

32. 32 Sky Segmentation

33. 33 Flight Demonstration

34. 34 Vision Processing  Classification: balloon/horizon correctly found in ~ 90% of images  Time results: ~2Hz (120ms SVM, 200-600 ms horizon)

35. 35 Flying Low  Helicopter pilots fly low  FAA requires see and avoid  Find the freeway and follow it

36. 36 Used Sectionals to build a Manhattan model at 300 feet (approx.)  Simulation testing of Control

37. 37 Cal UAV: Target Capabilities Obstacle Avoidance  Simulation testing of Control Flight through Manhattan model (300+ feet)

38. 38 End 2