1. Appearance Contrast for Fast, Robust Trail-following
Christopher Rasmussen, Yan Lu, & Mehmet Kocamaz
Dept. Computer & Information Sciences
University of Delaware
U.S.A.

2. Introduction
This work describes a method for segmenting hiking/mountain-biking trails for autonomous navigation
Triangular image regions are directly hypothesized and scored using a general contrast measure that works on a wide range of trail types
No a priori color model of trail
Method is extended to ladar obstacle information projected into image for low-contrast situations
2018/12/3

3. Outline Hardware Appearance-based trail segmentation
Incorporating structural information
Experimental results
Conclusions and future work
2018/12/3

4. Hardware Robot platform: Segway RMP 400 Primary sensors
Data in this paper was collected under manual control
Autonomous control for segmented trails has been validated in earlier work and at competitions such as IGVC (
Primary sensors
Single color camera
Laser range-finders
SICK LMS for trail-side obstacles
Hokuyo URG for groundstrike filtering via slope estimation
2018/12/3

5. Outline Hardware Appearance-based trail segmentation
Incorporating structural information
Experimental results
Conclusions and future work
2018/12/3

6. Appearance-based trail segmentation
Trail region geometry
Trail region appearance characterization
Trail likelihood function
Likelihood maximization and tracking
Dynamic feature switching
Trail/No trail classification
2018/12/3

7. Trail region geometry
Approximate trail boundary as a triangle T with bottom side coincident with image bottom
Parametrize with 4-D vector consisting of top vertex and left and right bottom vertices (xt, yt, xl, xr)
Letting T’s width w = xr – xl, define left and right neighbor regions as TL = (xt, yt, xl - w, xl) and TR = (xt, yt, xr, xr + w) , respectively
(xt, yt)
2018/12/3 xl xr

8. Trail region geometry
Approximate trail boundary as a triangle T with bottom side coincident with image bottom
Parametrize with 4-D vector consisting of top vertex and left and right bottom vertices (xt, yt, xl, xr)
Letting T’s width w = xr – xl, define left and right neighbor regions as TL = (xt, yt, xl - w, xl) and TR = (xt, yt, xr, xr + w) , respectively TL T TR
2018/12/3

9. Trail region appearance characterization
Compute color features (aka textons) via k-means clustering in CIE-LAB space (following Blas et al., IROS 2008)
Under- and over-saturated pixels not included in k-means—these are given special labels
Clustering done over 3 different feature sets (these will be used for feature switching)
LAB
AB (chromaticity only)
L (brightness only)
Model trail region T’s color distribution via texton histogram HT
LAB (k = 8)
AB (k = 8)
L (k = 8)
Input image
2018/12/3

10. Trail likelihood function
Intuitively, relevant cues for a candidate trail region’s appearance likelihood are:
Color/brightness contrast with left & right neighbor regions
Symmetry between neighbor regions
Homogeneity
Can quantify region color similarity by standard histogram metrics—e.g., chi-squared distance 2
Formulate likelihood as weighted sum*:
*Here we neglect homogeneity (inverse entropy) as it appears to create a bias toward smaller regions
left contrast
right contrast
left/right symmetry
2018/12/3

11. Likelihood maximization and tracking
Find and track good trail candidates via MAP estimation using particle filtering
For static images, trail estimate is highest likelihood particle found after t iterations
For image sequences, state is standard sum of particles weighted by their likelihoods
Small fraction of particles are sampled from image-wide prior (rather than near previous state)
2018/12/3

12. Dynamic Feature Switching
Depending on visual conditions (cloud cover, shadows) and trail terrain (tread material, adjacent vegetation), different feature sets may provide sharper contrast and therefore better segmentations
In example below, Lappear is 6% higher using AB textons than LAB and 32% higher than for L
Particle filter is run on each image using 3 different feature sets; feature yielding highest likelihood is propagated to next iteration
For results here, feature set selected is indicated by color of fitted triangle: LAB = red, AB = green, L = blue (see previous slide)
LAB AB L
2018/12/3

13. Trail/No trail Classification
Threshold on estimated likelihood Lappear can be used to detect non-trail images
In our results, such classifications are indicated by shading the image red
Two data sets*
30 trail images collected by us from web sources such as Flickr, Google image
30 non-trail images taken from data used in “Recovering surface layout from an image,” Hoiem et al., IJCV 2007
Median Lappear for trail images: 0.700; for non-trail: Threshold of 0.57 results in correct classification of 28/30 trail images, 28/30 non-trail. Non-trail errors and selected correct classifications are shown below
2018/12/3 *

14. Outline Hardware Appearance-based trail segmentation
Incorporating structural information
Experimental results
Conclusions and future work
2018/12/3

15. Incorporating structural information
Analogous to visual feature switching, we may want to switch between vision-based and ladar-based trail tracking in certain situations
SICK ladar is mounted to sweep plane parallel to the ground
Project ladar points into camera image with size determined by depth
Lladar rewards hypotheses with “emptiest” T and “fullest” TL, TR
Overhead view of obstacle points
Obstacle points projected to image
(state here from appearance likelihood)
2018/12/3

16. Groundstrike Filtering
When robot pitches down because of bumps or ruts in trail, SICK ladar may “see” ground as obstacle
Hokuyo ladar is mounted to sweep a sagittal plane
Estimate ground slope with RANSAC line fit to predict SICK beam-ground intersection
Any obstacle points beyond intersection are assumed groundstrikes and removed before projection to image
2018/12/3

17. Groundstrike Filtering
When robot pitches down because of bumps or ruts in trail, SICK ladar may “see” ground as obstacle
Hokuyo ladar is mounted to sweep a sagittal plane
Estimate ground slope with RANSAC line fit to predict SICK beam-ground intersection
Any obstacle points beyond intersection are assumed groundstrikes and removed before projection to image
2018/12/3 17

18. Outline Hardware Appearance-based trail segmentation
Incorporating structural information
Experimental results
Conclusions and future work
2018/12/3

19. Experimental results
Anecdotally, segmentation is good over long, varied stretches of trail and through difficult visual conditions
Feature switching is important, particularly in shadowed sections 10 20
2018/12/3 30 40 50

20. Experimental results
On 30-image web trail data set, our segmentations have median overlap score* with ground truth (100 PF iterations)
For comparison, consider surface layout method in Hoiem et al., 2007, treating “ground” class as trail, all else as background
Median overlap score is 0.431
2018/12/3
*Overlap(R1, R2) = A(R1  R2)2 / (A(R1)A(R2)), from Sclaroff & Liu, 2001

21. Experimental results
Results of tracking on a multi-km (~18K frames) hiking trail sequence for which we have ground truth segmentations at 100-frame intervals
0.681 median overlap score vs for Hoiem et al.
LAB cue was used in 40.4% of frames, AB cue in 34.6%, L cue in 25.0%
Excerpts intervals; captions are appearance likelihoods
0.774
0.724
0.733
0.641
0.698
0.743
0.792
0.683
0.646
0.614
0.713
0.744
0.628
2018/12/3

22. Experimental results Results of running on images from similar work
Blas et al., IROS 2008
Hadsell et al., RSS 2007
Kim et al., IROS 2007
Input
Ours
Hoiem et al.
2018/12/3

23. Outline Overview Appearance-based trail segmentation
Incorporating structural information
Experimental results
Conclusions and future work
2018/12/3

24. Conclusions Robust approach to visually finding and following a trail
Works well over a wide range of trail types and illumination conditions without parameter adjustment
Runs in real time suitable for control of a ground robot
20+ Hz on downsampled 80 x 60 images (all results in this paper)
Uncalibrated: state is in image coordinates, but need vehicle coordinates for motion planning
Some problems with very wide trails such that neighbor regions are substantially outside of image
Need wider FOV camera
2018/12/3

25. Ongoing work Updated visual system to stereo omnidirectional cameras
Add curvature to state, maintain color model of the trail region over time
GPU-based optimizations of core image processing
Trail state in vehicle coordinates
Stereo depth estimation for small & negative obstacles
Optical flow for visual odometry
2018/12/3

26. Thank you!
Questions?