1. Curb Detector

2. Detect and classify features using learning-based method
Automated Image Analysis for Robust Detection of Curbs
Project Leader Name & Functional Area
Wende Zhang (GM R&D / ECS Lab)
David Wettergreen (CMU)
Timing Date______________
Initial May, 2014
Midterm iMay, 2015
Final May, 2016
Detect and classify features using learning-based method
Resources
Total Material Cost[US$] k k k
Total Headcount (GM)
Total Headcount (CMU)
Description
Curbs are important cues on identifying the boundary of a roadway. Drivers understand an appropriate parking spot as defined by the curbs when reverse or parallel parking. Detecting curbs and providing information to assist drivers is an important task for active safety. Curb location is also crucial to autonomous parking systems.
Visual indications of curbs are widely various in the appearance. For example, under perspective imaging, projection of 3-dimensional curbs into 2-dimensional image plane distorts most of the curbs’ geometry properties, such as its angle, distance, and ratio of angles. Also, all curbs might be seen different because of age, wear, damage and lighting. Methods of detecting, localizing, and classifying curbs must address this diversity. This is to say, there is not a fixed template or set of templates that could be applied to reliably detect curbs through images.
Nevertheless visual appearance is how human drivers successfully detect curbs. Although physical structure can be sensed with some ranging sensors it not distinctive (two offset planes) or diagnostic of the roadway edge. Therefore we choose to pursue visual appearance.
This new project will develop an automated curb detection through :
Choosing appropriate features, learning those features to detect, and classifying the detected curbs
Utilizing the calibrated camera to fuse the 3D geometry information
One year development plan: detect, localize, and classify curbs using in-vehicle vision sensor with backward looking view with wide field of view
Motivation/Benefits
Identify the boundary of a road way in urban driving
Understand an appropriate parking spot as defined by the curbs when reverse or parallel parking
Deliverable / Technology Insertion into GM (What, When, Where)
Problem Definition: Survey of curbs
Data collection: Database of definite curb images and diverse curb images
Application: Detect curb features in perspective imagery
Experimental validation and performance analysis
Annual report
GM Confidential

3. Use Case Slides

4. Assumptions Color monocular camera Known camera motion
Known intrinsic parameters
Maximum speed dependent upon frame rate

5. Use Cases Parking lots Driveways Roadways Backward parking
Parallel parking
Driveways
Roadways
Single lane
Multi-lane

6. Parking lots
Scenario : Curbs exist behind of a vehicle; rear-view camera with wide field of view
Success : Detect and localize curbs on images; (Optional) estimate the distance from a vehicle to curbs CU R B CU R B GM

7. Parking lots
Scenario : Parking curbs exist behind of a vehicle; rear-view camera with wide field of view
Success : Detect and localize parking curbs on images; (Optional) estimate the distance from a vehicle to parking curbs CU R B CU R B GM

8. Driveways
Scenario : Curbs exist at the side of the entrance of driveway; front-view camera with wide field of view
Success : Detect and localize curbs on images and indicates driveways as traversable path GM
CURB
Driveways
CURB

9. Roadways
Scenario : Curbs exist at the side of the road; wide field of view camera
Success : Detect and localize curbs on images and indicates curbs as the non-traversable path and the boundary of road
CURB GM GM
CURB

10. Flow Chart Detection Edge Segmentation Texture Tracking
: Localize relevant curbs in each image
Edge
Segmentation
Texture
: Localize the detected curbs in remained
images
Tracking

11. Edge Detection
Distorted
Undistorted
Edge
Bird’s-eye view
Edge

12. Segmentation

13. Texture Classification

14. Development Plan Develop and test simple features
Train classifiers to detect and localize curbs
Evaluate classifier performance
Add complex features
Test quantify detection and localization performance
Train color classifiers to interpret appropriate parking spots
Motion Stereo to exploit 3D geometry

15. Scheme Extract Features Edge Detection Classification Tracking Filters
Edges
Intensity differences
Gradients
Geometric consideration
Horizontal
Long features
Thin features
Color
Texture
Curvature
Appearance based tracking

16. Data collection Using 180 degree field of view camera
Install underneath the side mirror, tilt 45 degree down to the ground
Sample images

17. Camera Calibration Wider field of view, more distortion
Camera calibration is necessary in order to find geometry constrains (e.g., edges…)
Using OCamCalib (Omnidirectional Camera Calibration Toolbox) to calibrate camera
Sample undistorted images

18. Shape Information
Edge detection
HOG feature

19. Extract Dominant Edges
Edge Detection
Extract Dominant Edges
Input Image at t
Undistorted Image
Edge Detection
Sequential
RANSAC

20. Extract Dominant Edges
Edge Detection
Extract Dominant Edges
Input Image at t
Undistorted Image
Edge Detection
Sequential
RANSAC

21. Extract Dominant Edges
Edge Detection
Extract Dominant Edges
Input Image at t
Undistorted Image
Edge Detection
Sequential
RANSAC
Two or Three parallel lines with small offsets are important cue for curbs

22. HOG feature

23. HOG feature

24. HOG feature
Input image
HOG map
Score map
Output image

25. Prerequisite
The maximum distance of ‘Curb Detection’ from a vehicle should be defined.
Given extrinsic parameters and the maximum distance, the followings can be estimated.
Different size of HOG model
Region of interest
short
medium
long

26. Geometry calculation Maximum detect distance CURB Local Area
m (9-12 feet) GM GM
Maximum detect distance
CURB

27. Examples Cadillac SRX Cadillac CTS
Cadillac CTS

28. Geometry calculation GM
Maximum detect distance = 2.1 meter
CURB

29. Image Sample with distance measure
The center of the camera is 1.05m from the ground.
The angle of the camera is 45 degree down from the horizontal.
ROI will be reduced. (Red transparent rectangle)
ROI will be changed based on the extrinsic parameter.

30. Lane Markings
Since lane markings have strong edges, we need to eliminate outputs from lane markings.
Parts of images which contain lane markings can be removed by detecting white blobs.

31. Result Video

32. Performance Measure Choose 300 testing images
Positive samples: images which contains full length of curbs
Negative samples: images without curbs
We consider curbs are detected when the horizontal length of the detected curbs are bigger than half of the horizontal length of image.
Since the size of image is 480 by 720, we consider curbs are detected and the sum of the length of the detected curbs are bigger than 360.

33. length of detected curb
Performance Measure
length of detected curb
> 0.5
total length of image
Groundtruth
Positive
Negative 80 13 24
183

34. Future Works Features of curb detection Front-view camera
Redundant information through multiple images
Include tracking system to recover false negatives
Continuity
Develop likelihood function to recover false negatives and remove false positives
Height
Front-view camera
Mount 180 degree field of view camera on the front bumper

35. Front-view Camera Configuration