Curb Detector

Detect and classify features using learning-based method Automated Image Analysis for Robust Detection of Curbs Project Leader Name & Functional Area Wende Zhang (GM R&D / ECS Lab) David Wettergreen (CMU) Timing Date______________ Initial May, 2014 Midterm iMay, 2015 Final May, 2016 Detect and classify features using learning-based method Resources 2013 2014 2015 Total Material Cost[US$] 100k 100k 100k Total Headcount (GM) 0.1 0.1 0.1 Total Headcount (CMU) 1 1 1 Description Curbs are important cues on identifying the boundary of a roadway. Drivers understand an appropriate parking spot as defined by the curbs when reverse or parallel parking. Detecting curbs and providing information to assist drivers is an important task for active safety. Curb location is also crucial to autonomous parking systems. Visual indications of curbs are widely various in the appearance. For example, under perspective imaging, projection of 3-dimensional curbs into 2-dimensional image plane distorts most of the curbs’ geometry properties, such as its angle, distance, and ratio of angles. Also, all curbs might be seen different because of age, wear, damage and lighting. Methods of detecting, localizing, and classifying curbs must address this diversity. This is to say, there is not a fixed template or set of templates that could be applied to reliably detect curbs through images. Nevertheless visual appearance is how human drivers successfully detect curbs. Although physical structure can be sensed with some ranging sensors it not distinctive (two offset planes) or diagnostic of the roadway edge. Therefore we choose to pursue visual appearance. This new project will develop an automated curb detection through : Choosing appropriate features, learning those features to detect, and classifying the detected curbs Utilizing the calibrated camera to fuse the 3D geometry information One year development plan: detect, localize, and classify curbs using in-vehicle vision sensor with backward looking view with wide field of view Motivation/Benefits Identify the boundary of a road way in urban driving Understand an appropriate parking spot as defined by the curbs when reverse or parallel parking Deliverable / Technology Insertion into GM (What, When, Where) Problem Definition: Survey of curbs Data collection: Database of definite curb images and diverse curb images Application: Detect curb features in perspective imagery Experimental validation and performance analysis Annual report GM Confidential

Use Case Slides

Assumptions Color monocular camera Known camera motion Known intrinsic parameters Maximum speed dependent upon frame rate

Use Cases Parking lots Driveways Roadways Backward parking Parallel parking Driveways Roadways Single lane Multi-lane

Parking lots Scenario : Curbs exist behind of a vehicle; rear-view camera with wide field of view Success : Detect and localize curbs on images; (Optional) estimate the distance from a vehicle to curbs CU R B CU R B GM

Parking lots Scenario : Parking curbs exist behind of a vehicle; rear-view camera with wide field of view Success : Detect and localize parking curbs on images; (Optional) estimate the distance from a vehicle to parking curbs CU R B CU R B GM

Driveways Scenario : Curbs exist at the side of the entrance of driveway; front-view camera with wide field of view Success : Detect and localize curbs on images and indicates driveways as traversable path GM CURB Driveways CURB

Roadways Scenario : Curbs exist at the side of the road; wide field of view camera Success : Detect and localize curbs on images and indicates curbs as the non-traversable path and the boundary of road CURB GM GM CURB

Flow Chart Detection Edge Segmentation Texture Tracking : Localize relevant curbs in each image Edge Segmentation Texture : Localize the detected curbs in remained images Tracking

Edge Detection Distorted Undistorted Edge Bird’s-eye view Edge

Segmentation

Texture Classification

Development Plan Develop and test simple features Train classifiers to detect and localize curbs Evaluate classifier performance Add complex features Test quantify detection and localization performance Train color classifiers to interpret appropriate parking spots Motion Stereo to exploit 3D geometry

Scheme Extract Features Edge Detection Classification Tracking Filters Edges Intensity differences Gradients Geometric consideration Horizontal Long features Thin features Color Texture Curvature Appearance based tracking

Data collection Using 180 degree field of view camera Install underneath the side mirror, tilt 45 degree down to the ground Sample images

Camera Calibration Wider field of view, more distortion Camera calibration is necessary in order to find geometry constrains (e.g., edges…) Using OCamCalib (Omnidirectional Camera Calibration Toolbox) to calibrate camera Sample undistorted images

Shape Information Edge detection HOG feature

Extract Dominant Edges Edge Detection Extract Dominant Edges Input Image at t Undistorted Image Edge Detection Sequential RANSAC

Extract Dominant Edges Edge Detection Extract Dominant Edges Input Image at t Undistorted Image Edge Detection Sequential RANSAC

Extract Dominant Edges Edge Detection Extract Dominant Edges Input Image at t Undistorted Image Edge Detection Sequential RANSAC Two or Three parallel lines with small offsets are important cue for curbs

HOG feature

HOG feature

HOG feature Input image HOG map Score map Output image

Prerequisite The maximum distance of ‘Curb Detection’ from a vehicle should be defined. Given extrinsic parameters and the maximum distance, the followings can be estimated. Different size of HOG model Region of interest short medium long

Geometry calculation Maximum detect distance CURB Local Area 2.7-3.6 m (9-12 feet) GM GM Maximum detect distance CURB

Examples Cadillac SRX Cadillac CTS http://www.cadillac.com/srx-luxury-crossover/features-specs/dimensions.html Cadillac CTS http://www.cadillac.com/cts-sport-sedan/features-specs/dimensions.html

Geometry calculation GM Maximum detect distance = 2.1 meter CURB

Image Sample with distance measure The center of the camera is 1.05m from the ground. The angle of the camera is 45 degree down from the horizontal. ROI will be reduced. (Red transparent rectangle) ROI will be changed based on the extrinsic parameter.

Lane Markings Since lane markings have strong edges, we need to eliminate outputs from lane markings. Parts of images which contain lane markings can be removed by detecting white blobs.

Result Video

Performance Measure Choose 300 testing images Positive samples: images which contains full length of curbs Negative samples: images without curbs We consider curbs are detected when the horizontal length of the detected curbs are bigger than half of the horizontal length of image. Since the size of image is 480 by 720, we consider curbs are detected and the sum of the length of the detected curbs are bigger than 360.

length of detected curb Performance Measure length of detected curb > 0.5 total length of image Groundtruth Positive Negative 80 13 24 183

Future Works Features of curb detection Front-view camera Redundant information through multiple images Include tracking system to recover false negatives Continuity Develop likelihood function to recover false negatives and remove false positives Height Front-view camera Mount 180 degree field of view camera on the front bumper

Front-view Camera Configuration