GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen, CMU Wende Zhang, GM Inna Stainvas, GM ContributorsJongHo Lee, CMU 1

GM-Carnegie Mellon Autonomous Driving CRL Schedule Curb locationSensor locationMethodologyDate On the sideBottom of the side mirrorVisual appearance~ Jan In frontThe front bumperGeometric structure~ May In frontThe front bumperAppearance + Geometry with production camera ~ Nov Deliverables Demonstration: In-vehicle curb detection Annual reports 2

GM-Carnegie Mellon Autonomous Driving CRL Objectives Develop reliable methods of detecting, localizing, and classifying features associated with curbs using in- vehicle, low-cost, monocular vision sensor Localize curbs within a range of 5 meters with 99% accuracy 3

GM-Carnegie Mellon Autonomous Driving CRL Approaches for Curb Detection Using Mono Camera Images Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) 4

GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) Approaches for Curb Detection Using Mono Camera Images 5

GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis 6

GM-Carnegie Mellon Autonomous Driving CRL Edge Detection 7

GM-Carnegie Mellon Autonomous Driving CRL Detect Curb Using HOG * Feature * Histogram of Oriented Gradients Input image HOG image Curb model 8

GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) Approaches for Curb Detection Using Mono Camera Images 9

GM-Carnegie Mellon Autonomous Driving CRL Geometry-based image analysis Input imageDepth image Ground plane estimation 3-D point cloud 10

GM-Carnegie Mellon Autonomous Driving CRL Plane Fitting 11

GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) Approaches for Curb Detection Using Mono Camera Images 12

GM-Carnegie Mellon Autonomous Driving CRL Appearance at t+1 Schematic Overview Input at t+1 Input at tAppearance at t Geometry Candidate regions Annotate curb region

GM-Carnegie Mellon Autonomous Driving CRL 14 Appearance -For each image, divide into m x n grids -m: image height / grid size (pixels) -n: image width / grid size (pixels) Image at t

GM-Carnegie Mellon Autonomous Driving CRL 15 Appearance -For each grid, classify among two classes (road, curb) -uniform Local Binary Pattern (LBP) Image at t

GM-Carnegie Mellon Autonomous Driving CRL 16 Local Binary Pattern threshold Binary: Decimal:

GM-Carnegie Mellon Autonomous Driving CRL 17 Appearance -Once all the grids of two images are classified, get the intersection of them Output at t+1Output at tIntersect

GM-Carnegie Mellon Autonomous Driving CRL Geometry -Green lines shows the vectors from the interesting points of image at time t (blue dots) to those of image at time t+1 (red dots) -Calculate the 3-D points using camera matrix

GM-Carnegie Mellon Autonomous Driving CRL Appearance + Geometry -For each grid, -Fit the best plane using 3-D points -Compute the normal vector -Determine the normal vector is a road surface or a curb surface

GM-Carnegie Mellon Autonomous Driving CRL Appearance + Geometry

GM-Carnegie Mellon Autonomous Driving CRL Extend Curb Region -If the appearances are similar, extend the curb region -Calculate the distance of LBPs using chi-square

GM-Carnegie Mellon Autonomous Driving CRL Extend Curb Region

GM-Carnegie Mellon Autonomous Driving CRL Track Curb Region -For the next frames, tracking the appearance of the curbs -When tracking, keep checking the geometry constraint to remove the false positives if exist Input at t+2Input at t+3Input at t+4

GM-Carnegie Mellon Autonomous Driving CRL Curved curb case Extend Curb RegionCombine Analyses

GM-Carnegie Mellon Autonomous Driving CRL Curb Detection using Production Camera Image size : 480 by 640 FOV: 180 degree

GM-Carnegie Mellon Autonomous Driving CRL Test Curb Detection Image Size: 640 x 480 (pixels) ROI: 640 x 160 (pixels) Size of grid: 20 x 20 (pixels) Number of grids: 32 x 8 Output of the appearance- based curb detection

GM-Carnegie Mellon Autonomous Driving CRL Test Curb Detection Remove outliers based on cluster size Find edges using Canny operator inside candidate region

GM-Carnegie Mellon Autonomous Driving CRL Test Curb Detection - Fit polynomials to each segments, and check lines for similar curvatures (blue), and remove high curvatures (red) Annotate curb region on the original input image

GM-Carnegie Mellon Autonomous Driving CRL Future Work Application: Operate real-time curb detection in vehicle ~ May

GM-Carnegie Mellon Autonomous Driving CRL Thank you Questions ? 30