1. GM-Carnegie Mellon Autonomous Driving CRL 1 TitleRobust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen, CMU Wende Zhang, GM Inna Stainvas, GM ContributorsJongHo Lee, CMU

2. GM-Carnegie Mellon Autonomous Driving CRL 2 Schedule Problem definition: Survey of curb features2M Data collection: Database of definite curb images3M Application: Detect curb features in perspective imagery6M Design: Framework for real-time image analysis6M Data collection: Database of diverse curb images9M Experimental validation and performance analysis12M Implementation: In-vehicle classification system (detection + recognition) 12M

3. GM-Carnegie Mellon Autonomous Driving CRL 3 Deliverables Demonstration: Detect, localize, and classify curbs using in-vehicle vision sensor with backward looking view with wide field of view 1Y Annual Reports

4. GM-Carnegie Mellon Autonomous Driving CRL 4 Status as of Last Review Development –Channelizer localization –Orientation dependent color model –Evaluation of wide-FOV camera (product line camera)

5. GM-Carnegie Mellon Autonomous Driving CRL 5 Progress Since Last Review Development –Shape filtering –Kernel-based sign tracking –Preliminary results of curb detection

6. GM-Carnegie Mellon Autonomous Driving CRL 6 Curbs

7. 7 Damaged by Curbs

8. GM-Carnegie Mellon Autonomous Driving CRL 8 Objectives Develop reliable methods of detecting, localizing, and classifying sufficient set of indicative features associated with curbs using in-vehicle vision sensor with backward looking view Learn to recognize the curbs to know –Understand an appropriate parking spot as defined by the curbs when reverse or parallel parking –Identify the boundary of a road way in urban driving

9. GM-Carnegie Mellon Autonomous Driving CRL Approach Extend prior methods (multi-feature classification) Use visual methods of detection (color, texture, shape) Learn classification from large training set Utilize camera calibration information to understand 3D geometry and change view points Parallelize for multiple features 9

10. GM-Carnegie Mellon Autonomous Driving CRL Approach Extend prior methods (multi-feature classification) Use visual methods of detection (color, texture, shape) Learn classification from large training set Utilize camera calibration information to understand 3D geometry and change view points Parallelize for multiple features 10

11. GM-Carnegie Mellon Autonomous Driving CRL Framework of Highway Workzone Recognition Detection Tracking INPUT: A sequence of highway images OUTPUT: What is the road condition now? “Normal-highway” or ”Work-zone” : Localize relevant signs in each image : Localize the detected sign in remained images before it disappears 11 Inference : Infer the current driving region based on the results of sign classification so far Classification : Identify the types of detected signs

12. GM-Carnegie Mellon Autonomous Driving CRL Detection Pixel-wise Classification Connected Component Grouping Non- maximum Suppression 12 DetectionTracking Input Image at t Classification Log-polar Transform Sign Classifier Detection Classification TrackingDetection

13. GM-Carnegie Mellon Autonomous Driving CRL The sign image sub-regions from two consecutive image frames overlap each other. Small variations of their appearances and locations. 13 DetectionTrackingClassificationTracking

14. GM-Carnegie Mellon Autonomous Driving CRL Detection Pixel-wise Classification Connected Component Grouping Non- maximum Suppression 14 Input Image at t Classification Input Image at t+1 Tracking Log-polar Transform Sign Classifier Sub-region Sampling Candidate PDF Candidate Localization Choose Highest Score Target PDF for t+2 Target PDF for t+1 Tracking DetectionTrackingClassificationTracking

15. GM-Carnegie Mellon Autonomous Driving CRL 15 Performance of sign detection per frame Performance of sign classification Detection and TrackingDetection only Precision0.9820.930 Recall0.8850.875 Number of tracked signs512 Number of covered signs51 1.0 / 0.235 0.667 / 1.0 0.333 / 1.0 0.125 / 1.0 0.250 / 1.0 0.714 / 0.556 1.0 / 1.0 0.867 / 1.0 0.533 / 1.0 0.585 / 0.774 0.366 / 0.714 0.688 / 1.0 0.0625 / 1.0 0.667 / 0.857 0.5 / 1.0 0.478 / 0.786 0.087 / 0.25 0.5 / 1.0 0.0 / 0.0 0.231 / 1.0 0.0 / 0.0 N/A 0.625 / 1.0 0.0 / 0.0 0.667 / 1.0 0.647 / 1.0 0.444 / 1.0 0.222 / 1.0 N/A 0.25 / 0.4 0.125 / 0.333 -The first row represents precision / recall of ‘detection and tracking’ -The second row represents precision /recall of ‘detection only’

16. GM-Carnegie Mellon Autonomous Driving CRL Workzone Recognition Created framework for detection and tracking based on color and shape Achieved detection precision of 98.2% (with recall 88.5%) Trained classifiers from DOT uniform signage code and from real road imagery Achieved classification precision for signs of 96.5% (with recall 95.7%) Demonstrated on-road, real-time recognition of work zones Given high precision and recall, ‘Highway Workzone Recognition’ is ready for tech transfer. 16

17. GM-Carnegie Mellon Autonomous Driving CRL 17 Framework of Highway Work Zone RecognitionCurb Detection Highway Workzone Recognition Detection Tracking INPUT: A sequence of roadway images OUTPUT: Where is the curb? What does this curb mean? : Localize relevant curbs in each image : Localize the detected curbs in remained images Inference : Infer the detected curb based on the results of classification Classification : Identify the types of detected curbs

18. GM-Carnegie Mellon Autonomous Driving CRL Approach Extend prior methods (multi-feature classification) Use visual methods of detection (color, texture, shape) Learn classification from large training set Utilize camera calibration information to understand 3D geometry and change view points Parallelize for multiple features 18

19. GM-Carnegie Mellon Autonomous Driving CRL Concept Multiple cameras (rear, side, forward) Recognize important cues of curbs Estimate relative position from vehicle (Potential) Understand appropriate parking spots as defined by the curbs to assist self- parking system 19

20. GM-Carnegie Mellon Autonomous Driving CRL Potential Features 20 3D StructureColor Segmentation Edge Detection Texture Classification

21. GM-Carnegie Mellon Autonomous Driving CRL Structure from Motion 21

22. GM-Carnegie Mellon Autonomous Driving CRL Color Segmentation 22

23. GM-Carnegie Mellon Autonomous Driving CRL Edge Detection 23 UndistortedDistorted Bird’s-eye view Edge

24. GM-Carnegie Mellon Autonomous Driving CRL Texture Classification 24

25. GM-Carnegie Mellon Autonomous Driving CRL Complex Features Ground plane estimation –To exploit 3D geometry Edge continuity –To interpret curvy curbs Curb scale and geometry –To utilize the height of curb above the ground plane 25

26. GM-Carnegie Mellon Autonomous Driving CRL Color Classification 26 * Images from LADOT Yellow Curb ZonesBlue Curb Zones White Curb ZonesGreen Curb ZonesRed Curb Zones Normal Curb Zones

27. GM-Carnegie Mellon Autonomous Driving CRL Development Plan Develop and test simple features Train classifiers to detect and localize curbs Evaluate classifier performance Add complex features Test quantify detection and localization performance Train color classifiers to interpret appropriate parking spots 27

28. GM-Carnegie Mellon Autonomous Driving CRL Acknowledgements This project is sponsored by GM-CMU AD-CRL. Questions or Comments? 28