1. PixelLaser: Range scans from image segmentation Nicole Lesperance ’11 Michael Leece ’11 Steve Matsumoto ’12 Max Korbel ’13 Kenny Lei ’15 Zach Dodds ‘62 REU

2. Inspiration Horswill (polly) ’94 Saxena (rccar) ’05

3. Scans ? C. Plagemann et al., ICRA 2008 platform"omnicam" imageserrors...

4. Cheap! Expensive! Less information: 180 rays of distances More information: 640 columns of pixels, each of which we can get a range from But it’s harder (for a computer) to extract data from an image than from a laser range scan. Which is where we come in! Motivation

5. ClassifySegmentScans Maps! (in pictorial form!) obstacle ground (and localization) Overall Strategy

6. Training and Classifying hand-segmented imagefeatures, in kd-trees 194 191 211 3.2 25.6 4.1... 138 87 53 -1.14 8.6 1.4... filters

7. Nearest-neighbor classification RGB alone doesn’t work well…

8. Nearest-neighbors RGB + texture produces better segmentations

9. Classifier Variations Many attempts were made, but the overall accuracy for reasonable versions hovered around 95% 3-tree Classifier: used above, below, and line categories Genetic Algorithm: attempted to find the optimal weight values, but optimized for the set of images used to evolve the classifier. The weight-set did not generalize to other environments Different Environments: the classifier was tested in environments varying in difficulty

10. Overall Accuracy Results Regardless of the environment, the Classifier performs consistently

11. Hallway: Overall Accuracy 95.40 % Library Downstairs: Overall Accuracy 94.01 %

12. FLANN (Fast Library for Approximate Nearest Neighbors) C++ knn implementation with variable precision levels Decreased kd-tree lookup time by 2 orders of magnitude Allows for more complex real-time segmentation algorithms Allows the robot to autonomously wander in real time using only the segmenter Marius Muja: Our hero

13. Now Arriving at…Segmentation! Uses the information from the classifier to draw the line between the ground and obstacles Sends the information to the scanner to make laser range scans Classify → Segment → Scan → Map

14. Initial approaches (1)bottom-up (2)patch pairs (3)multiresolution

15. Transition Segmentation Uses transition strength rather than classification Assigns transition strength based on the certainty that the below patch is traversable and the top patch is not Takes the difference of the “above strength” of the top patch and the “below strength” of the bottom one Issues Misclassifications can result in incorrectly strong transitions

16. Transitions using a Line Tree The strongest transitions are found and the area closest matching the line category is chosen

17. Seam Carving Start at one end of the image and find the strongest transition Assign a cost to deviating from the line height in the previous column

18. Smoothing Certainty Locally Calculate certainties for all patches in the image Apply averaging filter on the certainties to suppress rogue misclassifications Look for the strongest transition in each column and draw line

19. Edge Detection Global image edge detection is not useful, but “snapping” to the nearest edge in the strongest transition area reduces segmentation error

20. Results Sprague LabSprague First FloorLibra Hallways Multi-resolution9.6716.2419.75 Smooth7.445.1022.11 Transition Line4.124.5214.27 Sprague LabSprague First FloorLibra Hallways Multi-resolution12.2816.0564.52 Smooth11.089.8130.15 Transition Line11.018.7222.78 Average Pixel Error per Column Average Distance Error per Column (inches)

21. Results - Moving Average Pixel Error per Column Sprague LabSprague First FloorLibra Hallways Multi-resolution8.8611.1314.10 Smooth3.627.9910.34 Transition Line2.796.0814.01 Sprague LabSprague First FloorLibra Hallways Multi-resolution27.3435.1055.34 Smooth17.6433.0159.18 Transition Line19.6836.4575.28 Average Distance Error per Column (inches)

22. From segmentation to scan pixel row number range row-to-range map scan segmentation

23. Scan Matching Match two consecutive laser scans and find the corresponding transformation Can be used to correct poor odometry

24. SLAM Map Building Combine all laser scans to buildmaps using CoreSLAM (simultaneous localization and mapping)

25. Monte Carlo Localization Create random list of possible locations of robot Monte Carlo Localization Cycle: Move: Move all points based on change in odometry Sense: Compare the laser scan from segmented image with laser scans at possible locations Resample: Redistribute points based on probability at each location

26. Monte Carlo Localization

27. Point and Click Navigation Given a known map, we want to be able to click on a location in the map and have the robot navigate there The current implementation works only if there is a straight, uninterrupted path between the current location and the destination (haven’t implemented a path planning algorithm) Things to think about include navigation around obstacles and odometry correction

28. Autonomous Wandering

29. Future Work Segmenting with a projected laser level line – possibly to be used for automated training as well Stabilizing the camera – keeping the horizon constant while moving is almost impossible Speeding up the wandering speed – a more stable camera would help with this Autonomous mapping

30. Questions?

31. Segmentation classification + confidencesegmentations

32. ../TrainingImages/Playspacepswo13Patches/00029/randomBelow/0009.png 194.2575 191.4525 211.4775 195.0 192.0 212.0 8.67070895314 8.76885076564 7.29105573631 211.4775 191.4525 194.2575 212.0 192.0 195.0 7.29105573631 8.76885076564 8.67070895314 3113.18473022 2918.61969259 194.707808243 -8.2951355554e-09 -2.99999999989 3.48871383089 -0.1821058002 1.05861940504 0.798119246226 0.142200402323 1.37643522494 1.51710154734 - 3.21108530262 25.6053670274 4.08979546594 0.20963020652 3.42407524192 7.98287776033 0.655198704147 1.40172476457 1.2862398764 -0.068793854887 7.97682628048 3.32687477389 -0.83219461218 2.58844742173 1.83432937157 -0.239667915222 1.22096756236 3.56803311807 0.0253655120168 0.415881369811 7.11907142389 - 0.829352050768 1.14166676605 4.1432245279 -0.33452013644 -0.0121915477333 6.30267114206 -0.714448599523 0.711384510758 3.38445323448 -0.566103623094 -0.00278409428233 5.92992221434 -0.932328715083 1.17018711051../TrainingImages/Playspacepswo13Patches/00026/randomabove/0007.png 138.1075 87.9525 53.4725 138.0 88.0 53.0 3.65184114523 3.44532200962 5.05512054752 53.4725 87.9525 138.1075 53.0 88.0 138.0 5.05512054752 3.44532200962 3.65184114523 1584.29001984 1485.27385943 99.0473083038 2.67691915514e-09 -2.99999999998 0.839568062542 -0.0913685033632 5.68745353063 0.334375851147 -0.0494906892805 5.40416976808 0.617058294109 -1.14642893693 8.69695694778 1.38147297404 -0.87936321568 4.10330225187 1.59491339286 -0.69926257399 0.679234808869 0.526708837532 -0.253691320988 7.92226252642 0.619482921372 -0.155302887182 1.51598656224 0.455410613623 0.896134364593 4.90251778762 1.44298148203 -1.32247289798 1.54652876227 1.82381743704 1.10446810418 0.850926709651 1.23981013329 -0.346508022218 -0.252958061091 1.53636914298 1.26906228604 1.09496630534 1.45689240135 1.4011501228 1.85701734954 1.68840166501 1.0053643485 0.471348780351 194 191 211 3.2 25.6 4.1 138 87 53 -1.14 8.6 1.4