1. EFFICIENT ROAD MAPPING VIA INTERACTIVE IMAGE SEGMENTATION Presenter: Alexander Velizhev CMRT’09 ISPRS Workshop O. Barinova, R. Shapovalov, S. Sudakov, A. Velizhev, A. Konushin

2. Introduction Roadway monitoring systems are widely-used for supervising road pavement surface and repair planning

3. Problem statement Analysis road pavement only by video sequences

4. Problem statement (2) Object types: – Lane marking – Road patches and defects Solution requirements: – High object detection rate – Maximum automation

5. Problem statement (3) Source imageExpected result

6. Problem details Some real examples

7. Our algorithm outline 1. Video rectification 2. Image preprocessing 3. Image segmentation 4. Features calculation 5. Interactive classification Automatic offline stage Interactive online stage

8. Video rectification Image preprocessing Image segmentation Features calculation Interactive classification Video rectification Using of raw video has severe drawbacks: – Objects are represented with different spatial resolution on the same frame – Projective distortions – Elongated objects exceed the bounds of single frame

9. Video rectification (2) Image preprocessing Image segmentation Features calculation Interactive classification Video frames are converted to orthogonal projection and stitched to each other Video rectification

10. Image preprocessing Image segmentation Features calculation Interactive classification Video rectification Retinex transform Contrast adjustment Bilateral filter Image preprocessing Source image

11. Image segmentation Image preprocessing Image segmentation Features calculation Interactive classification Video rectification Main goal is representing all objects of interest as different segments We use the hierarchical version of mean shift algorithm

12. Features calculation Image preprocessing Image segmentation Features calculation Interactive classification Video rectification More than 100 various features are used for classification of segments Feature types: – Colour statistics (colour variance, Lab components’ percentiles,... ) – Shape statistics (elongation, orientation, area, …) – Difference with neighborhood of the segments

13. Interactive classification Image preprocessing Image segmentation Features calculation Interactive classification Video rectification

14. Interactive classification (2) Image preprocessing Image segmentation Features calculation Interactive classification Video rectification User manually marks object segments Learning of cascade of classifiers Automatic classification of the next road part User corrects classification results End Start

15. Cascade of classifiers Cascade of classifiers corresponds image segmentation levels We descend a hierarchy from large to small segments and reject segments that do not contain pixels of objects of interest Classifier training uses the data passed to a corresponding cascade layer by preceding version of cascade

16. Why do we use the cascade? To solve a problem of unbalanced classes To speed-up classification

17. Online learning We introduce an online version of the random forest algorithm Special class costing The algorithm’s code is a part of our open source “GML Balanced On-line Learning Toolkit ” – http://research.graphicon.ru/machine-learning/gml- balanced-on-line-learning-toolkit-2.html http://research.graphicon.ru/machine-learning/gml- balanced-on-line-learning-toolkit-2.html

18. Why do we use online learning? We don’t need to store all training database in memory Short learning time User actions immediately impact on the classification results

19. How to measure system efficiency ? We are modeling “ideal” user actions to measure the efficiency of the interactive classification Efficiency criterion: – a minimal number of mouse’s clicks for making correct classification

20. Results Source image Segmented image Analysis result

21. Results (2) Image part Clicks Manual classification Interactive classification

22. Results (3) Image part Error, %

23. Summary We present a tool for efficient interactive mapping of road defects and lane marking Intensive use of computer vision methods on different stages of our data processing workflow increases usability of the tool

24. Weak points Image segmentation errors can degrade classifier and true object bounds cannot be extracted Algorithm is not robust to user mistakes

25. Future work Ultimate goal: Development of the universal semantic segmentation system which can be used for object extraction from large class of images Nearest plan: Improving the quality of image segmentation by integration colour and range data

26. CMRT’09 ISPRS Workshop Efficient road mapping via interactive image segmentation O. Barinova obarinova@graphics.cs.msu.ru R. Shapovalov shapovalov@graphics.cs.msu.ru S. Sudakov ssudakov@graphics.cs.msu.ru A. Konushin ktosh@graphics.cs.msu.ru A. Velizhev avelizhev@graphics.cs.msu.ru