1. Self-Supervised Segmentation of River Scenes Supreeth Achar *, Bharath Sankaran ‡, Stephen Nuske *, Sebastian Scherer *, Sanjiv Singh * * ‡

2. Task Detect extent of river in monocular imagery Input Image Extent of River 1

3. Motivation: River Mapping Explore and map river, canopy with a low flying micro air vehicle Water segmentation used for guidance and mapping 2

4. Challenges 3

5. Appearance varies between environments Appearance varies within the same environment due to changing weather, wind, seasons Appearance of rivers changes so much that a single model or classifier isn't going to work well under different conditions and at different places 4

6. Intuition: Exploit Structure Horizon line is known and river can't appear above horizon Regions below horizon have a higher probability of being part of shore if they are visually similar to above horizon region river if they are visually dissimilar to above horizon region 5

7. Self Supervised Algorithm Overview Step 1: Features computed for each patch in input image Step 2: Extract and discard the sky region, find horizon line Step 3: Calculate for each patch in the image Step 4: Automatically select training patches, build classifier Step 5: Classify all patches in the image, use MRF to enforce spatial coherence Output: detected extent of river 6

8. 1. Compute Features Image broken into square patches with a grid Trade off : speed v/s fineness of segmentation Feature descriptor is computed for each patch 7

9. 2. Estimate Sky and Horizon Sky can be confused with shiny parts of water surface Segmented with pretrained classifier, ignored in future steps Horizon line can be fitted to image using IMU measurements 8

10. 9 3. Calculating

11. 9

12. 9

13. 9

14. Chow-Liu trees built to model and Trees built with from data in current image These are used to compute for each patch 10

15. 4. Train the Classifier Patches with are considered part of river All above horizon patches are used as non-river examples Linear SVM trained on these automatically selected patches 11

16. 5. Detect River SVM is used to classify the entire image into river / non river SVM output is smoothened spatially with a Markov Random Field 12

17. 5. Detect River SVM is used to classify the entire image into river / non river SVM output is smoothened spatially with a Markov Random Field 13

18. Datasets 4 datasets of 120-150 images each with manually labeled ground truth Datasets were collected from boats moving along river NameLocationTimeSeasonWeather Allegheny DayHerrs IslandAfternoonSummerOvercast Allegheny DuskHerrs IslandEveningSummerLight rain Allegheny FallHerrs IslandAfternoonFallSunny YoughioghenyPerryopolisNoonSummerSunny 14

19. Sample Results River detection on an image from the “Allegheny - Dusk” dataset. Top left is input, top right is probability of patch being non-river based on appearance (warmer colors map to higher probabilities). The bottom left image shows the automatically extracted training examples, the bottom right image shows the final river detection result. 15

20. Sample Results River detection on an image from the “Allegheny - Day” dataset. Top left is input, top right is probability of patch being non-river based on appearance (warmer colors map to higher probabilities). The bottom left image shows the automatically extracted training examples, the bottom right image shows the final river detection result. 16

21. Results Error Rate (Percentage of Image Area Missclassified) Supervised Self Supervised Allghy. Day Allghy. Dusk Allghy. Fall Yough.All but Self All Allghy. Day3.803.724.924.243.813.422.75 Allghy. Dusk8.323.1010.847.919.594.404.15 Allghy. Fall10.857.673.056.3410.574.722.73 Yough.5.134.295.833.113.873.233.22 Test Set Supervised approach performs poorly on previously unseen data Our self supervised method performs well on new data 17

22. Results Error Rate (Percentage of Image Area Missclassified) Supervised Self Supervised Allghy. Day Allghy. Dusk Allghy. Fall Yough.All but Self All Allghy. Day3.803.724.924.243.813.422.75 Allghy. Dusk8.323.1010.847.919.594.404.15 Allghy. Fall10.857.673.056.3410.574.722.73 Yough.5.134.295.833.113.873.233.22 Test Set Supervised approach performs poorly on previously unseen data Our self supervised method performs well on new data 17

23. Results Error Rate (Percentage of Image Area Missclassified) Supervised Self Supervised Allghy. Day Allghy. Dusk Allghy. Fall Yough.All but Self All Allghy. Day3.803.724.924.243.813.422.75 Allghy. Dusk8.323.1010.847.919.594.404.15 Allghy. Fall10.857.673.056.3410.574.722.73 Yough.5.134.295.833.113.873.233.22 Test Set Supervised approach performs poorly on previously unseen data Our self supervised method performs well on new data 17

24. Results Error Rate (Percentage of Image Area Missclassified) Supervised Self Supervised Allghy. Day Allghy. Dusk Allghy. Fall Yough.All but Self All Allghy. Day3.803.724.924.243.813.422.75 Allghy. Dusk8.323.1010.847.919.594.404.15 Allghy. Fall10.857.673.056.3410.574.722.73 Yough.5.134.295.833.113.873.233.22 Test Set Supervised approach performs poorly on previously unseen data Our self supervised method performs well on new data 17

25. Video #1 Self Supervised River Segmentation on the Allegheny Fall sequence 18

26. Ongoing Work Detect Reflections Memory Adaptive models Find novel objects 19

27. Video #2 Self Supervised River Segmentation on the Allegheny Day sequence 20

28. Conclusions River appearance changes, segmenting is hard Difficult to build a single classifier that works everywhere By exploiting scene structure, we can make a system adapt and train itself 21

29. Thank You 22

30. Backup Slides

31. Related Work Geometry from Appearance Road Following Water Hazard Detection 5

32. Execution Time Single threaded C++ implementation takes 350ms to segment a 640x360 image This rate is fast enough for vehicle guidance C++ Running Time Profile StepTime (s) Feature Extraction0.25s Appearance Modeling0.02s SVM update0.03s Classification0.05s Total0.35s