1. Summary Presentation

2. Week 1 - 3 Papers Setting up RefineNet
Sky Segmentation in the Wild (Mihail et al 2016), Sky is Not the Limit (Tsai et al 2016)
Two-Class Weather Classification, Lu et al (2014)
Superparsing: Scalable Nonparametric Image Parsing with Superpixels, Tighe et al (2010)
Geometric Context from a Single Imagee Hoiem et al (2005)
Pyramid Scene Parsing Network, Zhao et al (2017)
RefinNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation, Lin et al (2016)
And more…
Setting up RefineNet
Running dataset through Refinenet’s Res101 model trained on Cityscapes dataset

3. Week 4 - Results Refinenet’s Cityscapes model
45 Cameras ~ 60,000 images
mIOU across all cameras %
Next: Finetuning Refinenet
Training it on a small dataset to make minor corrections
Caffe - PSPNet in the interest of time we’re focusing on Refinenet because we were able to set that up Refinenet - had troubles with the GPU so our main focus was just running the models and then fix the GPU (now fixed)
Currently we’re working with a small subset to retrain the cityscapes model to focus on sky segmentation
After that we’ll be training the full dataset on it 5 times and averaging the error
If we still have time to consider other models or run into more issues we will be making a CNN of about 5 layers and then fully connected layers in order to connect the output of the images it to an LSTM trained on weather statistics for a possible application in weather prediction given a sequence of images

4. Week 5 Papers Error handling and Results
Dev et al - Short-term Prediction of Localized Cloud Motion Using Ground-Based Sky Imagers
Chu et al - Camera as weather sensor: Estimating weather information from single images
Error handling and Results
One camera (4679) cannot be used by the model for an unknown reason
Dataset has been split into train(28 cameras)-validation(4 cameras)- test(13 cameras)
Preliminary results shown to the right
mIOU of one camera (858) is 54.09% after 10 epochs
Dataset has been shuffled to finetune the model again (same train-val-test amount but different cameras) for comparison

5. Week 6 Papers Error handling…
Sky Region Detection in a Single Image for Autonomous Ground Robot Navigation - Shen et al
Horizon Detection Using Machine Learning Techniques - Fefiltyev et al
Convolution LSTM Network: A Machine Learning Approach for Precipitation Nowcasting - Shi et al
Error handling…
Visual and qualitative data doesn’t correlate
Camera
Cityscapes
Refinenet
3395
49.25%
59.96%
8733
54.01%
58.80%
17218
31.70%
49.42%
1093
62.56%
39.81%
4679
58.75%
28.80%

6. Split 1 - Visual

7. Week 7 More miscorrelation upon seeing the rest of the results…
Theories included
Training from scratch (partially yes)
Not a good learning rate
Something is wrong (ding ding! - we spent well over a week on this)

8. Week 8 So what was wrong (this time)?
RefineNet has a very picky requirement in regards to setup, training, and more.
To properly use RefineNet, the segmentation maps (ground truth) must be of uint8 type, be 2 dimensional, and it’s segmentation labels must match the indices of the labels dictated in another file…
While replicating the ground truth for each camera for each image and replacing the filename with it’s corresponding original image, the segmentation map gained another dimension, and the labels became color values and not the labels they were originally...

9. Week 8 - cont.
In the interest of time and making sure we were doing things correctly
We resized all the images to be within the ratio of 320x240
Selected a much smaller subset for proof of concept
Got better results! Really good ones! That match!
Camera
162
623
17244
mIOU
96.21%
95.31%
82.61%
MCR
2.74%
2.97%
4.48%

10. Results - Cam162

11. Week
Using the whole dataset but at the same size (within 320x240) and the original data-split
Finetuned on the model trained on Cityscapes
Trained on our dataset initializing from ImageNet
Shuffled the cameras to repeat the above actions to compare to the first instance
Improved on the off-the-shelf method and did better than Mihail et al (our project inspiration)
(Finetune)
Cityscapes
Split 1
mIOU
49.31%
79.48%
MCR
17.12%
5.17%
(Finetune)
Cityscapes
Split 2
mIOU
46.31%
72.18%
MCR
18%
7.48%

12. Results - Cam Cam5020

13. Results - Cam4795

14. Current Results Split 1 Cityscapes Finetune ImageNet mIOU 49.31%
79.48%
87.07%
MCR
17.12%
5.17%
5.08
Split 2
Cityscapes
Finetune
ImageNet
mIOU
46.31%
72.18%
73.84%
MCR
18%
7.48% 7%