1. REU - End to End Self Driving Car
Farzain + Kishan

2. BEWARE

3. Quick Recap
Create an end-to-end system that is able to autonomously maneuver a car through a variety of scenarios. Only allowed to use CAMERAS. Extremely complex situations.
IMPOSSIBLE.
Deep learning can help us out.

4. Related NVIDIA has released several reports on their work.
Works very well for simpler situations (staying in lane, not hitting cars in front)
Doesn’t work so well in more complex situations that are harder for a network to recognize (traffic signal, pedestrians)
This is partially because there isn’t enough training data and partially because the network may have a hard time differentiating between whats important and what isn’t.

5. Related Princeton paper recently proposed an FCN-LSTM
Wanted to use ONLY dash cam videos, nothing else, to predict future movement. Almost turned the problem into a text prediction problem.
Semantic segmentation label side training to “learn a meaningful feature representation”. Here they are implicitly feeding it segmentation information.
Doesn’t go directly from action -> input, “future motion action feasibility distribution”

6. Other

7. Udacity Labeled datasets.
Challenged programmers to create a deep network to predict steering angles.
First place used a 3D CNN + LSTM + RNN
Second place used Optical Flow + NVIDIA CNN
Top 5 models are open source.

8. Model
2D CNN
DECISION

9. Model
SEG LOSS
FCN
LSTM
DECISION

10. Model
3D CNN
LSTM
DECISION

11. Where do we go from here?

12. Model
2D CNN

13. Model
3D CNN

14. Model
3D CNN
LSTM

15. Model
3D CNN
LSTM
DECISION

16. Issues: output of two stream CNN, memory, usage of another model
3D CNN
RGB
3D CNN
LSTM
DECISION
Segmentation
Model Output
3D CNN
Issues: output of two stream CNN, memory, usage of another model
Pre-training?

17. Output as probability distribution
Model
DECISION
Output as probability distribution

18. Next 7 Days Code up/train NVIDIA base model in Keras (half-done)
Code up Udacity dataset reader script (half-done).
Get results of NVIDIA model to establish a solid base.
Create 2D CNN + LSTM and see results