REU - End to End Self Driving Car Farzain + Kishan
BEWARE
Quick Recap Create an end-to-end system that is able to autonomously maneuver a car through a variety of scenarios. Only allowed to use CAMERAS. Extremely complex situations. IMPOSSIBLE. Deep learning can help us out.
Related NVIDIA has released several reports on their work. Works very well for simpler situations (staying in lane, not hitting cars in front) Doesn’t work so well in more complex situations that are harder for a network to recognize (traffic signal, pedestrians) This is partially because there isn’t enough training data and partially because the network may have a hard time differentiating between whats important and what isn’t.
Related Princeton paper recently proposed an FCN-LSTM Wanted to use ONLY dash cam videos, nothing else, to predict future movement. Almost turned the problem into a text prediction problem. Semantic segmentation label side training to “learn a meaningful feature representation”. Here they are implicitly feeding it segmentation information. Doesn’t go directly from action -> input, “future motion action feasibility distribution”
Other
Udacity Labeled datasets. Challenged programmers to create a deep network to predict steering angles. First place used a 3D CNN + LSTM + RNN Second place used Optical Flow + NVIDIA CNN Top 5 models are open source.
Model 2D CNN DECISION
Model SEG LOSS FCN LSTM DECISION
Model 3D CNN LSTM DECISION
Where do we go from here?
Model 2D CNN
Model 3D CNN
Model 3D CNN LSTM
Model 3D CNN LSTM DECISION
Issues: output of two stream CNN, memory, usage of another model 3D CNN RGB 3D CNN LSTM DECISION Segmentation Model Output 3D CNN Issues: output of two stream CNN, memory, usage of another model Pre-training?
Output as probability distribution Model DECISION Output as probability distribution
Next 7 Days Code up/train NVIDIA base model in Keras (half-done) Code up Udacity dataset reader script (half-done). Get results of NVIDIA model to establish a solid base. Create 2D CNN + LSTM and see results