End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners Simon Hecker, Dengxin Dai, and Luc Van Gool Computer Vision Lab, ETH Zurich Dataset & Code Experimental Results Motivation This work is funded by Toyota Motor Europe via the project TRACE-Zurich Dataset Dataset: 60 Hours of driving. Diverse weather conditions, day/night time and road situations (city, countryside, highway). Sensors: 8 x GoPro Hero 5 cameras @ 60Hz with 1080p and full 360 degree coverage. TomTom and OpenStreetMap route planning representations. Low-level CAN bus maneuver recording including steering wheel angle and vehicle speed. GPS & IMU odometry. Overall Performance Ablation Study Surround-view Cameras Previous methods only use a front-facing camera Surround-view cameras mimic side-view and rear-view mirrors, vital for learning driving models Map (Route Planner) is important for navigation, but not exploited in published work for driving models Method Implementation Details: Learning to drive via imitation learning. Predict human driving maneuver (steering wheel angle and vehicle speed) for a diverse set of maneuvers based on video data and route planning data. Learning driving models in a fully end-to-end manner: resnet34 as visual encoder, two fully connected regression networks for steering angle and vehicle speed prediction. Simultaneous learning of control via normalized maneuver labels. TomTom OSM Learning Details We propose learning driving models in real world settings with a `complete’ sensor configuration Using historical steering angle , historical speed , historical vehicle trajectory , video data , planned route to predict future speed , and steering angle Map helps significantly, esp. for steering prediction TomTom route planner outperforms OSM planner Multiple cameras help esp. for low-speed, city driving and driving at intersections