Invisible Sensing of Vehicle Steering with Smartphones Dongyao Chen Kyong-Tak Cho, Sihui Han, Zhizhuo Jin, Kang G. Shin May 19, 2015

Camera + Various CV algorithms … Affordable, easy to use Camera helps detect lane markers for Detecting steering maneuvers http://www.phonearena.com/news/Nexus-5-likely-coming-with-an-OIS-camera_id47409 Affordable, easy to use + Various CV algorithms … https://www.youtube.com/watch?v=NJ8iXYZfUmA Autonomous vehicles use cameras to detect Lane markers Objects on road … Camera determines the functionalities of Performance is closely relate to the functionaility http://www.theguardian.com/technology/2013/jul/09/google-android-reader-why 1

CarSafe[MobiSys’ 13] app How’s Camera Doing? iOnRoad app https://play.google.com/store/apps/details?id=com.picitup.iOnRoad Citroën DS5 http://www.mycitroen.be/nl/citroen-ds5/innovatie/ CarSafe[MobiSys’ 13] app Performance of camera depends on the environment http://wallpaperswide.com/road_to_death_valley-wallpapers.html Smartphones with camera Built-in devices also use … Research works also mentioned … Our observation is that: the camera-based approaches 2

CarSafe[MobiSys’ 13] app How’s Camera Doing? iOnRoad app https://play.google.com/store/apps/details?id=com.picitup.iOnRoad Citroën DS5 http://www.mycitroen.be/nl/citroen-ds5/innovatie/ CarSafe[MobiSys’ 13] app Performance of camera depends on the environment Lighting Weather Pavement http://www.annarbor.com/news/residents-frustrated-with-lack-of-solutions-for-pothole-ridden-north-territorial-road/ Placement http://smartphones.wonderhowto.com/how-to/keep-your-smartphone-charged-during-power-outage-0140088/ However, this slides will show most of existing driving assistant systems still overly-rely on the usage of camera. Camera face down, if it cannot see, it definitely cannot work. 2

CarSafe[MobiSys’ 13] app How’s Camera Doing? iOnRoad app https://play.google.com/store/apps/details?id=com.picitup.iOnRoad Citroën DS5 LDW System http://www.mycitroen.be/nl/citroen-ds5/innovatie/ CarSafe[MobiSys’ 13] app How well a camera can see Bottleneck of application’s performance Real road condition Lighting Weather Pavement http://www.annarbor.com/news/residents-frustrated-with-lack-of-solutions-for-pothole-ridden-north-territorial-road/ Placement http://smartphones.wonderhowto.com/how-to/keep-your-smartphone-charged-during-power-outage-0140088/ Performance of driving assistant systems is undermined by a fundamental limitation of camera, which is. 2

Key Idea: Camera-free Sensing of Vehicle Steering Reliability -Resistant to changing environments Accuracy -Visual similarity will not effect the detection results Lightweight -Lower computational consumption The challenge is to find the right signature Which can directly gives us a boost of reliability, accuracy, and lightweight. “the right signature”: what we detect, how we detect 3

Differentiates maneuvers V-Sense Utilizes gyroscope Differentiates maneuvers Base on this idea, we propose V-Sense. V-Sense utilizes… To increase the accuracy… In the end… Works as a middleware 4

Sensing Steering with Gyroscope In left turns In lane changes Gyroscope in the smartphone Here is a logic jump 5

Understanding Vehicle Steering In left & right turns Detect bumps in gyroscope reading How to detect bumps? How to relate bump detection to different steering maneuvers? Lane changes Lane changes Gyroscope in the smartphone Here is a logic jump 5

Bump Detection Three-state algorithm: No bump → One bump→ Waiting for bump To determine if the following bump is source from the same maneuver, we detect the delay of next bump. T_bump = 1.5s; T_next_delay=3; Delta_s = 0.05; Delta_h = 0.07 ----- 会议笔记(4/1/15 16:11) ----- Peak detection. Spend more time on the analytical things. Show that the result is analytical reliable. ----- 会议笔记(4/28/15 11:34) ----- Dont need hyphend 6

Increase Accuracy: Steering & Curvy Road Lane change S-shape road Turn L-shape road We also need to increase the accuracy, because, the driving trajectory of steering and on curvy road could be similar. Horizontal displacement: WCURVY >WLANE , WCURVY >WTURN Integrate horizontal displacement at each time step 7

Calculate Horizontal Displacement W4 W3 W2 W1 Ts 2Ts 3Ts 4Ts Velocity v1 v2 v3 v4 θ1 θ2 θ3 θ4 Time Ts 2Ts 3Ts 4Ts Velocity v1 v2 v3 v4 θ1 θ2 θ3 θ4 Time Ts 2Ts 3Ts 4Ts Velocity v1 v2 v3 v4 Time The whole lane change process can be separated into multiple time slots; At each time slot, combine the changing angle with the previous result; We Figure: Horizontal displacement of lane change 8

Detect Horizontal Displacement on Curvy Road Let’s see the performance of this algorithm in the real world 9 Route image adapted from Google map: https://www.google.com/maps/place/2345+Lancashire+Dr,+Ann+Arbor,+MI+48105/@42.3099558,-83.7171271,17z/data=!3m1!4b1!4m2!3m1!1s0x883cac2d8254dfbb:0x2df93ff4aa13179b

Horizontal Displacement on Curvy Road Show the driving direction 10

Horizontal Displacement on Curvy Road Show the driving direction 10

Evaluation Setups Tested V-Sense on Samsung Galaxy S4 and S3 5 volunteers, 2 cars 40+h test 11

Reliability and Accuracy Resilient to different placements: Success Rate ----- 会议笔记(4/1/15 16:11) ----- 1. what could happen if misclassification happens. ----- 会议笔记(4/28/15 10:12) ----- success ratio for correctly detect the maneuvers 12

Computational Costs 13 ----- 会议笔记(4/1/15 15:57) ----- show bars. S4 is even better than S5. 13

Comparison with Other Applications ~1,000,000 installs ~50,000 installs ~50,000 installs 110 ratings Success Rate ----- 会议笔记(4/1/15 15:57) ----- Divde with ground truth. ----- 会议笔记(4/1/15 16:00) ----- parameters are different from previous slides ----- 会议笔记(4/1/15 16:11) ----- mention the evaluation condition. Specify the evaluation scenario. Specify highway, and local road. If you only drive on highway, the performance could be Success rate Caption: compare of lane change detection Figure: Success rate of detection of lane change in urban area 14

When Steering your Car... 16 Before we show the first application. Image: http://www.wakeforesttoday.com/voices-letters/note-to-self/4569-turn-signals-people.html#.VUPwONpViko

Application I: Detection of Careless Steering ----- 会议笔记(4/1/15 16:11) ----- 1. Specify the detail; 2. Change to careless. 17 Car icon: http://publicdomainvectors.org/en/free-clipart/Top-view-car-vector/2260.html

Turn Signal Detection Matched filter results > ground truth? Explain SAME Size!!! ----- 会议笔记(4/28/15 11:29) ----- Should have captions for all figures Use real image Add CAPTIONS! 18

Application I: Detection of Careless Steering ！ ----- 会议笔记(4/1/15 16:11) ----- 1. Specify the detail; 2. Change to careless. 19

Application I: Detection of Careless Steering ----- 会议笔记(4/1/15 16:11) ----- 1. Specify the detail; 2. Change to careless. 19

Application II: Fine-grained Lane Guidance https://idroid.us/google-maps-app-for-ios-updated-with-optimized-for-ios-8.html GPS is unstable and its accuracy varies with environment GPS Accuracy http://365stagandhenweekends.co.uk/blog/ 20

Application II: Fine-grained Lane Guidance Lane tracking at intersection: Left to Right Right to Left ----- 会议笔记(4/28/15 11:50) ----- ADAPTIVE TO DIFFEREWNT CASES 21

Conclusion V Two proof-of-concept apps A reliable, accurate and light weighted middleware for detecting steering maneuvers Two proof-of-concept apps Careless steering warning Fine-grained lane guidance ----- 会议笔记(4/1/15 15:57) ----- ssss! 22

Q&A Thank you! 23 ----- 会议笔记(4/1/15 15:57) ----- include email. chendy@umich.edu 23