WalkSafe: A Pedestrian Safety App Presented by, Ramya Deepa Palle CS 541

Paper Overview: Summarizes a historical view on a conference about mobile sensors development Discuss about path toward cognitive phones Explains few phone applications based on mobile sensors Discuss problems and solution while analyzing Big Sensor Data

Sensors in mobile: Hardware (Apple iPhone 3GS as an example) Mobile phone or smartphone becomes the most important communication device in people lives. A smartphone is a mobile phone offering advanced capabilities, often with PC-like functionality Hardware (Apple iPhone 3GS as an example) CPU at 600MHz, 256MB of RAM 16GB or 32GB of flash ROM Wireless: 3G/2G, WiFi, Bluetooth Sensors: camera, acceleration, proximity, light Functionalities Communication News & Information Socializing Gaming Schedule Management etc.

Embedded sensors to phone as a sensor The view moved away from traditional embedded sensors to phone as a sensor.

Small history: In early 2007 Nokia released a phone with an embedded accelerometer. They did not mention about this in N95 because it was only there for video stabilization and photo orientation When Nokia’s peter Boda (the SensorPlanet leader) visited Dart-mouth college, he casually mentioned that the N(% included an embedded accelerometer. This, along with the phone’s GPS sensors, led researchers to study and develop new phone-based sensing applications, such as the CenceMe app.

CenceMe CenceMe, the first social sensing application running on mobile phones. Sensor application Accelerometer Microphone GPS Bluetooth Track user information Physical activity Sitting, walking, running or driving Social interaction Public conversation Share using social networking applications Ex. Facebook

CenceMe App Analysis Data pushed to servers for analysis Reconstruct status of users Phone Classifiers: Audio - Fourier transform to determine talking pattern Understand behavior, mood, and health Activity - Accelerometer data to determine if user is sitting, standing, walking, or running

CenceMe App Analysis Cont… Backend Classifiers: Conversation combines multiple audio primitives to determine whether someone is in a conversation, which may contain pauses Social Context Neighborhood conditions - are there CenceMe friends around? Social status - combination of classifiers Partying  Dancing  Mobility Mode GPS to determine if user is traveling in a vehicle or not Am I Hot Nerdy - often alone and spends time in locations such as libraries Party animal - often at parties and often near others Cultured - often visits theaters/museums Healthy - often walking, jogging, cycling, or at the gym Greeny - low environmental impact - not typically in a car and often walks, cycles, or runs

Smart phone sensing: Today’s top-end smartphones come with 1.4-GHz quad-core processors and a growing set of inexpensive yet powerful embedded sensors. They include: An accelerometer, a digital compass, a gyroscope, a GPS, quad microphones, dual cameras, near-field communication, a barometer, and light, proximity, and temperature sensors. They also have multiple radios for body, local, and wide area communications; 64 Gbytes of storage; and the touchscreen.

BeWell: A Mobile Health App The BeWell app continuously tracks user behaviors along three key health dimensions without requiring any user input — the user simply downloads the app and uses the phone as usual. Monitor and promote Physical and emotional well-being Persuasive feed back techniques Tracks along three dimensions Physical activity Social interaction Sleep duration Weighted scorecard Between 0 to 100 Ex. 8 Hrs/day sleep is score 100

BeWell : Health App Cont… Physical activity tracking Walking, stationary or running Daily metabolic equivalent of task value Centers for Disease Control and Prevention guidelines Sleep tracking Over 24 hrs period No phone interaction Daily bed time activity Recharging phone Keeping at same place longer time Guidelines by National Sleep Foundation

BeWell : Health App Cont… Social Isolation tracking Total speech time during day Phone microphone usage No recordings Social apps presence on the phone and usage Low score linking to depression Work as stand-alone and interwork with cloud

BeWell : Health App Cont… Persuasive feedback techniques Animated aquatic ecosystem as wallpaper Orange clown fish Reflects user’s activity Blue clown fish Reflects user’s social interaction Ocean’s ambient lighting Reflects user’s sleep duration Score history accessibility

Ambient Display:

BeWell+ Store data on clouds Share with friends on social networks Compare with others Targeted messages Challenges Low energy consumption More accurate classification Population Diversity Problem

BeWell : Big Sensor Data No raw data to server Upload features, inferences, scores and usability data 20MB data traffic per day per user Population Diversity Problem Walking patterns changes for a child and an adult Community Similarity Networks (CSN) Personalized model per user Measure similarity between people

WalkSafe : Pedestrian Safety App Research in social science Safety app for pedestrians crossing roads while on phone Back camera to detect vehicles Machine learning algos Difficulty Minimum battery utilization During active calls only Trained model Decision tree

WalkSafe: Pedestrian Safety App A mobile-phone user deep in conversation while crossing a street is generally more at risk than other pedestrians not engaged in such behavior. WalkSafe uses the smartphone’s back camera to detect vehicles approaching the user, alerting the user of any potentially unsafe situations.

Algorithms used: More specifically, WalkSafe uses machine-learning algorithms implemented on the phone to detect moving vehicles. It also exploits phone APIs to save energy by running the vehicle-detection algorithm only during active calls. The WalkSafe app offers real-time detection of the front and back views of cars, noting when a car is approaching or moving away from the user, respectively . It alerts the user using sound and phone vibration.

Image recognition: The core WalkSafe car-detection technology is based on image- recognition algorithms. Image recognition is a computationally intensive process that, if not carefully designed, can easily drain the smartphone’s computational resources and batteries.

Continued…… To address this, WalkSafe bases its vehicle recognition process on a model that’s first trained offine and then uploaded to the phone and used for online vehicle recognition, which runs automatically whenever there’s an ongoing phone call. WalkSafe activates the smartphone’s camera and captures video of the surroundings. Each video frame is preprocessed to compensate for the phone tilt and illumination variations, and is then analyzed by the decision tree model built during the offline training phase. If the decision tree detects a car in the picture, it triggers an alert to warn the user of possible danger.

NeuralPhone: A Brain-to-Smartphone Interface There’s a growing interest in new handsfree interfaces for smartphones based on voice and face recognition systems. They developed the EyePhone, which lets the user select and activate applications with the blink of an eye.1 We then wondered if a thought could also drive a smartphone application—and it turns out it can.

NeuralPhone : A brain to smartphone interface Hands free interfaces for smart phone Thought driving smart phone app EEG headsets Brain controlled address book dialing P300

Summary Towards cognitive phones Pushing intelligence towards phone Robust classification techniques Mobile sensors to improve life conditions

Queries?

If time permits: A small video based on how accelerometer inside the cell phone are made and How a Smartphone Knows Up from Down. http://www.youtube.com/watch?v=OjwLnAZoDFE