Introduction to Mobile Sensing with Smartphones Uichin Lee KSE 801 Nov. 16, 2011

iPhone 4 - Sensors

Applications Transportation – Traffic conditions (MIT VTrack, Nokia/Berkeley Mobile Millennium) Social Networking – Sensing presence (Dartmouth CenceMe) Environmental Monitoring – Measuring pollution (UCLA PIER) Health and Well Being – Promoting personal fitness (UbiFit Garden)

Citysense MacroSense CabSense

Eco-system Players Multiple vendors – Apple AppStore – Android Market – Microsoft Mobile Marketplace Developers – Startups – Academia – Small Research laboratories – Individuals Critical mass of users

Scale of Mobile Sensing

Sensing Paradigm Participatory: active sensor data collection by users – Example: managing garbage cans by taking photos – Advantages: supports complex operations – Challenges: Quality of data is dependent on participants Opportunistic: automated sensor data collection – Example: collecting location traces from users – Advantages: lowers burden placed on the user – Challenges: Technically hard to build – people underutilized Phone context problem (dynamic environments)

SENSE LEARN INFORM, SHARE, PERSUASION Mobile Sensing Architecture Mobile Computing Cloud

Sense Programmability – Managing smartphone sensors with system APIs – Challenges: fine-grained control of sensors, portability Continuous sensing – Resource demanding (e.g., computation, battery) – Energy efficient algorithms – Trade-off between accuracy and energy consumption Phone context – Dynamic environments affect sensor data quality – Some solutions: Collaborative multi-phone inference Admission controls for removing noisy data SENSE LEARN INFORM, SHARE, PERSUASION

Learn Integrating sensor data – Data mining and statistical analysis Learning algorithms – Supervised: data are hand-labeled (e.g., cooking, driving) – Semi-supervised: some of the data are labeled – Unsupervised: none of the data are labeled Human behavior and context modeling Activity classification Mobility pattern analysis (place logging) Noise mapping in urban environments SENSE LEARN INFORM, SHARE, PERSUASION

Learn: Scaling Models Scaling model to everyday uses – Dynamic environments; personal differences – Large scale deployment (e.g., millions of people) Models must be adaptive and incorporate people into the process If possible, exploit social networks (community guided learning) to improve data classification and solutions Challenges: – Lack of common machine learning toolkits for smartphones – Lack of large-scale public data sets – Lack of public repository for sharing data sets, code, and tools SENSE LEARN INFORM, SHARE, PERSUASION

Inform, Share, Persuasion Sharing – Data visualization, community awareness, and social networks Personalized services – Profile user preferences, recommendations, persuasion Persuasive technology – systems that provide tailored feedback with the goal of changing user’s behavior – Motivation to change human behavior (e.g., healthcare, environmental awareness) – Methods: games, competitions, goal setting – Interdisciplinary research combining behavioral and social psychology with computer science SENSE LEARN INFORM, SHARE, PERSUASION

Privacy Issues Respecting the privacy of the user is the most fundamental responsibility of a mobile sensing system Current solutions – Cryptography – Privacy-preserving data mining – Processing data locally versus cloud services – Group sensing applications is based on user membership and/or trust relationships

Privacy Issues Reconstruction type attacks – Reverse engineering collected data to obtain invasive information Second-hand smoke problem – How can the privacy of third parties be effectively protected when other people wearing sensors are nearby? – How can mismatched privacy policies be managed when two different people are close enough to each other for their sensors to collect information? Stronger techniques for protecting people’s privacy are needed

Understanding Smartphone Sensors: accelerometer, compass, gyroscope, location, etc

Smart Phone/Pad Sensors Nexus OneNexus SiPhone4 Samsung Galaxy S HTC Incredible Galaxy Tab/ iPad2 Accelerometer OOOOOO Magnetometer OOOOOO Gyroscope OO?O Light OOOOOO Proximity OOOOOO Camera OOOOOO Voice OOOOOO GPS OOOOOO

Accelerometer Mass on spring GravityFree FallLinear AccelerationLinear Acceleration plus gravity 1g = 9.8m/s 2 -1g 1g

Compass Magnetic field sensor (magnetometer) Z X Y X Y Z 3-Axis Compass? Magnetic inclination Horizontal Gravity Magnetic field vector Magnetic declination Magnetic north Geographic north

Orientation: Why Both Sensors? Two vectors are required to fix its orientation! (i.e., gravity and magnetic field vectors) Tutorial:

Gyroscope Angular velocity sensor – Coriolis effect – “fictitious force” that acts upon a freely moving object as observed from a rotating frame of reference

Accelerometer vs. Gyroscope Accelerometer – Senses linear movement, but worse rotations, good for tilt detection, – Does not know difference between gravity and linear movement Shaking, jitter can be filtered out, but the delay is added Gyroscope – Measure all types of rotations – Not movement – Does not amplify hand jitter A+G = both rotation and movement tracking possible

Other Sensors Light: Ambient light level in SI lux units Proximity: distance measured in centimeters (sometimes binary near-far) Temperature Pressure Barometer, etc…

Global Positioning System (GPS) 27 satellite constellation Powered by solar energy Originally developed for US military Each carries a 4 rubidium atomic clocks – locally averaged to maintain accuracy – updated daily by US Air Force Ground control – Satellites are precisely synchronized with each other The orbits are arranged so that at any time, anywhere on Earth, there are at least four satellites "visible" in the sky. A GPS receiver's job is to locate three or more of these satellites, figure out the distance to each, and use this information to deduce its own location. This operation is based on a mathematical principle called tri-lateration

Tri-lateration: GPS, Cell-tower Imagine you are somewhere in Korea and you are TOTALLY lost -- for whatever reason, you have absolutely no clue where you are. You find a friendly local and ask, "Where am I?" He says, "You are 70 km miles from 대전 “ You ask somebody else where you are, and she says, "You are 60 km from 대구 " Now you have two circles that intersect. You now know that you must be at one of these two intersection points. If a third person tells you that you are 100 km from 광주, you can eliminate one of the possibilities. You now know exactly where you are 70km 100km 60km

Assisted-GPS: GPS + Network Reduce satellite search space by focusing on where the signal is expected to be Other assistance data from cellular nets – Time sync – Frequency – Visible satellites – Local oscillator, etc.. MS-based vs. assisted GPS

GPS Errors (in Smartphones) In-vehicle signal attenuation Smartphone’s inferior antenna (worse!) – PND uses Spiral helix; Microstrip antenna vs. Galaxy S (single wire) low GPS reading under high speed environments – 4800bps (600 B/s) – research/papers/MitigatingGPS Error-UWTechReport08.pdf research/papers/MitigatingGPS Error-UWTechReport08.pdf Galaxy S GPS Antenna PND GPS antennas V.S.

Network Positioning Method Cell-tower localization with tri-lateration Cell-tower

Wi-Fi Positioning System Fingerprinting (e.g., RADAR, Skyhook) Training phase (building a fingerprint table): for each location, collect signal strength samples from towers, and keep the average for each location Positioning phase: – Calculate the distance in signal strength space between the measured signal strength and the fingerprint DB – Select k fingers with the smallest distance, and use arithmetic average as the estimated location RSSI (x, y, z) = (-20, -10, -15) (-15, -12, 18) …………  L1=avg(x, y, z) = (xx, yy, zz) RSSI (x, y, z) = (-21, -40, -18) (-16, -42, 12) …………  L2=avg(x, y, z) = (xx’, yy’, zz’) RSSI: Received Signal Strength Indicator

Mobile Sensing Applications

Acceleration data gathering from vehicles (geo-tagged) Simple data processing to detect a pothole, and statistical processing (clustering) for accurate detection Pothole Patrol Smooth Road Pothole The Pothole Patrol: Using a Mobile Sensor Network for Road Surface Monitoring, Eriksson et al, MobiSys, 2008

Community Awareness: Health and Wellness Personal environmental impact report (PIER) on “health and wellness” Participants use mobile phones to gather location data and web services to aggregate and interpret the assembled information (e.g., air pollution, CO2 emission, fast food exposure) CO2 emissions Fast food exposure Air pollution exposure (PM 2.5) Existing Infrastructure Annotation /Inferences Scientific Models Activity Classification e.g., staying, walking, driving GIS Data Annotation e.g. weather, traffic Impact and Exposure Calculation Data Aggregation Tracklog format School,hospital,fast food restaurant locations Weather, traffic data User profile "Sensing Pollution without Pollution-Sensors” PEIR, the Personal Environmental Impact Report as a Platform for Participatory Sensing Systems Research, Mun et al., Mobisys 2009

SoundSense Admission Control Acoustic Features Decision Tree Classifier Markov Model Recognizer Ambient Sound Learning Voice Analysis Music Analysis

Social Sensing with Twitter some users posts “earthquake right now!!” some earthquake sensors responses positive value ・・・ tweets Probabilistic model Classifier observation by sensors observation by twitter users target event target object Probabilistic model values Event detection from twitter Object detection in ubiquitous environments ・・・ search and classify them into positive class detect an earthquake earthquake occurrence Earthquake shakes Twitter users: real-time event detection by social sensors, Takeshi et al, WWW 2010

Summary SENSE LEARN INFORM, SHARE, PERSUASION Mobile Sensing Architecture Mobile Computing Cloud