SurroundSense: Mobile Phone Localization via Ambience Fingerprinting Martin Azizyan Duke University Ionut Constandache Duke University Romit Roy Choudhury Duke University

Abstract Mobile computing applications center around user’s location Term- Physical Location (coordinates Latitude and Longitude) Term- Logical Location (like Starbucks or McDonalds) Lots of Research available for physical location Few attempts recognizing logical location

Abstract Ambient sound, light, and color from phone’s camera and microphone Accelerometers for user-motion Adjacent stores can be separated logically They propose SurroundSense, a mobile phone based system explores ambience fingerprinting 51 different stores - average accuracy of 87% when all sensing modalities are employed

Introduction Starbucks -coffee machines and microwaves Restaurants -forks and spoons clinking Target – red colors Panera Breads- yellow colors Floors with carpets, ceramic tiles, or wooden strips Bars -dim yellow lights BlockBuster - bright white light

Introduction Wal-Mart - walking up and down aisles Barnes and Noble -relaxed stroll with long pauses Restaurants - short queuing followed by a long duration of sitting May not be unique based on any one attribute, the combination of all exhibits diversity

SurroundSense Architecture

Mobile phone user visits an unknown store. The phone senses the ambience Values forwarded to the fingerprinting factory Types of sensor data identified Phone’s (GSM-based) physical coordinates Geographical database Fingerprint database and Fingerprint matching The matching module -best match for test fingerprint

System Design Fingerprinting Sound Fingerprinting Motion using Accelerometers Fingerprinting Color/Light using Cameras Fingerprinting Wi-Fi Fingerprint Matching

Fingerprinting Sound Recorded ambient sound for one Time domain, a simple fingerprinting scheme based on signal amplitude Acoustic fingerprints - computed the pair-wise distances. Use sound only as a filter The output is fed to the accelerometer filter

Sound fingerprints from 3 adjacent stores

Fingerprinting Motion using Accelerometers Human movements in a location Restaurants -stationary for long durations Grocery store - more mobile Accelerometer readings - stationary vs in motion Vector machines (SVM), a popular data classification tool User movement is prone to fluctuation Clothing store - browse long time or purchase in haste Accelerometers as a filtering mechanism too

Sample Accelerometers Traces

Fingerprinting Color/Light using Cameras The wall and floor colors contribute to theme Use automatically-taken phone pictures Only floor-facing pictures are used Color/light extraction Why picture of the floor? Colors of carpets, tiles, marble, and wooden floors HSL - hue-saturation-lightness Clusters of color color-light fingerprint

Color/light fingerprint in the HSL space

Fingerprinting Wi-Fi WiFi fingerprinting no good for logical places WiFi based fingerprinting -fifth sensor MAC addresses of visible APs MAC addresses recorded every 5 seconds Computing the fraction of times each unique MAC address was seen over all recordings A tuple of fractions forms the WiFi fingerprint of that place

Fingerprint Matching SurroundSense uses 4 filtering/matching The (WiFi, sound, and accelerometer) filters are applied first Candidate set fed to the color/light-based matching scheme Use the color/light based matching scheme last The final output is an ordered list of candidates – the top ranked candidate is declared to be the location of the phone

ProtoType Implementation Client and Server Populating the Fingerprint Database Special Note: SurroundSense was implemented on Nokia N95 using Python platform. The server -MATLAB, Python code, data mining tools

Client and Server Sensor runs on threads and execute API calls The accelerometer samples - 4 readings per second. The audio sampling rate is 8 kHz. Pictures are taken every 5 seconds A meta file -stores date, time, GSM, camera mode The server - several modules. A Data Manager formats raw data appropriately. The formatted data sent to Fingerprinting Factory A MATLAB/ Python based Filtering/Matching Module -computes the top-ranked match.

Populating the Fingerprint Database How did they build a fingerprint database? 46 business locations 5 locations in India Students visited 51 stores Stores visited multiple times Design location labeling games. The person with a best match may win a prize. More people play larger fingerprint database

Evaluation Partially Controlled Experimentation Performance Pre-Cluster Accuracy Per-Shop Accuracy Per-User Accuracy Per-Sensor Accuracy

Partially Controlled Experimentation Not performed with a real user base Mobile phones in our hand (and not in our pockets) Phones took pictures for color and light fingerprinting. In uncontrolled environments, phones in pocket New wearable mobile phones Wrist watches and Necklaces

Mimicking Customer Behavior Groups of 2 people Went to different stores -time-separated Fingerprinted every store in cluster Behave like normal customers Purchase coffee and food Mimic the movement of another customers Atypical behavior -picking up pre-ordered food shopping very quickly.

Performance: Per-Cluster Accuracy Evaluate 4 modes 1. WiFi-only 2. Sound, Accelerometer, Light and Color 3. Sound, Accelerometer 4. SurroundSense (SS) combined all modes of ambience fingerprinting SurroundSense average accuracy of 87% All the sensors 90%...

Performance: Per-Shop Accuracy 47% of the shops can be localized perfectly using SurroundSense. WiFi displays bimodal behavior – it’s either high accuracy, or seriously suffers Clearly, the combination of multi-modal fingerprinting is best

Performance: Per-User Accuracy User assigned to a random set of stores Report the average accuracy SurroundSense users achieve between 73% and 75% accuracy The accuracy grows to an average of 83% or more for 80% of the users The median accuracy is around 88%, while 10% users experience 96% accuracy or more

Performance: Per-Sensor Accuracy hand-picked 6 examples -merits and demerits of each sensor Whenever the accelerometer is used-accuracy is always 100% Only the camera, -100% accuracy in this location Only color gives average accuracy of 91%. When sound is added, -66% If the correct location is filtered out, the final match incorrect

Limitations and Future Work Energy Considerations Energy efficient localization and sensing Simple sensing mechanisms – when outdoors Variation in GSM signal strengths Temperature sensing An accelerometer trace requires time Faster methods of localization without compromising accuracy

Conclusion Logical location, vs physical coordinates The main idea -ambient sound, light, color, RF, luser movement Fingerprint identifies user’s location SurroundSense not a stand-alone technique- use with GSM location SurroundSense step towards indoor localization Further research - better energy management SurroundSense a viable solution of the future

Questions/Comments?