SurroundSense: Mobile Phone Localization via Ambience Fingerprinting Written by Martin Azizyan, Ionut Constandache, & Romit Choudhury Presented by Craig McIlwee

Motivation Provide logical localization Using GPS only isn’t good enough – Doesn’t work well indoors – Doesn’t account for dividing walls Dedicated hardware is not scalable

Approach Create an ambience fingerprint using sound, light, color, and user movement – Noise signatures specific to type of location/store – Chain stores have color themes – User movement indicative of store type

Architecture/Algorithm

Data is recorded on the phone, preprocessed, and sent to a server Filter module – Subsets the candidates – Wifi, movement, sound Match module – Selects the best candidate – Color/sound, Wifi

Architecture/Algorithm No single module needs to be perfect – If each module is ‘good enough’ then all modules combined are sufficient – Being simple reasonably accurate instead of sophisticated and perfect reduces resources required for processing

Sound Module Filter – Sound varies over time Fingerprints captured from various times of day Similarity of fingerprints is used to create a threshold for a potential match Match if within the threshold, discard otherwise – Threshold is generous – More false positives is better than false negatives

Motion Module Filter – Variations in user behavior Record 4 samples/second, use moving average over last 10 samples Minor variations suppressed

Motion Module User movement is classified as stationary or mobile 3 profiles defined – Long stationary – restaurant – Frequent movement with longer stationary – browsing – Frequent movement with shorter stationary – shopping Some logical locations fit multiple profiles

Motion Module

Color/Light Module Match Images captured from camera while facing downward – Floor themes are consistent – Other orientations introduce noise – Common orientation when checking , text messages, etc

Color/Light Module Analyze patterns in the image First attempt was to convert pixels to RGB values – Failed due to shadow and reflection influences Second attempt was to convert to HSL values – Isolates light on its own axis

Color/Light Module

Same/similar colors result in clusters when graphed Dominant colors generate larger clusters Similarity calculated as distance between cluster centroids and size of the clusters Most similar candidate is the match

Wifi Module Normally a filter, match if camera is not available Capture MAC address of available access points every 5 seconds Compare occurrence ratio of currently available access points to known access points

Known Issues Sound varies over time – Split day into 2 hour windows, capture fingerprints during each window – No mention of day of week, time of year Camera in pocket – All testing done with phone in hand – Expected rise in wearable devices Mimicking user behavior – Initial data showed artificial behavior – Subsequent attempts shadowed real customers

Known Issues Resource (energy) intensive Accelerometer fingerprint takes time to capture Non-business locations may not exhibit enough diversity – Offices, airports, libraries

Evaluation Recorded fingerprints of 51 locations – “War-sensed” by students – 2 different groups during different times of day Group A’s fingerprints used as database while Group B was at the location collecting their own fingerprints Accuracy analysis was done on various combinations of sensors types All sensor types combined yielded 87% accuracy