1 SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

2 Context Pervasive wireless connectivity + Localization technology = Location-based applications

3 Location-Based Applications (LBAs) For Example:  GeoLife shows grocery list when near Walmart  MicroBlog queries users at a museum  Location-based ad: Phone gets coupon at Starbucks iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4 Most emerging location based apps do not care about the physical location GPS: Latitude, Longitude

5 Most emerging location based apps do not care about the physical location Instead, they need the user’s logical location GPS: Latitude, Longitude Starbucks, RadioShack, Museum, Library

6 Physical Vs Logical Unfortunately, most existing solutions are physical  GPS  GSM based  SkyHook  Google Latitude  RADAR  Cricket  …

7 Widely-deployable localization technologies have errors in the range of several meters Widely-deployable localization technologies have errors in the range of several meters Can we convert from Physical to Logical Localization? State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

8 Physical Location Error

9 Pizza HutStarbucks Physical Location Error

10 Pizza HutStarbucks Physical Location Error The dividing-wall problem

11 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

12 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

13 SurroundSense: A Logical Localization Solution

14 It is possible to localize phones by sensing the ambience It is possible to localize phones by sensing the ambience Hypothesis such as sound, light, color, movement, WiFi …

15 It is possible to localize phones by sensing the ambience It is possible to localize phones by sensing the ambience Hypothesis such as sound, light, color, movement, WiFi …

16 Multi-dimensional sensing extracts more ambient information Any one dimension may not be unique, but put together, they may provide a unique fingerprint

17 SurroundSense Multi-dimensional fingerprint  Based on ambient sound/light/color/movement/WiFi Starbucks Wall Pizza Hut

18 B A C D E Should Ambiences be Unique Worldwide? F G H J I L M N O P Q Q R K

19 Should Ambiences be Unique Worldwide? B A C D E F G H J I K L M N O P Q Q R GSM provides macro location (strip mall) SurroundSense refines to Starbucks

20 Economics forces nearby businesses to be diverse Not profitable to have 3 adjascent coffee shops with same lighting, music, color, layout, etc. SurroundSense exploits this ambience diversity Why does it work? The Intuition:

21 Architecture: Filtering & Matching Candidate Fingerprints

22 Fingerprints Sound: (via phone microphone) Color: (via phone camera) Amplitude Values Normalized Count Acoustic fingerprint (amplitude distribution) Color and light fingerprints on HSL space Lightnes s Hue Saturation

23 Fingerprinting Sound Fingerprint generation : Signal amplitude  Amplitude values divided in 100 equal intervals  Sound Fingerprint = 100 normalized values value X = # of samples in interval x / total # of samples Filter Metric: Euclidean distance  Discard candidate fingerprint if metric > threshold г Threshold г  Multiple 1 minute recordings at the same location  d i = max dist ( any two recordings )  г = max ( d i of candidate locations )

24 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Moving

25 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Queuing Seated Moving

26 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Pause for product browsing Short walks between product browsing Moving

27 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Walk more Quicker stops Moving

28 Fingerprinting Motion Fingerprint generation: stationary/moving periods  Sitting (restaurants, cafes, haircutters )  Slow Browsing (bookstores, music stores, clothing)  Speed-Walking (groceries) Filter Metric:  Discard candidate fingerprints with different classification 0.0 ≤ R ≤ 0.2 sitting 0.2 ≤ R ≤ 2.0 slow browsing 2.0 ≤ R ≤ ∞ speed-walking

29 Fingerprinting WiFi Fingerprint generation: fraction of time each unique address was overheard Filter/Ranking Metric  Discard candidate fingerprints which do not have similar MAC frequencies

30 Fingerprints Sound: (via phone microphone) Color: (via phone camera) Amplitude Values Normalized Count Acoustic fingerprint (amplitude distribution) Color and light fingerprints on HSL space Lightnes s Hue Saturation

31 Fingerprinting Color RGB based  Shadow of objects  Reflections of light  Light intensity differ

32 Fingerprinting Color Floor Pictures  Privacy concern  Less noisy  Rich diversity across different locations  Point their camera downward Fingerprint generation: pictures in HSL space  K-means clustering algorithm  Cluster’s centers + sizes Ranking metric

33 HSL

34 Discussion Time varying ambience  Collect ambience fingerprints over different time windows What if phones are in pockets?  Use sound/WiFi/movement  Opportunistically take pictures Fingerprint Database  War-sensing

35 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

36 Evaluation Methodology 51 business locations  46 in Durham, NC  5 in India Data collected by 4 people  12 tests per location Mimicked customer behavior

37 Evaluation: Per-Cluster Accuracy Cluster No. of Shops Accuracy (%) Cluster Localization accuracy per cluster Multidimensional sensing

38 Evaluation: Per-Cluster Accuracy Cluster No. of Shops Accuracy (%) Cluster Localization accuracy per cluster Sparse WiFi APs

39 Evaluation: Per-Cluster Accuracy Cluster No. of Shops No WiFi APs Accuracy (%) Cluster Localization accuracy per cluster

40 Evaluation: Per-Cluster Accuracy ModeWiFiSnd-Acc-WiFiSnd-Acc-Lt-ClrSS Accuracy70%74%76%87%

41 Evaluation: Per-Shop Accuracy Localization Accuracy per Shop Average Accuracy (%) CDF WiFi Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense

42 Evaluation : Per-User Accuracy Random Person Accuracy Average Accuracy (%) CDF WiFI Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense

43 Evaluation : Per-Sensor Accuracy

44 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

45 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw

46 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw  Detected outdoor, turn off the phone Localization in Real Time  User’s movement requires time to converge

47 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time  User’s movement requires time to converge Non-business locations  Office, Libraries, airports  Ambiences may be less diverse

48 Electroic compasses can fingerprint layout  Tables and shelves laid out in different orientations  Users forced to orient in those ways Limitations and Future Work

49 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

50 Ambience can be a great clue about location Ambient Sound, light, color, movement … None of the individual sensors good enough Combined they may be unique Uniqueness facilitated by economic incentive Businesses benefit if they are mutually diverse in ambience Ambience diversity helps SurroundSense Current accuracy of 89% Conclusion

51 SurroundSense evaluated in 51 business locations Achieved 89% accuracy But perhaps more importantly, SurroundSense scales to any part of the world Summary

52 Questions?