1. 1 SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

2. 2 Context Pervasive wireless connectivity + Localization technology = Location-based applications

3. 3 Location-Based Applications (LBAs) For Example:  GeoLife shows grocery list when near Walmart  MicroBlog queries users at a museum  Location-based ad: Phone gets coupon at Starbucks iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4. 4 Most emerging location based apps do not care about the physical location GPS: Latitude, Longitude

5. 5 Most emerging location based apps do not care about the physical location Instead, they need the user’s logical location GPS: Latitude, Longitude Starbucks, RadioShack, Museum, Library

6. 6 Physical Vs Logical Unfortunately, most existing solutions are physical  GPS  GSM based  SkyHook  Google Latitude  RADAR  Cricket  …

7. 7 Given this rich literature, Why not convert from Physical to Logical Locations?

8. 8 Physical Location Error

9. 9 Pizza HutStarbucks Physical Location Error

10. 10 Pizza HutStarbucks Physical Location Error The dividing-wall problem

11. 11 SurroundSense: A Logical Localization Solution

12. 12 It is possible to localize phones by sensing the ambience It is possible to localize phones by sensing the ambience Hypothesis such as sound, light, color, movement, WiFi …

13. 13 It is possible to localize phones by sensing the ambience It is possible to localize phones by sensing the ambience Hypothesis such as sound, light, color, movement, WiFi …

14. 14 Multi-dimensional sensing extracts more ambient information Any one dimension may not be unique, but put together, they may provide a unique fingerprint

15. 15 SurroundSense Multi-dimensional fingerprint  Based on ambient sound/light/color/movement/WiFi Starbucks Wall Pizza Hut

16. 16 B A C D E Should Ambiences be Unique Worldwide? F G H J I L M N O P Q Q R K

17. 17 Should Ambiences be Unique Worldwide? B A C D E F G H J I K L M N O P Q Q R GSM provides macro location (strip mall) SurroundSense refines to Starbucks

18. 18 Economics forces nearby businesses to be diverse Not profitable to have 3 adjascent coffee shops with same lighting, music, color, layout, etc. SurroundSense exploits this ambience diversity Why does it work? The Intuition:

19. 19 + + Ambience Fingerprinting Test Fingerprint Sound Acc. Color/Light WiFi Logical Location Matching Fingerprint Database = = Candidate Fingerprints GSM Macro Location SurroundSense Architecture

20. 20 Fingerprints Sound: (via phone microphone) Color: (via phone camera) Amplitude Values -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Normalized Count 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 Acoustic fingerprint (amplitude distribution) Color and light fingerprints on HSL space Lightnes s 1 0.5 0 Hue 0 0.5 1 0 0.2 0.4 0.6 0.8 1 Saturation

21. 21 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Moving

22. 22 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Queuing Seated Moving

23. 23 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Pause for product browsing Short walks between product browsing Moving

24. 24 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Walk more Quicker stops Moving

25. 25 Fingerprints Movement: (via phone accelerometer) WiFi: (via phone wireless card) CafeteriaClothes Store Grocery Store Static ƒ (overheard WiFi APs) Moving

26. 26 Discussion Time varying ambience  Collect ambience fingerprints over different time windows What if phones are in pockets?  Use sound/WiFi/movement  Opportunistically take pictures Fingerprint Database  War-sensing

27. 27 Evaluation Methodology 51 business locations  46 in Durham, NC  5 in India Data collected by 4 people  12 tests per location Mimicked customer behavior

28. 28 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Accuracy (%) Cluster Localization accuracy per cluster

29. 29 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Fault tolerance Accuracy (%) Cluster Localization accuracy per cluster

30. 30 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Accuracy (%) Cluster Localization accuracy per cluster Sparse WiFi APs

31. 31 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 No WiFi APs Accuracy (%) Cluster Localization accuracy per cluster

32. 32 Evaluation: Per-Scheme Accuracy ModeWiFiSnd-Acc-WiFiSnd-Acc-Lt-ClrSS Accuracy70%74%76%87%

33. 33 Evaluation: User Experience Random Person Accuracy Average Accuracy (%) 0 10 20 30 40 50 60 70 80 90 100 1 0.9 0.8 0.7 0.6 0.5 CDF 0.4 0.3 0.2 0.1 0 WiFI Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense

34. 34 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw

35. 35 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time  User’s movement requires time to converge

36. 36 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time  User’s movement requires time to converge Non-business locations  Ambiences may be less diverse

37. 37 Electroic compasses can fingerprint layout  Tables and shelves laid out in different orientations  Users forced to orient in those ways Limitations and Future Work

38. 38 Ambience can be a great clue about location Ambient Sound, light, color, movement … None of the individual sensors good enough Combined they may be unique Uniqueness facilitated by economic incentive Businesses benefit if they are mutually diverse in ambience Ambience diversity helps SurroundSense Current accuracy of 89% Conclusion

39. 39 Questions?

40. 40 SurroundSense evaluated in 51 business locations Achieved 89% accuracy But perhaps more importantly, SurroundSense scales to any part of the world Summary

41. 41 SurroundSense Today’s technologies cannot provide logical localization Ambience contains information for logical localization Mobile Phones can harness the ambience via sensors  More sensors, beter reliability Evaluation results:  51 business locations,  87% accuracy SurroundSense can scale to any part of the world

42. 42

43. 43 Evaluation: Per-Shop Accuracy Localization Accuracy per Shop Average Accuracy (%) 0 10 20 30 40 50 60 70 80 90 100 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 CDF WiFi Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense

44. 44 Architecture: Filtering & Matching Candidate Fingerprints

45. 45 Fingerprinting Sound Fingerprint generation : Signal amplitude  Amplitude values divided in 100 equal intervals  Sound Fingerprint = 100 normalized values value X = # of samples in interval x / total # of samples Filter Metric: Euclidean distance  Discard candidate fingerprint if metric > threshold г Threshold г  Multiple 1 minute recordings at the same location  d i = max dist ( any two recordings )  г = max ( d i of candidate locations )

46. 46 Fingerprinting Motion Fingerprint generation: stationary/moving periods  Sitting (restaurants, cafes, haircutters )  Slow Browsing (bookstores, music stores, clothing)  Speed-Walking (groceries) Filter Metric:  Discard candidate fingerprints with different classification 0.0 ≤ R ≤ 0.2 sitting 0.2 ≤ R ≤ 2.0 slow browsing 2.0 ≤ R ≤ ∞ speed-walking

47. 47 Fingerprinting WiFi Fingerprint generation: fraction of time each unique address was overheard Filter/Ranking Metric  Discard candidate fingerprints which do not have similar MAC frequencies

48. 48 Fingerprinting Color Floor Pictures  Rich diversity across different locations  Uniformity at the same location Fingerprint generation: pictures in HSL space  K-means clustering algorithm  Cluster’s centers + sizes Ranking metric

49. 49 Thinking about Localization from an application perspective…

50. 50 Emerging location based apps need place of user, not physical location Starbucks, RadioShack, Museum, Library Latitude, Longitude

51. 51 Emerging location based apps need place of user, not physical location Starbucks, RadioShack, Museum, Library Latitude, Longitude We call this Logical Localization …

52. 52 Can we convert from Physical to Logical Localization?

53. 53 Can we convert from Physical to Logical Localization? State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

54. 54 Widely-deployable localization technologies have errors in the range of several meters Widely-deployable localization technologies have errors in the range of several meters Can we convert from Physical to Logical Localization? State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

55. 55 Physical Vs Logical Lot of work in physical localization  GPS (5m accuracy, outdoor only)  GSM based (high coverage, 200m accuracy)  SkyHook (40m accuracy, need war-driving)  Google Latitude (similar)  RADAR (5m accuracy, need indoor mapping)  Cricket (install per-room devices)  … Each of them has distinct problems …

56. 56 Physical Vs Logical Lot of work in physical localization  GPS (5m accuracy, outdoor only)  GSM based (high coverage, 200m accuracy)  SkyHook (40m accuracy, need war-driving)  Google Latitude (similar)  RADAR (5m accuracy, need indoor mapping)  Cricket (install per-room devices)  … Each of them has distinct problems …

57. 57 Even if we assume the best of all solutions (5m accuracy indoors) logical localization still remains a challenge.

58. 58 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

59. 59 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

60. 60 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

61. 61 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

62. 62 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

63. 63 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Accuracy (%) Cluster Localization accuracy per cluster Multidimensional sensing

64. 64 Limitations and Future Work Energy-Efficiency Localization in Real Time Non-business locations

65. 65 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time Non-business locations

66. 66 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time  User’s movement requires time to converge Non-business locations

67. 67 Limitations and Future Work Camera inside pockets  Detect phone when out of pocket  Takes pictures when camera pointing downward