1 “Did you see Bob?”: Human Localization using Mobile Phones Ionut Constandache Co-authors: Xuan Bao, Martin Azizyan, and Romit Roy Choudhury Modified by Chulhong

2 Localization Technologies Outdoor Driving directions  GPS, Skyhook Indoor Localization in office  Cricket, Radar, BAT Energy-Efficient Continuous localization  EnLoc, RAPS Logical Context-aware ads  SurroundSense

3 Localization Technologies Outdoor Driving directions  GPS, Skyhook Indoor Localization in office  Cricket, Radar, BAT Energy-Efficient Continuous localization  EnLoc, RAPS Logical Context-aware ads  SurroundSense Human Localization: Guiding a user to finding another person Human Localization: Guiding a user to finding another person

4 Usage Scenario Bob Alice

5 Usage Scenario Where is Bob? Please escort me to Bob. Bob Alice

6 Usage Scenario Where is Bob? Bob Alice

7 Usage Scenario Where is Bob? Bob Alice General approach today: 1.Stroll around in the hotel until Alice can visually spot Bob 2.Ask “Have you seen Bob around?” 3.Phone call However, what if Alice does not know Bob?

8 Usage Scenario Where is Bob? Please escort me to Bob. Bob Provide an electronic Escort system. Alice

9 Usage Scenario Alice’s Phone 20 steps North 5 steps East N Bob

10 Usage Scenario Alice’s Phone

11 Usage Scenario Alice’s Phone Bob

12 Human Localization Finding Bob in unfamiliar place (E.g. library, mall, engineering building)

13 Human Localization Finding Bob in unfamiliar place (E.g. library, mall, engineering building)

14 Human Localization Finding Bob in unfamiliar place (E.g. library, mall, engineering building) Better for Alice to be escorted to Bob

15 Human Localization Finding Bob in unfamiliar place (E.g. library, mall, engineering building) Better for Alice to be escorted to Bob Challenges: Bob’s location unknown Challenges: Bob’s location unknown

16 Human Localization Finding Bob in unfamiliar place (E.g. library, mall, engineering building) Better for Alice to be escorted to Bob Challenges: Bob’s location unknown Even if known still require … WALK-able routes to Bob Challenges: Bob’s location unknown Even if known still require … WALK-able routes to Bob

17 Human Localization Finding Bob in unfamiliar place (E.g. library, mall, engineering building) Better for Alice to be escorted to Bob Challenges: Bob’s location unknown Even if known still require … WALK-able routes to Bob Once in his vicinity, identify Bob Challenges: Bob’s location unknown Even if known still require … WALK-able routes to Bob Once in his vicinity, identify Bob

Can current localization schemes help?

too heavy on requirements …  Infrastructure: specialized hardware (e.g. Cricket, BAT, etc.) or  War-driving: build fingerprint DB (e.g. Radar, Skyhook, etc.) Can current localization schemes help?

too heavy on requirements …  Infrastructure: specialized hardware (e.g. Cricket, BAT, etc.) or  War-driving: build fingerprint DB (e.g. Radar, Skyhook, etc.) … need lightweight localization solution Can current localization schemes help?

21 Contents Escort Evaluation Limitations and Future Work Conclusion

22 Contents Escort Evaluation Limitations and Future Work Conclusion

23 Our Solution Accelerometers/compasses track human movements  Standard sensors in mobile phones  Each user has a trail trail

24 Our Solution Accelerometers/compasses track human movements  Standard sensors in mobile phones  Each user has a trail trail step i, direction i > = TRAIL <

25 Our Solution Accelerometers/compasses track human movements  Standard sensors in mobile phones  Each user has a trail trail

26 Our Solution Deploy coordinate system to localize users  Any (fixed) location can be the origin  N, E directions are the Y, X axises N E Origin

27 Our Solution Users join the coordinate system  When passing the origin  At encounters with users already in the system (0,0) N E Origin

28 Our Solution Users join the coordinate system  When passing the origin  At encounters with users already in the system N E Origin (x,y)

29 Our Solution Users join the coordinate system  When passing the origin  At encounters with users already in the system N E Origin (x,y)

How does Escorting work?

C A B D

C A B D Escort Service Cloud A’s Trail

C A B D Escort Service Cloud A’s Trail

C A B D Escort Service Cloud

C A B D Escort Service Cloud

C A B D I BD I BC I AC Escort Service Cloud

C A B D I BD I BC I AC Escort Service Cloud

C A B D I BD I BC I AC I BC I BD A CB D Trail Graph Escort Service Cloud

39 Trail Graph I AC I BC C B D I BD A

40 Escort along the Trail Graph I AC C D I BD B I BC Alice Bob A

41 Escort along the Trail Graph I AC C D I BD B I BC Alice Bob A

42 Escort along the Trail Graph I AC C D I BD B I BC Bob A Alice

43 Escort along the Trail Graph I AC C D I BD B I BC Bob A Alice WALK-able routes Alice guided along user trails: Trails need to be accurate Alice guided along user trails: Trails need to be accurate

44 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

45 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

46 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

47 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

48 Challenges Trails drift: acc. missed steps, compass biases t1t1 Compass bias t2t2 θ

49 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

50 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

51 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

52 Challenges Trails drift: acc. missed steps, compass biases t1t1 t2t2

53 Challenges Trails drift: acc. missed steps, compass biases t1t1 Actual Drifted t2t2

54 Challenges Trails drift: acc. missed steps, compass biases t1t1 Actual Drifted t2t2

55 Challenges Trails drift: acc. missed steps, compass biases t1t1 Actual Drifted t2t2 Error

56 Challenges Trails drift: acc. missed steps, compass biases t1t1 Actual Drifted t2t2 Error Need to correct: User Location User Trail Need to correct: User Location User Trail

57 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin

58 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin

59 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin

60 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin Encounter with origin (0,0)

61 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin Encounter with origin (0,0)

62 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin Encounter with origin (0,0)

63 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin Encounter with phone with good location estimate (x,y)

64 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin Encounter with phone with good location estimate (x,y)

65 Correct User Location Opportunities  When passing the origin, the user is at (0,0)  Close-encounters with users who passed the origin recently Take this user’s position (it’s accurate) N E Origin Encounter with phone with good location estimate (x,y) How to detect encounters with origin/users?

66 Detecting Encounters using Sound Phones periodically beacon their presence  Beacons = unique audio tones  Phones also listen for neighboring beacons

67 Detecting Encounters using Sound Phones periodically beacon their presence  Beacons = unique audio tones  Phones also listen for neighboring beacons Tone amplitude above threshold  encounter

68 Correct User Trail Actual Drifted t1t1 t2t2 Error

69 Origin Correct User Trail Actual Drifted t1t1 t2t2 Error

70 Origin t1t1 Correct User Trail Actual Drifted t2t2

71 Origin t1t1 Correct User Trail L(t) L’(t) t2t2 Actual Drifted

72 Origin t1t1 Correct User Trail Actual Drifted Corrected L(t) L’(t) t2t2

73 Escort After drift correction  Escort users along the corrected trails  Place user in vicinity of the searched-for person

74 Escort After drift correction  Escort users along the corrected trails  Place user in vicinity of the searched-for person I BC I BD A C B D I AC

75 Escort After drift correction  Escort users along the corrected trails  Place user in vicinity of the searched-for person E F I BC I BD A C D I AC B

76 Escort After drift correction  Escort users along the corrected trails  Place user in vicinity of the searched-for person E F I BC I BD A C D I AC How to visually identify Bob? B

77 Solve human localization end-to-end  Create visual fingerprint for each user Alice’s Phone Bob Visual Identification

78 Visual Fingerprint User picture

79 Visual Fingerprint Upper Region Lower Region Fingerprint User picture

Recognizing Bob

Users advertise

Recognizing Bob Alice’s Phone

Recognizing Bob Alice’s Phone

Recognizing Bob Alice’s Phone

Recognizing Bob Alice’s Phone Bob

86 Contents Escort Evaluation Limitations and Future Work Conclusion

87 Evaluation Escort target accuracy: several meters  Require high ground-truth accuracy ~ 1m  GPS not accurate enough Our approach  Run experiments in a testbed with dense markers  Markers have known position

88 A C D 36 m 48 m Testbed Markers Origin B

89 A C D 36 m 48 m Testbed Markers Origin User Paths B

90 A C D 36 m 48 m Testbed Markers Origin User Paths B 4 Test Users 13 minutes experiment User locations known at markers 40 escorting tests

91 Average Localization Error Average Localization Error across all users: ~ 6m

92 Final Distance from Destination Average Error at end of escorting: ~ 8m

93 Visual Identification Accuracy

94 Contents Escort Evaluation Limitations and Future Work Conclusion

95 Limitations and Future Work Employees only access  Trails may have restrictions Phone placement  Assumed in hand, investigate placement as future work Imprecise navigation  Humans can make educated guesses Testing under heavy user load

96 Conclusions We asked ourselves:  Can mobile phones help in “routing” a person A to a person B Challenging because:  Require walkable routes  Needs to be free of infrastructure, war-driving Possible because:  Rich sensing capabilities on mobile phones  High density of such devices

97 Conclusions Our approach:  “Stitching” human walking traces to compose a graph.  Route humans on this walkable graph  Solution is analogous to routing in DTNs …  Only packets are now humans Alice Bob

98 Questions? Thank You! Visit the SyNRG research