1. SixthSense: RFID-based Enterprise Intelligence Lenin Ravindranath, Venkat Padmanabhan (MSR India) Piyush Agrawal (IIT Kanpur)

2. RFID  Radio Frequency Identification  Components  RFID Reader with Antennas  Tags (Active and Passive)  Electromagnetic waves induce current  Tag responds  Globally unique ID  Data  UHF (865-956 MHz)  Range – up to 7m  Applications  Tracking, Inventory, Supply Chain, Authentication, …  Novel Research Applications

3. Motivating Scenario Lenin missed an object in the conference room – 2 nd floor Scientia

4. SixthSense Goal  Making people, objects and workspaces, the first class citizens of the enterprise computing Components  Use RFID to capture the rich interaction between people and their surroundings  Combine with other enterprise systems/sensors to make automated inferences  Enable Useful Services

5. Setting People and objects tagged Camera Calendar Presence RFID Antennas

6. Assumptions  Widespread coverage of RFID readers in the workspace  Users are free to pick up new tags and affix them to objects  Can put multiple tags on an object  No dependence on cataloging  Cataloging is an overhead  TagID  Entity  Antenna  Workspace  Error Prone  Tags are fragile – may have to be replaced  Readers/Antenna could be moved  Start with an undifferentiated mass of tags and infer everything Lenin missed an object in the conference room – 2 nd floor Scientia 13548234 – Ant 1 15574523 – Ant 1 13548234 – Ant 6 15574523 – Ant 1

7. Architecture People and objects tagged Camera Calendar Presence RFID Antennas SixthSense

8.  Automated Inference  Platform  Programming Model  Applications

9. Inference Engine  Person-Object Differentiation  Object Ownership Inference  Zone Identification  Person Identification  Person-Object Interaction

10. Person-Object Differentiation  People can move on their own  Objects move only when carried by a person Co-movement based heuristic  Relative Movement (RM) Zone 1 Zone 2Zone 3

11. Object Ownership Inference  Co-Presence  Calculate the amount of time the object is concurrently present in the same zone as a person  Owner is the person with which the object is co-present the most and greater than a threshold

12. Person Identification 1 xyz abc 1 2 Workspace Entrance Event Log-in event Coincidence count 1 xyz 1 time t1 t2 1

13. Object Interaction (only in zones of interest)  Intra zone  Identify interaction in zones of interest  A person lifted an object  A person turned the orientation an object  Signal Strength of tag varies  Change in distance  Change in orientation  Contact  Monitor variation in RSSI

14. Object Interaction  Sample the RSSI of each object tag every 200ms  Sliding 4-second wide window  Difference between the 10 th percentile and 90 th percentile of the RSSI  Object is said to be interacted - If the difference > threshold  Minimizing spurious detections  Use multiple antennas

15. Object Interaction Antenna 1 Antenna 2 Interacted

16. Ensuring Privacy  Enterprise will deploy and manage the system  Expose appropriate set of information  Trust - Analogous to the enterprise e-mail system  Defend against rogue readers  Relabeling approach [A. Juels, 2006]  EPC code rewritten at random times  SixthSense will be aware of the mapping between the old and new tag IDs

17. Implementation Simulator (Trace Generation) Experimental Setup (Real-time feed)

18. Experimental Setup

19. Results  Inter-zone movement detection ObjectReliability (1m)Reliability (2m) Badge on belt clip100%96% Small box in hand94%88%  Object Interaction  Testbed deployment  To make correct inferences  Average inter-zone movements needed – 4  Average log-ins required - 3 Distance between antennasDetection time 1.5m2.39s 2m3.4s 2.5m5.03s

20. Simulation  Probabilistic model to generate artificial traces  Simulated  Inter-zone movement (walk)  People carrying multiple objects  Log-in events  Untagged people Zone 1Zone 2Zone 3

21. Results  Person-object differentiation  Person Identification  Varying average walk length  Effects of untagged people

22. Person-Object differentiation and ownership  20 people, 100 tags, probability of walk – 0.1, walk length - 5

23. Person Identification  10% of users entering workspace simultaneously

24. Programming Model  Callbacks  InterZoneMovementEvent (tagID, startZone, endZone, Time)  ObjectInteractedEvent(tadID, Zone, Time)  Lookups  GetTagList()  GetPersonTags()  GetOwnedObjects(tagID)  GetTagType(tagID)  GetTagOwner(tagID)  GetPersonTagIdentity(tagID)  GetZoneType(zone)  GetTagsInZone(zone)  GetTagWorkspaceZone(tagID)  GetCurrentTagZone(tagID)  GetCalendarEntry(ID, Time)

25. Example  Misplaced Object Alert personTags = GetPersonTags() For each ownerTagID in personTags ObjTags = GetOwnedObject(ownerTagID) OwnerZone = GetCurrentTagZone(ownerTagId) OwnerWorkspace = GetTagWorkspaceZone(ownerTagId) For each obj in ObjTags objZone = GetCurrentTagZone(ownerTagId) if (objZone != OwnerZone && objZone != OwnerWorkspace) Raise Alert

26. Applications  Annotated video  Semi-automated image catalog  Misplaced object alert  Automatic conference room booking

27. Annotating video with physical events  Events  Inter-zone movements  Object Interaction  Tag video feed with events  Person X interacted an object Y  Rich video database  Support rich queries  Give me all videos where Person A interacted with Object B  Application: Integrate with enterprise security camera system

28. Semi-automated Image catalog  TagIDs are not user friendly  Catalog tagID with its Image  User picks up an object  Shows before the camera and takes a photo  Automatic cataloging (TagID, Image)

29. Annotating video with physical events

30. Related Work  Localization  LANDMARC  Indoor Location Sensing Using Active RFID  Sherlock (UMass)  Automatically locating objects for humans  Ferret (UMass)  RFID Localization for Pervasive Multimedia  Platform  Cascadia (UWashington)  Specifying, detecting and managing RFID events  Object Interaction  I sense a disturbance in the force (Intel Research, Seattle)  Unobtrusive detection of Interactions with RFID-tagged Objects  With other sensors  Fusion of RFID and Computer Vision (MSR)

31. Summary  SixthSense  Enterprise Setting  People and Objects tagged  RFID with other enterprise sensors  Components  Automated Inference  Platform  Applications http://research.microsoft.com/research/mns/projects/SixthSense/ Questions?

32. Backup

33. Semi-Automated Image Catalog  TagID-Image Cataloging  User picks up a tagged object  Hold it in front of the camera  Clicks a picture  Automatically identify the tagID of the object

34. SixthSense System Inference Engine, Database, Applications RFID Reader RFID Antennas Calendar DataPresence Data Applications Queries

35.  Industry  Tracking, Inventory, Supply Chain, Authentication  Research  Measurements  Improving reliability, security  Localization  RFID + Computer Vision  Interaction detection  RFID + other sensors RFID Applications

36. Person-Object Differentiation  People can move on their own  Objects move only when carried by a person Co-movement based heuristic  For every tag T, find co-movement tag set {T 1, T 2..T n }  m – total inter-zone movement of T  m i – total inter-zone movement of T i  c i – amount of co-movement exhibited by T i with T  Declare the tag with the highest RM as person  Eliminate this tags movements  Repeat the algorithm till RM is positive  Tags with negative RM are objects

37. Zone Identification  Individual workspace  If there is one person predominantly present in a zone  Workspace of that person  Shared workspace  If no one person is predominantly present in a zone  Length of time from a person entry to exit < threshold  Reserved shared workspace  Length of time people are present > threshold  Common meeting entries in their calendars  Common areas  Any space that is not classified as one of the above

38. Challenges – Improving Reliability  Multi-tagging scheme  Affix multiple tags in different orientation  Increases the probability that atleast one of the tags being detected  Automatic Inference  Initially assume all tags belong to one giant super object  Fully connected graph  When two tags are detected simultaneously in different zones  Tags belong to different objects  Delete edges between them  Connected components  Set of tags attached to the same object

39. Evaluation – with untagged people

40. Automatic Conference Room Booking  Conference Room Zone is automatically identified  Reserved Space  Automatically book conference room  If it is not reserved  And bunch of people go into the conference room  And spend say 5 minutes

41. Discussion  Privacy  Deployed and managed by enterprise  Limited access to users  Relabeling approach  Economic Feasibility  Passive Tags are cheap  Prices are RFID readers expected to drop (Intel R1000)  Health Implications  Transmitted RF power (up to 2W) is well within safe limits  this question will undoubtedly continue to receive much attention and study