1. Context Fabric: Infrastructure Support for Context-Aware Systems Jason I. Hong Qualifying Exam Proposal

2. 2 Mar 20 2001 Why Context-Awareness? Context TypesExisting ExamplesHuman Concern Room ActivitySmoke AlarmSafety Room ActivityAuto Lights On / OffConvenience Object IdentityBarcode ScannersEfficiency Personal Identity & Time File SystemsFinding Info TimeCalendar RemindersMemory

3. 3 Mar 20 2001 Technology Trends Sensors ­GPS, Active Badges, Active Bats ­Smart Dust ­Cameras and microphones Recognition algorithms ­MSR Radar location from 802.11 ­Smart Floor footstep force (Orr & Abowd 2000) Wireless technologies ­Bluetooth, 802.11, cell phone

4. 4 Mar 20 2001 Existing Examples Why Context-Awareness? Context Types Potential ExamplesHuman Concern ActivitySafety ActivityConvenience IdentityEfficiency Identity & TimeFinding Info TimeMemory Identity Time Location Proximity Activity History … Smoke Alarm Auto Lights On / Off Barcode Scanners File Systems Calendar Reminders Health Alert Auto Cell Phone Off In Meetings Service Fleet Dispatching Tag Photos Proximal Reminders

5. 5 Mar 20 2001 A New Class of Context-Aware Apps Active Badge (Olivetti) ParcTabs (Xerox PARC) Cyberguide (Abowd et al)

6. 6 Mar 20 2001 Problems with State of the Art Coordinating all the context ­Will have many distributed, heterogeneous sensors, services, and devices out there ­Won't be designed a priori for interoperability Building robust context-aware apps ­Dynamically changing computing environment ­Sensors, services, and devices will vary in capability and availability Developers are forced to devote a lot of time and energy to non-application issues

7. 7 Mar 20 2001 Proposed Meta-Solution Shift burden of context-awareness from individual devices onto infrastructure ­Discovering, Acquiring, Transforming, Refining, Routing, and Reacting to Context Provide services that can be easily deployed everywhere and used by any device ­Have simple, uniform system interfaces ­Any device with a network connection can discover and use these services Abstract out messy details of sensors and services

8. 8 Mar 20 2001 Thesis Statement A set of core infrastructure services coupled with a specification language for stating context needs will reduce the effort in developing robust context-aware applications.

9. 9 Mar 20 2001 Sensor Management Service Services Proposed Solution Context Event Service Context Query Service Automatic Path Creation App Application Layer Context Layer Logical Sensor Layer Physical Sensor Layer Context Specification Language Agents Device Agents User Agents

10. 10 Mar 20 2001 Research Contributions A way of explicitly specifying context needs independently of specific sensors and services (+ ontology for doing it) Automatically assembling a dataflow from heterogeneous sensors and services at run time to satisfy context needs Seeing how far queries, events, and data flow processing can go for context

11. 11 Mar 20 2001 High-Level Plan (Mir Space Station View) Design Context Specification Language Design, implement, and deploy core context services Evaluate by implementing several simple- to-medium complexity apps on top Re-iterate design and implementation based on what was learned

12. 12 Mar 20 2001 Outline  Background and Motivation  Problem and Proposed Solution  Context Fabric Architecture  Evaluation  Related Work  Proposed Schedule  Summary

13. 13 Mar 20 2001 Context Specification Language [requestor identification] [requestor authentication] [requestor input] [context need]

14. 14 Mar 20 2001 Context Specification Language Need to express context about and relationships between People, Places, Things Predicates ­Identity (Who is…? What is…? Is with…?) ­Location (Near? Nearest? Distance? Path?) ­Activity (Is busy? Is in meeting? Current task?) ­Time (In past? In present? In future? On date?) Some of this vocabulary done by Schilit ­Implicitly encoded in his APIs ­One goal is to extend his work in spec language ­Another is to make it extensible for future context types

15. 15 Mar 20 2001 Context Specification Language Common parameters ­Max number of results wanted ­Return name ­Return data type (e.g. String, List, Table) ­Minimal probability of correctness desired ­Relevant sensor input requestor has Event parameters ­Event rate (e.g. at most 1 event per second) ­Event callback (e.g. RPC, socket port) ­Max number of events desired ­Granularity of change (e.g. 1 meter)

16. 16 Mar 20 2001 Context Event Service Context Query Service Uniform abstraction for apps ­Pass a Context Specification to Context Event or Context Query service ­Everything is handled by infrastructure Event and queries two sides of same coin ­Query is synchronous and one-time ­Event is asynchronous and repeating Bulk of work is done by Automatic Path Creation though

17. 17 Mar 20 2001 ZIPWeather GPSZIP CellGPS CellZIP Automatic Path Creation A way of assembling small services into larger, more interesting ones Operators Paths ZIPTheaters GPSZIP Weather GPSZIP

18. 18 Mar 20 2001 Automatic Path Creation Sensor Input ZIPTheaters GPS ZIP What are the nearby movie theaters? Sensor Input ZIPTheaters Cell GPSZIPCellGPS Sensor Input ZIPTheaters Cell Context Output CellZIP Movie Theaters Automatically assemble path based on resources Don't have to know specific sensors and services Easier to add new ad hoc functionality Will need a Service Description Language

19. 19 Mar 20 2001 Re-implement from ParcTabs (Schilit 1995) Device Agents – Bridge for physical devices into infrastructure – Holds context state about devices – Centralizes context to and from device (better for preferences and filtering) – Job is to handle all context issues that cannot / should not be handled on device itself User Agents – Holds context state about people – Preferences, location info, filters, etc Agents

20. 20 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Mobile PDA tour guide 1.Display map of local area 2.Display nearby points of interest 3.Bring up info on proximity 4.Display location of others 5.Capture history

21. 21 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database Palm Pilot Cell Phone Infrastructure

22. 22 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 1. Display map of local area 1. Turn on device, local GPS registers with Sensor Management Service Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

23. 23 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 1. Display map of local area 2. Start up tour guide application Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

24. 24 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 1. Display map of local area 3. Setup context query for changes in location (granularity depends on if indoors or outdoors) Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

25. 25 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 1. Display map of local area 4. Receive location updates Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

26. 26 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 1. Display map of local area 5. Display location updates Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

27. 27 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 2. Display nearby points of interest 1. Periodically query for nearby points of interest and Cache results (e.g. restaurants, landmarks, etc) Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

28. 28 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 2. Display nearby points of interest 2. Local Context Query Service delegates to Automatic Path Creation Service Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

29. 29 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 2. Display nearby points of interest 3. Automatic Path Creation Service takes GPS info and tries to answer it locally (fails). Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

30. 30 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 2. Display nearby points of interest 4. Local APC delegates to infrastructure APC Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

31. 31 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 2. Display nearby points of interest 5. Infrastructure APC looks for services that can provide location-based points of interest info or parts of this info Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

32. 32 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 2. Display nearby points of interest 6. Infrastructure APC returns relevant information (or "failure due to x" if it cannot) Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

33. 33 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 3. Bring up info on proximity 1. Setup proximity events on points of interest. Attach notes to event that display information about point of interest. Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

34. 34 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 4. Display location of others 1. Subscribe to location change events from others. Device Agent Device Agent Device Agent E Q E Q E Q Automatic Path Creation Service Discovery Palm Pilot E Q Cell Phone E Q

35. 35 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 4. Display location of others 2. Can't process locally, subscription request forwarded to mirroring Device Agent. Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

36. 36 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 4. Display location of others 3. Device Agent makes individual subscription requests. Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

37. 37 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 4. Display location of others 4. Individual device agents make a subscription request to their mirroring devices. Palm Pilot Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q E Q Cell Phone E Q

38. 38 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 4. Display location of others 5. Location change events are routed back to tour guide and displayed. Automatic Path Creation Service Discovery Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q

39. 39 Mar 20 2001 Context Fabric Example Context-Aware Tour Guide Palm Pilot Context Event Service Context Query Service GPS Automatic Path Creation Sensor Mgt Service Tour Guide App Map Database 5. Capture history 1. Just store all interesting events locally. Device Agent Device Agent Device Agent E Q E Q E Q Palm Pilot E Q Cell Phone E Q Automatic Path Creation Service Discovery

40. 40 Mar 20 2001 Outline  Background and Motivation  Problem and Proposed Solution  Context Fabric Architecture  Evaluation  Related Work  Proposed Schedule  Summary

41. 41 Mar 20 2001 Evaluation Iterative design process ­Work out designs, deploy, build apps, refine Initial evaluation apps will be existing ones ­Reimplement some existing context-aware apps ­Informally evaluate apps for robustness ­Informally evaluate apps for defaults (e.g. privacy policy) Continued evaluation will be novel apps ­Focus on robustness and defaults again ­Also get others to build context-aware apps

42. 42 Mar 20 2001 Related Work ParcTab System (Schilit 1995) ­Device Agents, User Agents, Active Maps ­Plan to re-implement these and new ones too Context Toolkit (Dey 2000) ­Widgets, Interpreters, Aggregators ­Much is subsumed in Automatic Path Creation Interactive Workspaces (Fox et al 2000) ­Paths path creation and blackboard architecture ­Self-describing and extensible events

43. 43 Mar 20 2001 How Context Fabric Differs Separate what context is needed from how it is processed via Context Spec Language Shift processing from static to dynamic ­Say what you need at design-time ­Figure out how to get it at run-time through Automatic Path Creation

44. 44 Mar 20 2001 Proposed Schedule Phase 1 (0-8 months / PARC) ­Initial design and implementation of Context Specification Language and core services ­Evaluate with simple context-aware apps Phase 2 (8-12 months) ­Re-iterate and incorporate what was learned ­Release source code ­Evaluate with medium complexity apps Phase 3 (12-18 months) ­Incorporate more feedback (smaller changes) ­Write dissertation

45. 45 Mar 20 2001 Exit Conditions and Deliverables Spec for Context Specification Language ­Basic ontologies for context ­Expressive enough for majority of context apps Working core services ­Context Query, Context Event, Automatic Path Creation, plus basic operators ­Service Description Language for APC operators Source code available for general use

46. 46 Mar 20 2001 Summary Infrastructure approach to context- awareness Context Specification Language for stating context needs Context Event Service, Context Query Service as primary APIs Automatic Path Creation for processing Evaluation focused on defaults and robustness

47. 47 Mar 20 2001 A New Cal Record!! Only one day to schedule quals… …through email… …with a professor at Stanford even!

48. 48 Mar 20 2001 Extra Slides

49. 49 Mar 20 2001 What is Context? Situated Action (Suchman) ­Responsiveness, improvisation to situation ­Actions are moment-by-moment, highly fluid on situation Activity Theory (Nardi) ­Subject, Object[ive], Operations, Artifacts, Env ­Activity defines context but it changes as list above changes ­Context depends on sensing and inferring, but difficult to know people or goals

50. 50 Mar 20 2001 What is Context? Provides fair criticisms on existing context- aware apps ­Response: Focus on the more routine and predictable actions ­Response: Focus on simple first-order approx ­Examples: Smoke alarms, Calendar reminders ­Response: Is occasionally wrong, but won't get better unless we do more research here ­Response: Infrastructure enables more research in this area

51. 51 Mar 20 2001 What is Context? Merging of virtual with physical and social ­Mostly implicit input, often secondary to task at hand People / Places / Things / Virtual / Time ­People: Location, Identity, Task, Affect ­Places: Activity, People, Temperature, Audio ­Things: Location, Identity ­Virtual: Services, Bandwidth ­Time: All of these Past, Present, and Future

52. 52 Mar 20 2001 What is Context? But it's a rathole to try to define it… ­Workflows? (Greenberg 2001) ­Intent and Desire? ­Natural Language Processing? ­What isn't context? Instead, operationalize it ­Distributed, multi-device, sensor-based ­Fairly well-defined and computable concepts (e.g. location and identity, but not intent or workflow)

53. 53 Mar 20 2001 Why Context-Awareness? Sensors are coming Increase channels of input / output Graphics Sound Mouse Keyboard

54. 54 Mar 20 2001 Why Context-Awareness? Skeptics toward the ubiquitous computing movement quickly point out that those involved in promoting Mark Weiser's initial vision are mainly interested in pushing technology… As a reaction against this fair criticism, many researchers spend countless hours trying to dream up the "killer application" that will cause everyone to adopt ubiquitous computing technology. I think this is the wrong goal to have in mind as a researcher for two reasons. First, we are not schooled in the practices to judge a market… Second, my interpretation of Weiser's vision is not that there would be a single, most compelling use of ubicomp technology, but rather that the totality of the experience would cause a significant paradigm shift in the way we as humans perceive our relationship to interactive computing. In short, Weiser argued for what I would like to call a "killer existence," in which many disparate devices would provide an array of services in ways that seamlessly complement our lifestyles. -- Gregory Abowd, 2001

55. 55 Mar 20 2001 Why Context-Awareness? SensorsComputers Context-Aware Computing Physical WorldVirtual World

56. 56 Mar 20 2001 Why Infrastructure Approach? Sharing of data ­Collaborative filtering, large data sets Sharing of processing power ­Heavy-duty processing on dedicated machines Sharing of sensors ­Sensors in the environment In-place evolution of services ­New algorithms, new features Supports heterogeneity ­Sensors, Devices, OS, Programming Langs

57. 57 Mar 20 2001 Why Infrastructure Approach? Building and evaluating now can ­Provide more design space for interaction ­Let us rapidly prototype and evaluate these apps ­Find problems now ­Develop the right mechanisms ­Evolve the right policies

58. 58 Mar 20 2001 Other Related Work Context Toolkit ­Widgets, Interpreters, Aggregators ParcTab system ­User agents, Device agents, Active Maps Interactive Workspaces ­Event heap, Path Creation MUSE ­Jini services, Bayesian reasoning

59. 59 Mar 20 2001 Some Sensors Temperature Pressure Humidity Soil makeup Acceleration Noise Material stress Biometric Motion Distance Touch Location Orientation Light

60. 60 Mar 20 2001 Related Work Context Toolkit Widgets, Interpreters, Aggregators Hardwired data flow Face Recognition Location Widget Location Widget In/Out Board Smart Card Reader ID to Name Interpreter Building Aggregator

61. 61 Mar 20 2001 Related Work ParcTab System Will need to re-implement parts of ParcTab system ­Device Agents / User Agents / Active Maps

62. 62 Mar 20 2001 Related Work Interactive Workspaces Event Heap ­Scaling of tuple spaces will be a problem ­Can use how events are described

63. 63 Mar 20 2001 Related Work MUSE Sensor management issues ­Sensor discovery (currently via Jini) ­Power consumption Probabilistically modeling sensor data ­Sensor fusion (via Bayesian nets and HMMs)

64. 64 Mar 20 2001 Rough Taxonomy of Context-Aware Apps Triggers Metadata Tagging Reconfiguration and Streamlining Input specification Presentation

65. 65 Mar 20 2001 Rough Taxonomy of Context-Aware Apps Triggers ­On X do Y ­"Notify doctor and nearby ambulances if serious health problem detected" (Citris) ­"Remind me to talk to Chris about user studies next time I see him" (Proxy Lady) Metadata Tagging ­"Where was this picture taken?" ­"Find all notes taken while Mae was talking" ­Memory prosthesis (Forget-me-not)

66. 66 Mar 20 2001 Rough Taxonomy of Context-Aware Apps Reconfiguration and Streamlining ­Telephone forwarding and Teleport (Active Badges) ­Turn off cell phone in theaters ­Automatically adjust brightness / volume ­Automatic file pre-caching (OceanStore) ­Select modes in multimodal interaction ­Multimedia/Bandwidth adaptation (Odyssey)

67. 67 Mar 20 2001 Rough Taxonomy of Context-Aware Apps Input specification ­Send mail to people in building (Geocasting) ­Print to nearest printer (ParcTabs) ­"Find gas stations nearest me" Presentation ­Current location (Cyberguide) ­Idle? (Yahoo Messenger, finger) ­Currently in? (In / Out board) ­Contextual info about objects (CoolTown) ­Proximate selection (ParcTab)

68. 68 Mar 20 2001 Context-Aware Mechanisms Events ­Something interesting has just happened ­e.g. "Larry has entered room 306" Queries ­Introspect current context state (discrete) ­e.g. "Where is Larry now?" Filters ­Modify events and queries based on preferences and privacy ­e.g. "Only show others if in office or not"

69. 69 Mar 20 2001 Are Queries and Events Enough? Triggers Metadata Tagging Reconfiguration Input Specification Presentation Query Query for relevant context state Query current context state before and while running Query current context state to fill in the blanks Query current state for more information Event On Event do Action On Event do Reconfigure On Event do Update Presentation

70. 70 Mar 20 2001 What I Will Not Be Working On Building a new service infrastructure Developing new sensors Extremely sophisticated forms of context ­Natural language processing ­User intention New algorithms for… ­Sensor / context fusion ­Machine learning ­Power consumption ­Just use algorithms that already exist!

71. 71 Mar 20 2001 What I Will Not Be Working On Sophisticated preferences ­Enable us to explore this space more Comprehensive security and privacy mechanisms ­Will implement basic preferences, filters, and access control ­One subgoal is to get the defaults right Rigorous user evaluations