Confab Tutorial Jason I. Hong Chris Beckmann Jeff Heer Alan Newberger G r o u p f o r User Interface Research University of California Berkeley

Feb Vision “A hospital Mirror World has a software version of every patient, doctor, bed, room – and every abstract entity that’s important: cash in the bank, drugs on order, and so on. Through permanent sensors and ordinary terminal- based record-keeping, the Mirror World reflects the real one.” – David Gelernter, Mirror Worlds

Feb Vision What if… – …we could create a world model that describes salient aspects of the world in real-time? – …everyone could contribute to this world model in the same way that everyone can contribute to the global World Wide Web? Such a World Model would greatly facilitate construction of context-aware apps – App developers wouldn’t have to hand craft each model – Streamline sharing of context data – Uniform façade around diverse sensors and software APIs

Feb Tutorial Outline Context Data Model Programming Model Liquid Distributed Querying The Messy Details

Feb Context Data Model High-Level Rationale

Feb Context Data Model High-Level Rationale Web (Global Scope) (Context for people) Web Service (Specialized API) (Global Scope) (Context for computers) Train Info Board (Local Scope) (Context for people) Unified API (Network effects) (Limited Scope) (Context for computers)

Feb Context Data Model High-Level Overview Beacon Web Scraping Sinks Data Stores Sources Context-Aware Applications Context Data Sensors, Beacons, Databases, Web pages Auto Diary HVAC Context Data Layer Sources Sinks Spaces Sensor Public Display Manual Input Context Browser

Feb Context Data Model Alice’s InfoSpace LocActivit y Health Room 525’s InfoSpace Person Device PDA- 1138’s InfoSpace Owner

Feb Context Data Model Division of Responsibilities InfoSpace Server InfoSpace Tuple Analogous to web servers Manages a collection of InfoSpaces Unit of administration Unit of deployment Analogous to a web site / homepage Represents context data about an entity Represents zone of protection Manages collection of context tuples Unit of ownership and addressing Analogous to individual web page Represents single piece of context data Contains privacy preferences and metadata Unit of storage

Feb Context Data Model InfoSpaces Distributed world model – Each with partial and incomplete knowledge of world – Each with a different perspective of the world Represents three different things – Context data about an entity “My name is John”, “I am hungry” – Context data perceived by that entity “I am with Alice”, “I am in room 525” – Context data queried by that entity “Carol tells me that the dog is in the kitchen” Managed by individual represented or by admin – Like a homepage

Feb Context Data Model InfoSpaces TupleSpace meets Web TupleSpace – A shared data space – add(), remove(), query(), subscribe(), unsubscribe() – Complexity shifted into data model and query language Web – Leverages existing technology (ex. firewalls) – Leverages well-understood models for administration, deployment, authoring, and programming – End-user mental model – Independent deployment & anarchic scalability [Fielding]

Feb Example Model InfoSpaces Alice’s InfoSpace Tuples Out-Log In-Log Out-Subscriptions In-Subscriptions Properties Policies

Feb Tuples Represent a discrete piece of context data Contains: – Context data – Metadata – History of that data – Privacy information

Feb Context Data Model Tuples StaticDynamic Intrinsic Name Height Extrinsic Room 525 part of Soda Room Temperature Hospital Bed Empty Alice is in Room 525 People in Room 606

Feb Querying XPath is a language for addressing parts of an XML document – Think of an XML document as a tree-structure – – – &sortorder=descending – &num=2 Snapshot – Get current state

Feb Querying XPath Explorer

Feb Operators – Small components for transforming data – Extensibility without having to modify the main code In-operators – Check Privacy Tag Out-operators – Add Privacy Tag – Clear Sources – Sort On-operators – Garbage Collection – Periodic Report

Feb Operators Example Data-Flow Alice’s InfoSpace In OperatorsOut Operators Tuple HTTP Front-end

Feb Sink-Side Overview Confab Client Alice’s InfoSpac e Bob’s InfoSpac e Room 22 InfoSpac e InfoSpace Server HTTP Front-end Active Properties Listeners

Feb Sink-Side ConfabClient ConfabClient – Java client-side API for accessing InfoSpaces – add(), remove(), query(), subscribe(), unsubscribe() ActiveProperties “lederer.location” “lederer.activity” “lederer.temp” OnDemandQuery PeriodicQuery Subscription “606” “Napping” “98.6”

Feb Source-Side Simulator

Feb Putting it all together In / Out Board for Room 410 Approach 1 – Query each individual Approach 2 – Query the room “lederer.loc” “klemmer.loc” “mattkam.loc” PeriodicQuery …… “room410.occupants”PeriodicQuery

Feb The Messy Details Download these packages JDK 1.4 Tomcat Web Server – 4.0/release/v4.1.18/bin/ CVS – we like TortoiseCVS, Ant Build System – Pageant and Putty Public-Key –

Feb The Messy Details SourceForge SourceForge Open Source Repository – Create account at – Create and upload your public-key – Join Confab dev-team if you want to CVS commit No team t-shirts yet – CVS checkout latest snapshot

Feb

Feb Privacy Layer perspective – Each layer responsible for security and privacy between layers Dataflow perspective – Tuples contain data about usage – Digital rights management

Feb Related Work – Semantic Web / DAML Semantic Web has no story for – Individuals managing their data – Handling sensor data and dynamic updates – Where specific pieces of data live Confab is simpler – Complexity of Semantic Web is huge barrier to entry – Start simple

Feb Related Work – Context Toolkit Focus first on the data model rather than sensors Early mapping of sensor to ontology Per sensor managment

Feb Related Work – ParcTab System Confab is an evolution of ParcTab system

Feb Related Work – EventHeap / iRoom

Feb Related Work –

Feb Related Work –

Feb Motivation Modern computers divorced from our reality – Unaware of who, where, and what around them – Mismatch between our expectations and functionality – Also limits what we can do with computers Computers have extremely limited input – Aware of explicit input only – A lot of effort to do simple things (or to remember) Context-Aware Computing – One line of ubiquitous computing research – Making computers more aware of the physical and social situations they are embedded in

Feb Examples of Using Context Context TypesExisting ExamplesHuman Concern Room ActivityAuto Lights On / OffConvenience Personal Identity & Time File SystemsFinding Info TimeCalendar RemindersMemory Activity Finding Info Safety Time Location Activity Health Alert Tag Photos History Identity Proximity Efficiency Service Fleet Dispatching Context Types Potential ExamplesHuman Concern

Feb Technology Trends Sensors – GPS, Active Badges, Active Bats – Smart Dust – Cameras and microphones Recognition algorithms – MSR Radar location from – Smart Floor footstep force Wireless technologies – Bluetooth, , cell phone

Feb A New Class of Context-Aware Apps Active Badge (Olivetti) ParcTabs (Xerox PARC) Cyberguide (Abowd et al)

Feb A Computational View of Context Context as a strategy for building apps Increasing the number of input channels into the computer – Pushing towards implicit acquisition of data Creating better models – Pushing towards the physical and social Using the input and models in useful ways – Proactively taking predictable and meaningful actions – Tagging other information for future lookup – Passing on more information to people

Feb A Computational View of Context Autonomy Sensing Lights FileSystem Calendar Tag Photos Health Realtime Dispatching Pervasiveness Inference Fusion Models

Feb Two Problems with Context- Awareness Scalability – Lots of people, places, things, and sensors – Over long periods of time – Over large geographic distances – Sharing resources (sensors and data) Privacy – Tremendous source of valid criticism – Need architecture and mechanisms to safeguard personal data and make it easy for people to manage

Feb Research Goals and Solution Overview Provide network-oriented set of abstractions, mechanisms, and programming model Scalability – Data-oriented P2P repositories called information spaces – Different infospaces federate when needed Privacy – Provide suite of mechanisms for app developers – Based on Fair Information Practices and Information Asymmetry

Feb Talk Overview  Motivation  Research Overview  Confab Architecture – Scalability  Confab Architecture – Privacy  Status  Context + Whisper thoughts  Context + SpeakEasy thoughts

Feb Architectural Abstractions Information Spaces – P2P TupleSpace repositories of context data and operators – Associated with entities (people, places, things) – Somewhat similar to web servers and home pages Context Data – Representation for context data Operators – Reusable and composable code operating on data Context Queries / Notifications – Simple API for accessing context

Feb Architectural Sketch Information Spaces Carol's InfoSpace (Desktop) Information Spaces Carol's InfoSpace (PDA) Soda 525 InfoSpace (Server)

Feb Architectural Sketch Context Data Loc Act Loc Context Data Information Spaces

Feb Architectural Sketch Operators Context Data Information Spaces TransFilter Log Operators

Feb Architectural Sketch Context Queries Loc Context Data Information Spaces TransFilter Operators Query Loc Trans

Feb Architectural Sketch Context Notifications Context Data Information Spaces Operators Notification (Standing Query)

Feb Architectural Sketch Peering of Information Spaces Carol's Context when Mobile Carol's Context in Room 525 Context = Set of Available Info Spaces

Feb Emergency Response Scenario Part of a suite of context-aware apps under development for fire or earthquake situations Keep track of people in a building – Allow building managers to check if a building is clear in the event of an evacuation – Allow firefighters to check where people were Provide reasonable privacy protection – People don't like to be tracked – Emergency situations relatively rare

Feb Emergency Response Scenario Registering with the Building's InfoSpace Building InfoSpace Carol's InfoSpace Smart Dust User="Carol" Location="525 Soda Hall" Time="Apr 12 1:05PM" Access Control User="Carol" Location="5th floor" Age="37 seconds" Send location info Logging User="Carol" Location="in" Age="37 seconds" Blurring

Feb Emergency Response Scenario Querying during an Emergency Building InfoSpace Carol's InfoSpace Smart Dust User="Carol" Location="5th floor" Age="37 seconds" User="Carol" Location="525 Soda Hall" Time="Apr 12 1:05PM" NotificationLogging User="Carol" Location="525 Soda" Age="7 seconds"

Feb Layers of InfoSpaces and Context Data Physical Logical View My Location on PDA My Location on PC My Location to Strangers My Location to Friends My Location to Family

Feb Scalability Recap Architecture analogous to web – Information spaces are like web servers – Information spaces contain context data – Context data is eventually consistent (helps availability) Differences from web architecture – Each device contains an information space (so devices can access context even w/o net access) – Information spaces contain reusable operators for manipulating and protecting context data

Feb Talk Overview  Motivation  Research Overview  Confab Architecture – Scalability  Confab Architecture – Privacy  Status  Context + Whisper thoughts  Context + SpeakEasy thoughts

Feb Privacy Philosophy Fair Information Practices Notice Choice Onward Transfer Access Security Data Integrity Enforcement

Feb Privacy Philosophy Information Asymmetry “In all of human history, no government has ever known more about its people than our government knows about us. And in all of human history, no people have ever been anywhere near as free.” (Brin)

Feb Some Desired Privacy Features Intentional ambiguity – "Where is Victoria?" "Chez Panisse" -> "Berkeley" -> "CA" – Give different answers depending on requestor Plausible deniability – "Is Adam busy?" "Yes" or "Unknown" according to prefs Risk Avoidance – "Mark does not trust this person / infospace" Tracking – Who has my data? What are they doing with it? – (Also a reverse-privacy issue?)

Feb A Privacy Design Space Legal Social Economic Technology Detection Avoidance Prevention CollectionSecond UseAccess Themes for Minimizing Asymmetry Data Lifecycle Anonymization Pseudonymization P3P RBAC Location Support Privacy Mirrors Wearables User Interfaces for Feedback, Notification, and Consent Goal: Provide reusable mechanisms that can populate this design space

Feb Privacy Trust Model Optimistic – I trust you and your current infospace – Make it easy for others to do "the right thing" [tm] Pessimistic – I don't trust you or your current infospace – Modify the data assuming you will do "the wrong thing" (more blurring or watermarking) – Or don't send the data to you at all Make it easy to support spectrum of trust models between full optimistic and full pessimistic

Feb Two Privacy Mechanisms Operators Privacy Tags – Preferences for how personal data should be used – "Don't forward to anyone else" – "Don't fuse with other pieces of data" Garbage collectionRemove or aggregate old data BlurringIncrease ambiguity Access ControlCheck authorization LoggingDetection FiltersRemove certain data

Feb Talk Overview  Motivation  Research Overview  Confab Architecture – Scalability  Confab Architecture – Privacy  Status  Context + Whisper thoughts  Context + SpeakEasy thoughts

Feb Status Still in early-to-mid phases – Currently developing initial implementation – JDK, JXTA (Java P2P), XML – Possibly also WSDL, SOAP Target applications – SpeakEasy (PARC) – Suite of Emergency Response apps – Possible Educational Technology apps "Metrics" – Types of and effectiveness of apps that can be built – Ease of adoption – Robustness

Feb The Ultimate Metric

Feb Some Context + Whisper Thoughts Use location + activity to help determine level of security – Within "safe" boundaries use low security – Within "unsafe" boundaries switch to high security, provide more feedback, and avoid risky situations (talking to strange computers) Boundaries can be based on: – People nearby (Social) – Activity – Location (Physical) Use contextual information from sensors and other sources to help determine these boundaries

Feb Some Context + SpeakEasy Thoughts Useful context for components – History of usage / Inferred patterns of usage – Location of component Useful context for people – Location of person – Personal history of usage / Inferred patterns – Shared history of usage (how others have used) – Activity ie "It looks like you're doing a presentation" Make it easy, or automate some things How well can you guess activity from simple data? How well can you do it over time?

Feb Q & A Focus, Jason, focus! Jen Mankoff Asst Prof Berkeley Privacy is good here, but be careful not to fall into the systems tarpit. Bill Schilit Intel Labs Seattle Co-director

Feb Q & A Agree with Bill do I, beware the dark side of systems you must! Yoda Jedi Master Kickass Dude

Feb Q & A Good work, Jason, I think you deserve a raise! James Landay Assoc Prof Berkeley My Advisor This party's started! Mace Windu Jedi Master Also a Kickass Dude

Jason I. Hong G r o u p f o r User Interface Research University of California Berkeley Thanks to: DARPA Expeditions PARC Intel Fellowship NSF ITR Yoda Context the circumstances in which an event occurs; a setting; to join; to weave

Feb Q & A Maybe privacy won't be a large issue in the future. Very difficult to say because of the tradeoffs in value, safety, convenience. One way of evaluating is to describe the design space, and show how your work makes it easy to build in that space.

Feb Q & A But do we really need ubicomp at all? And if so, how do we build and evaluate it so that it's socially relevant and meaningful? Maybe context itself isn't really the issue, because activity orders and delineates what is and isn't relevant at any point.

Feb Functional Requirements Context Acquisition – Getting the data from a variety of sources Context Modeling – Representing the data Context Storage and Dissemination – Storing the data – Making the data available when it is needed Context Usage – Using the data in a program

Feb Context Data Problem: how to represent context data? Entities – Like nouns, people, places, and things Attributes – Like adjectives or properties, key-value pairs Relationships – How one entity relates to another entity Aggregates – Actions, Groups of people

Feb Context Data Name="Location" Value="Room 525" Schema="Building:Room" Metadata= Time=" " Time-to-Live="60sec" Source="SmartDust" Name="Device" Value= Schema="Device" Entity Attribute Relationship

Feb Key Architectural Abstractions Information Spaces – Repositories of context data and operators Context Data – Representation for context data Operators – Composable code operating on context data Context Queries / Notifications – Simple query language (like SQL for DB) – Push / Pull semantics

Feb Information Spaces Problem: where to store context data? Information Spaces analogous to web servers – Have a unique name – Have an owner – Contain multiple (and not necessarily related) pieces of data – Can get / put pieces of data (given security and privacy prefs)

Feb Operators Problem: how to manipulate context data in a reusable manner? Chainable Operators Data-type ConversionEx. Celsius -> Farenheit FusionRefine same data type CompositionMerge different data types Garbage collectionRemove or aggregate old data BlurringIncrease ambiguity Access ControlCheck authorization LoggingDetection FiltersRemove certain data

Feb Context Queries Problem: how to use context data?

Feb Related Work Context Toolkit EventHeap ParcTab infrastructure

Feb Existing Examples of Using Context Context TypesExisting ExamplesHuman Concern Room ActivitySmoke AlarmSafetyRoom ActivityAuto Lights On / OffConvenienceObject IdentityBarcode ScannersEfficiency Personal Identity & Time File SystemsFinding InfoTimeCalendar RemindersMemory

Feb Potential Examples of Using Context Existing ExamplesContext Types Potential ExamplesHuman Concern ActivityConvenience ActivityFinding Info IdentityMemory Identity & TimeSafety TimeEfficiency Identity Time Location Proximity Activity History … Health Alert Auto Cell Phone Off In Meetings Service Fleet Dispatching Tag Photos Proximal Reminders

Feb Defining Context Abowd & Dey / Moran & Dourish "Any information that can be used to characterize the situation of an entity, where an entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and the application themselves. Context is typically the location, identity, and state of people, groups, and computational and physical objects." (Abowd and Dey) "Context refers to the physical and social situation in which computational devices are embedded" (Moran and Dourish)

Feb Defining Context Distributed Cognition Distributed cognition – Need to go beyond physical attributes (ex. temp) – Look at “state of digital resources, people’s concepts, task state, social relations, local work culture” (Kirsh) – Model key attributes and deep structure of whole system (individuals, offices, social structs, work practices) Problems – What are the key attributes? – How to represent?

Feb Defining Context Situated Action Situated action – Actions are fluid, moment-by-moment, improvised, often unplanned, and highly context- dependent – “[T]he context in which actions take place is what allows people to find it meaningful” (Dourish) Problems – Very high-level form of context – Can low-level computer-based context be useful? – Also, how does this really help us build systems?

Feb Defining Context Phenomenology Phenomenology – Reality consists of objects and events as they are perceived in human consciousness and not of anything independent of human consciousness. – Meaning (and hence context) arises from the ways in which we engage with and act within the world Problems – Need this level of sophistication to make progress? – How does this help us build systems? – Very wide chasm between philosophy and practice

Feb Defining Context My Perspective Point #1 – Not clear if we need a solid definition – Operating systems and Artificial Intelligence Point #2 – Let's treat it like "information" – Shannon treated it from a mechanical perspective (i.e. transmission) made great inroads – We are still debating the meaning of "information" – But now we can do it electronically Let's treat context from computer perspective – Let designers define context app-by-app – Provide generic reusable mechanisms (like DB)

Feb Privacy Privacy is a relatively new concept in society, and is “ultimately a psychological construct, with malleable ties to specific objective conditions” (Grudin) – Convenience, Safety, Efficiency – Ex. Credit cards and cell phones Open access to online calendars for efficiency and awareness (Palen)

Feb Designing Context-Aware Systems Minimize automatic actions – Probably cost-to-benefit via decision theory (value, error, correctness) Provide feedback – What is being captured? – Why did the system do that? Feed-forward – If you do that, then the system will do this Confirmation – The system just did the following action Endpoint – Context for people or context for computers?

Feb Vision Context-Aware Computing Today

Feb Vision Context-Aware Computing in the Future January

Feb Example Model Organizing End-User Devices, Services And Applications Sensors and Beacons (Mobile and Infrastructure) Data Sources (Personal, Group, Public) ??? January

Feb Example Model Organizing End-User Devices, Services And Applications Sensors and Beacons (Mobile and Infrastructure) Data Sources (Personal, Group, Public) ???

Feb Example Model Organizing Context Data Model Layer Sources Sinks

Feb Example Model Organizing Confab Client Alice’s InfoSpac e Bob’s InfoSpac e Room 22 InfoSpac e InfoSpace Server HTTP Front-end Active Properties

Feb Example Model Active Properties “scott.location” “scott.activity” “current-device.room”

Feb Example Model Division of Responsibilities InfoSpace Server InfoSpace Tuple Analogous to web servers Manages a collection of InfoSpaces Unit of administration and deployment Unit of deployment Analogous to a web site / homepage Represents context data about an entity Represents zone of protection Manages collection of context tuples Unit of ownership and addressing Analogous to individual web page Represents single piece of context data Contains privacy preferences and metadata Unit of storage

Feb Example Model Evolution of Context-Aware Systems

Feb Example Model A Predicted Evolution of Context-Aware Systems Web (Global Scope) (Context for people) Web Service (Global Scope) (Context for computers) Train Info Board (Local Scope) (Context for people) Unified API (Global Scope) (Context for computers) (Network effects) Restricted Scope (Global Scope) (Context for computers) (Network effects)

Feb Example Model A Predicted Evolution of Context-Aware Systems Train Info BoardSensor InputManual Input

Feb Example Model Physical, Logical, and View Sinks Data Stores Sources Context-Aware Applications Context Data Sensors, Beacons, Databases, Web pages

Feb Example Model Intrinsic and Extrinsic Context Loc

Feb Berkeley CS InfoSpace Server Example Model Single InfoSpace Server Alice’s InfoSpace Room 525’s InfoSpace HTTP Front-end In-Operators Out-Operators On-Operators Tuple InfoSpaceAccess

Feb Example Model Context Data Model Alice’s InfoSpace Room 525’s InfoSpace LocActivit y PDA- 1138’s InfoSpace Person Device Owner Health

Feb Example Model InfoSpaces Alice’s InfoSpace MotionGPS Health Monitor Service Heartbeat Personal Loc Triggers Auto Diary Calorie Tracker

Feb Example Model InfoSpaces Activit y Alice’s InfoSpace Room 525’s InfoSpace LocActivit y PDA- 1138’s InfoSpace Person Device Owner Health InfoSpac e Context Data Op

Feb Example Model Adding Data Loc Fusion Active Badge Wireless Triangulation Alice’s Laptop’s InfoSpace Alice’s Laptop’s InfoSpace Loc.Tri Alice’s InfoSpace Alice’s InfoSpace Loc. Active- Badge Loc.Tri SUBSCRIBE Loc.* SUBSCRIBE Loc.* SUBSCRIBE Loc

Feb Example Model Adding Data Loc Fusion Active Badge Wireless Triangulation Alice’s Laptop’s InfoSpace Alice’s Laptop’s InfoSpace Loc. Active- Badge POST Loc.Tri POST Loc.Tri Alice’s InfoSpace Alice’s InfoSpace Loc. Active- Badge Loc.Tri POST

Feb Example Model Transforming Data Acquaintance’ s InfoSpace Acquaintance’ s InfoSpace Type=“Location” User=“Alice” Location=“Berkeley, CA” Time=“Oct 06 1:05 PM” Pref=“Do not forward” Time-to-Live=“1 week” Type=“Location” User=“Alice” Loc=“525 Wozniak Hall” Time=“Oct 06 1:05 PM” Time-to-live=“Forever” Building’s InfoSpace Alice’s InfoSpace Type=“Location” User=“xyzzy” Loc=“5th floor Wozniak Hall” Time=“Oct 06 1:05 PM” Pref=“Do not forward” Pref=“Emergency use only” Time-to-Live=“1 hour”

Feb Address Name Occupied Hospital Phone# Name Patient Doctor Temperature Room Number Room

Feb Example Model Physical, Logical, and View Hospital's InfoSpace Room 525's InfoSpace Occupied Temperature Room Number Room 527's InfoSpace Occupied Temperature Room Number Dr. X's InfoSpace Room Phone# Name Room Doctor Patient Y's InfoSpace Heart Rate Room Address Name Patients

Feb <ContextTuple dataformat=" datatype="location.room" description="Represents location of an entity" entity-name="Doctor X" timestamp-created="2002.Dec.02 14:28:39 PST"> <Source datatype="location.room" link=" source="Active Badge" timestamp="2002.Dec.02 14:28:39 PST" out-tid="3F8B4528" in-tid="22A4610F" value="525" />

Feb Example Model InfoSpaces and Tuples Room 525's InfoSpace Occupied Temperature Room Number Dr. X's InfoSpace Activity Room Phone# Name Power Monitor Personal Diary Context Sources Context Sinks Context Data Model Dr. X

Feb DBCA

Feb ACF AccessSecond Use Alice's Location Alice's Location

Feb B A