Ubiquitous Computing Toolkits Tara Matthews Advance User Interface Software Fall 2004

Goals of Ubicomp Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals Scalable interfaces multiple devices Ubicomp Infrastructure inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals Scalable interfaces multiple devices inter-device communication multiple users CSCW (not covered) Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop Physical UIs / Mobile augmenting surrounding environment (other lectures) provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Context-Aware Calmness more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users Peripheral Displays aware of user’s interruptiblity

Toolkits for Goals Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity Interruptibility (not covered)

Toolkits for Goals Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity Goals accomplished? Evaluation

Toolkit Categories Ubicomp infrastructure Context-aware Peripheral displays Evaluation Not covering: CSCW (separate field) Interruptibility (no toolkits yet) Physical UIs (covered in Jack’s lecture) Mobile Devices (covered in an unclaimed lecture)

Ubicomp Infrastructure iROS Stanford Interactive Room Operating System iStuff Physical application toolkit build on iROS Aura CMU “Distraction-Free Ubiquitous Computing” project Distributed services infrastructure Gaia University Of Illinois at Urbana-Champaign Infrastructure for Active Spaces

iROS Interactive Room (iRoom) Operating System Definition: A ubiquitous computing environment where groups come together to collaborate on solving problems (not a toolkit) Contains: Embedded devices: large display screens, table- top display, & other output devices Mobile devices: mobile devices brought into space can be used with embedded facilities

iROS Goals Facilitate common Interactive Workspace usages observed in iRoom prototype workspace: Move data between devices Control devices & apps from other devices Coordinate apps, including ones not written to work together Additional goals: Provide for heterogeneity of devices in workspace, & new devices being added over time Allow easy integration of legacy devices Robust to transient failures Portable across installations Actions appear instantaneous to users

iROS 3 sub-systems: Event Heap: stores and forwards events Data Heap: facilitates data movement in an interactive workspace iCrafter: provides a system for service advertisement and invocation, along with a user interface generator for services.

Event Heap Tuple space model Associated memory: content is addressable using a pattern Distributed systems based on blackboard model are easily created in tuple space Decouples apps in space & time Added semantics: Apps not designed to work together, so need Self-describing tuples, Flexible typing, Typed tuples Apps can snoop or interpose Tuple sequencing Data shared & can build up Tuple expiration Benefits Anonymous coordination Interposability Snooping

Event Heap Figure: Example operation showing placed events, and using template events for matching

Data Heap Provides type & location independent storage Machine independent: Don’t need explicit location of data Type independent: If a set of type converters are available, system automatically transforms the data Figure: Shows automatic retrieval of a Word document in PostScript form

iCrafter Universal control system: control anything from anywhere Services advertise how they can be controlled System returns appropriate interface per device Interfaces can range from fully custom to dynamically generated

iCrafter 1. USER REQUEST The user requests an interface for a service. The appliance sends this request to the Interface Manager, along with appliance description information.

iCrafter 2. GENERATOR SELECTION Upon receiving the request, the Generator Selector selects an appropriate generator based on the service(s) selected and the appliance description.

iCrafter 3. GENERATOR PROCESSING The Generator Processor runs the selected generator with access to all context (appliance, service, environment), which outputs a UI description.

iCrafter 4. USER INTERFACE RENDERING The UI description is sent to the UI Renderer on the appliance, which renders the interface and handles user interactions, including service invocations over the EH.

iStuff Toolkit to prototype physical UIs Flexible, lightweight components Protocol independence Platform independence with cross-platform capabilities Ease of integration with existing applications Support for multiple simultaneous users Built on iROS iStuff wireless devices (e.g., iButton, iSlider, iLight) send/receive events to/from Event Heap via a proxy PatchPanel translates events (e.g., iButton: light on | light off) Apps get events from devices via PatchPanel + Event Heap

iStuff Architecture Event Heap PatchPanelProxy iStuff Device Transceiver iStuff component Application Wireless connection TCP/IP connection

iStuff Devices iSpeaker iLight iBuzzer Anoto Pen iMouse iMike iDog X10 iStylus iSlider RF iButtons Input Output

Aura CMU “Distraction-Free Ubiquitous Computing” theme: Mobility: allow users to move computational tasks from one place to another Maximize use of available resources Minimize user distraction & drains on attention Personal information aura: spans wearable, handheld, desktop, & infrastructure devices Requires new system design: hardware, network, OS, middleware, & UI support

Aura Research Framework Users Tasks Services OS / Network Physical Devices Research Examples UI / Agents / Speech Task-driven computing Rapid service configuration Energy-aware adaptation Multi-fidelity computation Nomadic data access Intelligent networking Power-aware devices Wearable computers

Aura Example Scenario Fred prepares presentation & is late Fred gets PDA & Aura transfers slides to it Finishes slides on PDA walking to meeting Aura finds location of meeting from calendar & uploads slides to projection machine Aura recognizes unfamiliar meeting attendees faces & skips budget slides

Aura Architecture User tasks are represented as set of distributed services Database query interface to access distributed services Easily search mixed environments for needed services A key service is People Location Service Environments self-monitor & re-negotiate task support given runtime variation of capabilities & resources Can detect when task requirements (e.g., min response time) aren’t met, & search & deploy alternative configurations to support task Uses software architecture models (graph of components & connectors) for runtime adaptability

Aura Architecture Task Manager: tracks user context, environment, & task changes Context Observer: provides info on physical context & reports relevant events Task & Environment Managers Environment Manager: handles access to distributed services Suppliers: provide services for tasks (text editing, video playing,...)

Privacy in Aura Aura services depend on user location info Introduces privacy concerns Location policies: define who gets location info, where, when, and at what granularity e.g., “Bob is allowed to know Alice’s location when she is in NSH” Similar to Bell Labs’ Houdini system’s “rules” Not based on how user’s actually disclose location

GAIA Infrastructure for Active Spaces, UIUC Like Aura: Distributed services architecture accessed via structured queries Main difference: Focus on user customization of apps based on resources in current space

Toolkit Categories Ubicomp infrastructure Context-aware Peripheral displays Evaluation

Context Context is key to ubicomp environment that reacts correctly to user’s current situation Definition: Info used to characterize the situation of a person, place or object relevant to the interaction between a human & a computational service Primary context types: Identity (who), Location (where), Time (when), Activity (what) Secondary context types: e.g., for identity: address, phone number, birthdate, etc

Context-Aware Applications Definition: Uses context to provide relevant info and/or services to the user, where relevancy depends on the user’s task 3 categories: Present information & services to users  e.g., nearby wireless networks, directions Automatically execute services  e.g., phone vibrates when in meeting Tag info with context for later retrieval  e.g., class lectures and notes

Toolkits for Context 3 main approaches: 1. Widget model Context Toolkit Dey, Abowd, & Salber, 2001 Based on architecture of GUIs 2. Distributed services model Context Fabric Hong & Landay, 2001 Based on client-server dialog 3. Blackboard model iRoom Winograd, 2001 Based on artificial intelligence apps (Engelmore, 1988)

Widget Model: Context Toolkit Operating system view Abstraction to hardware (sensors); data is secondary Similar to GUI input handling: uses widget abstractions for handling context input Uses 3 abstractions: widgets, interpreters & aggregators

Benefits Separates semantics from low-level input sensor handling Apps can focus on context, not sensors or analysis of sensor input Allows re-use of context widgets, interpreters & aggregators

Drawbacks Tight coupling – less robust to component failure Single-manager (Discoverer) control – less flexible

Context Toolkit Architecture

Widgets Encapsulate info about a single piece of context (e.g., location or activity) Hide details of underlying sensors from apps Provide customizable & reusable building blocks of context sensing Provide historical context info

Widgets Include optional services, or actuation of output in environment Like Interactors: responsible for input & output e.g., light widget could sense light level & adjust lights accordingly Get context by subscription or polling Callback methods used to notify subscribers

Aggregators Widgets + ability to aggregate context info Collect sensory info about an entity (person, place, thing, etc.) from multiple sources into one widget Hide more complexity about context-sensing mechanisms by combining multiple sensors Enable maintainability and efficiency

Interpreters Abstract sensor info Use lower level sensory information to deduce higher level info e.g.,identity + location + sound sensors  “is there a meeting?”

Putting an App Together Discoverer enables apps to locate and subscribe to context components Distributed infrastructure uses p2p comm.

Distributed Services Model Database view (vs. OS view of CTK) Data is primary, hardware is secondary Focus on how data is modeled, distributed, protected, & used Middleware technologies that can be accessed through a network Example: infrastructure=Internet; service=DNS Any kind of device or application can use context gathering & processing services by adhering to predefined data formats & network protocols

Benefits Interoperable: independent of hardware platform, OS, & programming language Decoupling: Sensors, services, & apps can be changed w/o affecting each other Simpler devices: sensors, processing power, data, & services w/in infrastructure shared

Drawbacks Applications must poll for data (proactively) Servers are specialized per sensor

Context Fabric Includes 3 pieces 1. Distributed data model of people, places, things  each device has a space: private data goes to local space  multiple views of context (mine, yours, theirs) 2. Context Specification Language (like SQL)  apps specify what context info they need in uniform way  e.g., "What are the nearby movie theaters?” 3. Context service as API (per device)  interpret CSL queries and provide context info

Privacy in Context Fabric Decentralized architecture allows it to capture, store, process, & share in privacy- sensitive manner Capture, store, & process data on user’s device Provides mechanisms for end-user control of sharing Plausible deniability built-in e.g., if request for context info denied, system can reply “unknown”

Aura Also follows distributed services model Uses SQL-like interface to access a set of distributed contextual services Handles privacy through user-set policies

Blackboard Model Communication goes through a centralized server Components post messages shared blackboard subscribe to get posted messages matching specified pattern Route by matching message content to subscriber's pattern Pattern matching uses AI (often apply sophisticated inference procedures to logical representations) Direct communication between components simulated by including an identifier for path endpoints as a field in message Announce-listen style Components that provide services periodically post events Component that uses the data listens to these events, & if one does not appear within the expected time can initiate restart operations

Benefits Ease of configuration Robustness to component failure When component fails, only communication link broken is from it to blackboard Dependent components can detect failure and call for restart Simplicity provided by a uniform communications path (to the blackboard) No complex protocol for finding ports or resources, establishing connections

Drawbacks Communication less efficient Each communication requires 2 hops General message structure not optimized for particular data or interaction protocol

iRoom & iStuff Event Heap: the blackboard & central communication server

Review of Context Approaches Widget Model – CTK Widget architecture where Discoverer acts as a manager of context widgets Distributed Infrastructure Model – ConFab, Aura, Gaia Client-server architecture where client apps communicate directly w/ services using structured language Blackboard Model – iRoom, iStuff Central server architecture where apps communicate only w/ server & events are routed using subscriber pattern matching

Toolkit Categories Ubicomp infrastructure Context-aware Peripheral displays Evaluation

Peripheral Displays Provide awareness with min attention Separate from primary task Important to ubicomp vision of many devices to 1 user Non-focal display not used unless providing peripheral info Enable you to monitor many info sources while maintaining a calm environment But, only calm if designed to manage attention they attract

Why is creating PDs hard? Need to abstract info to be glance-able Need mechanisms for dynamically managing attention PDs attract: Deciding attention levels to attract (notification levels) Displaying info appropriately (transitions) Peripheral Display Toolkit (PTK) Supports these 3 key issues in PD creation

Example PTK Applications Remote Activity Social Guitar Audio Monitor Motion Monitor Remote Awareness Display Bus Displays Bus Mobile Bus LED Instant Messenger Status IM Picture FrameSocial Guitar Bus LED BusMobile Orb showing remote activity

PTK Architecture 1. Support for managing impact on human attention using abstraction, notification levels, and transitions 2. Simplified code design and code re-use 3. Library of common PD components Output Transition Input Notification Map Abstractor

Architecture Diagram

Library Components Input audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port

Input audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port Output ticker text, Ambient Orb, Phidgets Library Components

Input audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port Output ticker text, Ambient Orb, Phidgets Abstractors motion, people counting, voices, phone ringing Library Components

Input audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port Output ticker text, Ambient Orb, Phidgets Abstractors motion, people counting, voices, phone ringing Notification exact match, threshold, contains, degree of change Library Components info urgencyuser attention

Toolkit Categories Ubicomp infrastructure Context-aware Peripheral displays Evaluation

3 tools for evaluating people in natural settings from Intille et al.: 1. Context-aware experience sampling based on GPS or other sensors e.g., ask questions when user goes to store 2. Ubiquitous sensing system that detects environmental changes e.g., tape-on touch, proximity, & light sensors 3. Image-based experience sampling system i.e., take a picture & ask user about it later

Summary: Ubicomp Goals Scalable interfaces multiple devices inter-device communication multiple users Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Calmness more info w/o overwhelming users aware of user’s interruptiblity

Summary: Ubicomp Toolkits Ubicomp infrastructure iROS, iStuff, Aura, Gaia Context-aware Context Toolkit, Context Fabric, iRoom Peripheral displays Peripheral Display Toolkit Evaluation Intille’s toolkit

Thanks!

CSCW Groupware Architectures Phillips, W.G., Architectures for Synchronous Groupware, Tech. Rep.. Greenberg, S. and Roseman, M., Groupware Toolkits for Synchronous Work. In: Beaudouin-Lafon, M. (Ed.), Trends In CSCW'99, No. 7 in Trends in Software, John Wiley & Sons, New York, NY, USA, ch. 6, pp. 135– 168. Roseman, M. and Greenberg, S., GROUPKIT: a groupware toolkit for building real-time conferencing applications. In: Proceedings of the conference on Computer-supported cooperative work, ACM Press, pp. 43–50.