Continuing Work in Model-Based User Interfaces Jeffrey Nichols : Advanced User Interface Software

Last time… Model-based User Interfaces  Automatic generation of the user interface so the programmer won’t do a bad job.  Dialog boxes are relatively easy to generate  The full application interface is hard to generate  Abstract descriptions of the interface can be longer and harder to generate than implementing the interface itself.  Interface builders turned out to be easier…

Research continued… Multiple models were integrated  Relational models were developed to explicitly link other kinds of models Data models became more detailed Task models became very important

Research continued… Fragmented into two different approaches  Software engineering approach (early 90’s-) Very detailed models to improve overall design process Intelligent design assistants instead of automatic generation Significant use of task models  Ubiquitous computing approach (2000-) Tons of “invisible” processors that perform tasks for us UIs for these processors are presented on other devices (mobile phone, PDA, speech, etc.) Impossible to manually build user interfaces for every combination

What are task models, anyway? Description of the process a user takes to reach a goal in a specific domain Typically have hierarchical structure  Introduced by GOMS Number of different task modeling languages  GOMS  UAN  ConcurTaskTrees

ConcurTaskTrees Developed by Fabio Paterno et al. for the design of user interfaces Goals  Graphical for easy interpretation  Concurrent model for representing UI tasks  Different task types Represent all tasks, including those performed by the system Used almost universally by new model-based systems

Task Building Process Three phases  Hierarchically decompose the tasks  Identify the temporal relationships among tasks at same level  Identify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate. Temporal Relationships  T1 [] T2 - Choice  T1 ||| T2 - Interleaving  T1 |[]| T2 - Synchronization  T1 >> T2 - Enabling  T1 []>> T2 - Enabling with Information Passing  T1 [> T2 - Deactivation  T1* - Iteration  T1(n) - Finite Iteration  [T1] - Optional  T – Recursion

Example Note: First example is ambiguous

Another Example

Building/Editing Task Models Tools are available  ConcurTaskTrees Environment or Google for “ConcurTaskTrees”

Software Engineering Approach Lots and lots of systems!  MasterMind  Angel Puerta’s work at Stanford Mecano, Mobi-D, XIML  UIML  Cameleon Project Teresa  USIXML  etc…

MasterMind One of the first system to integrate multiple models together

Mobi-D Set of tools to support a clearly defined development cycle Uses a series of different models Explicit relationships that specify how the models are related to each other Explicit interaction between end users and developers

Mobi-D Models User-task Model  Describes tasks the user performs and what interaction capabilities are needed Domain Model  Models of the entities that are manipulated in an interface and their properties Dialog Model  Describes the human-computer conversation at a low level Presentation Model  Specifies how the interaction objects appear in all of the different states of the interface. Relations  Describes how each of the models relate to each other  Tasks/Domain, Dialog/Presentation, Tasks/Dialog, etc.

Mobi-D Process  Elicit user tasks  Start with informal description and then convert to outline (basis for task models in Mobi-D)  More formal task analysis methods could probably be used  User-task and domain modeling  Skeleton domain model is built from task outline  Developer refines model  Explicit methods for generalizing pieces of model for reuse in other interface designs  Presentation and Dialog design  Decision support tools provide recommendations from model and interface guidelines (if available)  System steps through task model and helps designer build final interface By carrying the task model through the process, Mobi-D’s designers believe users can provide more useful feedback

Mobi-D Discussion Provides assistance rather than automating design Recommendations do not limit flexibility, but organize the design process For usability engineers, not everyday users  Benefits come from reuse among small projects or for managing interaction data from a large project  Models can be large and appear to require significant effort to develop Spawned a profitable company that does UI work

XIML eXtensible Interface Markup Language Based on Mobi-D work Supports full development lifecycle Used by RedWhale Software to drive their interface consultant business  They have developed many tools move interaction data to/from XIML Leverage data in XIML to better understand various interfaces Automate parts of the interface design process

Example Use for XIML Multi-platform interface development

Other Software Engineering Systems UIML (  Originally a research project at Virginia Tech, now being developed commercially by Harmonia  Goal is platform independent language for describing UI  Early versions were not very platform independent  Recent versions using task models to automatically generate parts of the old language that were not platform independent Teresa (  Transformation Environment for inteRacti  Tool for taking ConcurTaskTrees models, building an abstract interface, and then building a concrete interface on multiple platforms. USIXML (  Many of the same features of XIML  Novel aspect is the use of graph structure for modeling relations (seems very complex)

Ubiquitous Computing Approach “Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.” -Dan Olsen (Xweb paper) The web comes close to solving this problem, but is interactively insufficient.

Ubiquitous Computing Approach There are two problems here:  Infrastructure issues How do devices communicate? How do devices discover each other?  User Interfaces issues Are devices described sufficiently to build a good UI? How are interfaces generated? How can one interface be created for controlling combinations of related devices?

Infrastructure Issues Possible to investigate these issues without automatically generating UIs Being addressed by lots of systems  Commercially UPnP, JINI, Salutation, HAVi  Research Speakeasy (PARC), many others… Most systems that address the UI issues also have some infrastructure component

Systems addressing UI issues XWeb  Now known as ICE – Interactive Computing Everywhere ICrafter  A system for integrating user interfaces from multiple devices Supple  A system for automatically generating interfaces with a focus on customization/personalization. Personal Universal Controller  My research…

XWeb Work by Dan Olsen and group at BYU Premise:  Apply the web metaphor to services in general  Support higher levels of interactivity

XWeb Protocols Based upon the architecture of the web  XTP Interaction Protocol  Server-side data has a tree structure  Structured Data in XML  URLs for location of objects xweb://automate.home/lights/livingroom/ xweb://automate.home/lights/familyroom/-1

XWeb & XTP CHANGE message (similar to GET in HTTP)  Sequence of editing operations to apply to a sub-tree Set an attribute’s value Delete an attribute Change some child object to a new value Insert a new child object Move a subtree to a new location Copy a subtree to a new location

Platform Independent Interfaces Two models are specified  DataView – The attributes of the service  XView – A mapping of the attributes into high-level “interactors” Atomic  Numeric  Time  Date  Enumeration  Text  Links Aggregation  Group  List

XWeb Example DataView

XWeb Example XView

XWeb Example Interface

Other XWeb Details Has simple approach for adjusting to different screen sizes  Shrink portions of the interface  Add additional columns of widgets Also capable of generating speech interfaces  Based on a tree traversal approach like Universal Speech Interfaces

ICrafter Part of the Interactive Workspaces research project at Stanford Main objective:  “to allow users of interactive workspaces to flexibly interact with services” Contribution  An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.  In practice, much of the interface generation is done by hand though automatic generation is supported.

ICrafter Architecture

How is aggregation accomplished? High-level service interfaces (programmatic)  Data Producer  Data Consumer The Interface Manager has pattern generators  Recognize patterns in the services used  Generate interfaces for these patterns This means that unique functionality will not be available in the aggregate interface

Automatic Generation in ICrafter

Manual Generation in ICrafter

Supple Eventual goal is to support automatic personalization of user interfaces Treats generation of interfaces as an optimization problem Can take into account usage patterns in generation Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.

Modeling Users with Traces Supple uses traces to keep a usage model  Sequences of events:  Interfaces are rendered taking the traces into account (though traces are not required)  Trails are segmented at interface close or reset

Generating with Optimization Uses a branch-and- bound search to explore space of alternatives  Guaranteed to find an optimal solution

Optimization Equation Cost of navigation between subsequent controls in a trace Device-specific measure of how appropriate a control is for manipulating a variable of a given type Total cost for one user trace Total cost of all traces

Screenshots

Personal Universal Controller My work with Brad Problem:  Appliance interfaces are too complex and too idiosyncratic. Solution:  Separate the interface from the appliance and use a device with a richer interface to control the appliance: PDA, mobile phone, etc.

Control Feedback Approach Specifications Appliances Mobile Devices Use mobile devices to control all appliances in the environment Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

The PUC System Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

Research Approach 1.Hand-design remote control interfaces 2.Determine functional information needed from appliances to design user interfaces 3.Design language for describing appliance functions 4.Build interface generators for multiple platforms 5.Perform user studies to evaluate the interface generators

Language Design Informed by hand-designed interfaces  What functional information was needed to create interfaces? Additional Requirements  Support complete functionality of appliance  No specific layout information  Only one way to specify anything Full documentation available at:

Language Elements Elements  State variables & commands  Labels  Group tree  Dependency information Example media player specification  Play, stop, pause, next track, previous track  Play list

Language Elements, cont. State Variables and Commands  Represent functions of appliance  State variables have types Boolean, Enumeration, Integer, String, etc.  Variables sufficient for most functions but not all e.g. “seek” button on a Radio

Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” A group of semantically similar labels Different lengths Information for different modalities

Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” A group of semantically similar labels Different lengths Information for different modalities

Language Elements, cont. Group Tree  Specify organization of functions  We use n-ary tree with variables or commands at leaves  Also used for specifying complex types Lists Unions

Language Elements, cont. Group Tree  Specify organization of functions  We use n-ary tree with variables or commands at leaves  Also used for specifying complex types Lists Unions

Language Elements, cont. Dependency Information  Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR)  Allows feedback to user when a function is not available

Graphical Interface Generator Rule-based approach  Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface  Small groups of controls have basic layouts  Complexity comes from structure of groups  Structure can be inferred from dependency info!

Generation Process 1.Determine conceptual layout  Infer panel structure from dependencies using “mutual exclusion” property  Choose controls (decision tree)  Choose row layout (one column, two column, etc.) 2.Allocate space  Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

Generation Process 1.Determine conceptual layout  Infer panel structure from dependencies using “mutual exclusion” property  Choose controls (decision tree)  Choose row layout (one column, two column, etc.) 2.Allocate space  Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

Generation Process 1.Determine conceptual layout  Infer panel structure from dependencies using “mutual exclusion” property  Choose controls (decision tree)  Choose row layout (one column, two column, etc.) 2.Allocate space  Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

Generation Process 1.Determine conceptual layout  Infer panel structure from dependencies using “mutual exclusion” property  Choose controls (decision tree)  Choose row layout (one column, two column, etc.) 2.Allocate space  Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

Generation Process 1.Determine conceptual layout  Infer panel structure from dependencies using “mutual exclusion” property  Choose controls (decision tree)  Choose row layout (one column, two column, etc.) 2.Allocate space  Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

Generation Process 1.Determine conceptual layout  Infer panel structure from dependencies using “mutual exclusion” property  Choose controls (decision tree)  Choose row layout (one column, two column, etc.) 2.Allocate space  Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems Without layout fixing rules With layout fixing rules

Controlling Appliances We have built adaptors for many actual appliances  Sony Digital Camcorder  Windows Media Player  Axis UPnP Pan & Tilt Camera  Lutron Lighting  X10 Lighting  Audiophase Shelf Stereo  AudioReQuest MP3 player  GM Vehicle Information System  GM Vehicle Climate Control Written specifications for others  Elevator  Telephone/Answering Machine  GM Navigation System  Several Alarm Clocks