1. Recent Work in Model-Based User Interfaces
Jeffrey Nichols
Lecture #13
05-830: Advanced User Interface Software

2. Last time… Model-based User Interfaces
Automatic generation of the user interface so the programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be longer and harder to generate than implementing the interface itself.
Interface builders turned out to be easier…

3. But work continued… Focus Changed A Couple Projects of Interest:
Task models were leveraged more
Design assistant aspect emphasized
A Couple Projects of Interest:
Trident
Mecano & Mobi-D
FUSE
AIDE

4. TRIDENT An interface design assistant Interesting features:
Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience, Technical Report RP , Facultes Universitaires Notre-Dame de la Paix, Institut d’Informatique, Namur, 1995.
An interface design assistant
Interesting features:
Knowledge-based approach (i.e. expert system)
Choosing Widgets
Doing Layout
Use of Task Models
Decides where separate windows are needed

5. Choosing Widgets Used a decision tree
Chose abstract interaction objects (AIO)
Similar to Brad’s Interactor Model
Lots of parameters
Continuous?
Capacity
Etc.

6. Choosing Layout Uses Right/Bottom Strategy
Next component is placed to the right or below the current component
Decision made by heuristics or designer

7. Right/Bottom Strategy

8. Windows from Task Models
Basically used for constructing wizard-like interfaces
What information should be on the first screen, etc.

9. What are task models, anyway?
Description of the process a user takes to reach a goal in a specific domain
Typically have hierarchical structure
Introduced by GOMS
Number of different task modeling languages
GOMS
UAN
ConcurTaskTrees

10. ConcurTaskTrees
Developed by Fabio Paterno et al. for the design of user interfaces
Goals
Graphical for easy interpretation
Concurrent model for representing UI tasks
Different task types
Represent all tasks, including those performed by the system

11. Task Building Process Three phases Temporal Relationships
Hierarchically decompose the tasks
Identify the temporal relationships among tasks at same level
Identify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate.
Temporal Relationships
T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion

12. Example
Note: First example is ambiguous

13. Another Example

14. Building/Editing Task Models
Tools are available
ConcurTaskTrees Environment
or Google for “ConcurTaskTrees”

15. Recent Systems XIML – eXtensible Interface Markup Language XWeb
Developed by the makers of Mecano/Mobi-D and Trident
Kitchen-sink language for modeling any part of the interface design process
XWeb
Now known as ICE – Interactive Computing Everywhere
ICrafter
A system for integrating user interfaces from multiple devices
Personal Universal Controller
My research…

16. XIML eXtensible Interface Markup Language
Designed by RedWhale Software
Intended to support the full lifecycle of interface building

17. XIML Requirements Central Repository of Data
For one user interface or many
Comprehensive Lifecycle Support
Abstract and Concrete Elements
Relational Support
Underlying Technology
XIML must be independent of particular tools

18. Models in XIML An XIML document can contain any type of model Task
Domain
User
Presentation
Dialog

19. Example Use for XIML
Multi-platform interface development

20. Status of XIML
Used by RedWhale Software to drive their interface consultant business
They have developed many tools
move interaction data to/from XIML
Leverage data in XIML to better understand various interfaces
Automate parts of the interface design process

21. Model-Based Interfaces for Control
XWeb
ICrafter
PUC

22. XWeb Work by Dan Olsen and group at BYU Premise:
“Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”
The web comes close to solving this problem, but is interactively insufficient.

23. XWeb Protocols Based upon the architecture of the web
XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1

24. XWeb & XTP CHANGE message (similar to GET in HTTP)
Sequence of editing operations to apply to a sub-tree
Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location

25. Platform Independent Interfaces
Two models are specified
DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”
Interactors are somewhat like abstract interaction objects
Atomic
Numeric
Time
Date
Enumeration
Text
Links
Aggregation
Group
List

26. XWeb Example
DataView

27. XWeb Example
XView

28. XWeb Example
Interface

29. Other XWeb Details
Has simple approach for adjusting to different screen sizes
Shrink portions of the interface
Add additional columns of widgets
Also capable of generating speech interfaces
Based on a tree traversal approach like Universal Speech Interfaces

30. ICrafter
Part of the Interactive Workspaces research project at Stanford
Main objective:
“to allow users of interactive workspaces to flexibly interact with services”
Contribution
An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.
In practice, much of the interface generation is done by hand though automatic generation is supported.

31. ICrafter Architecture

32. How is aggregation accomplished?
High-level service interfaces (programmatic)
Data Producer
Data Consumer
The Interface Manager has pattern generators
Recognize patterns in the services used
Generate interfaces for these patterns
This means that unique functionality will not be available in the aggregate interface

33. Automatic Generation in ICrafter

34. Manual Generation in ICrafter

35. Personal Universal Controller
My work with Brad
Problem:
Appliance interfaces are too complex and too idiosyncratic.
Solution:
Separate the interface from the appliance and use a device with a richer interface to control the appliance:
PDA, mobile phone, etc.

36. Idea Control existing appliances Generate multi-modal interfaces
Specifications
Control
Feedback
Control existing appliances
Generate multi-modal interfaces

37. - Interface Generators Architecture
- Appliance Adaptors
- Comm. Protocol
- Specification Lang.
- Interface Generators
Architecture
XML-based

38. Language Design Approach Create reference interfaces
AIWA Shelf Stereo
AT&T Telephone/Answering Machine
Test interfaces with subjects
Users twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfaces
Analyze interfaces for functional information
Be careful of saying “same manufacturer’s appliances”

39. Language Elements State Variables and Commands Label Information
Represent functions of appliance
State variables have types
Boolean, Enumeration, Integer, String, etc.
Variables sufficient for most functions but not all
“seek” button on a Radio
Label Information
One label not suitable everywhere
The optimal label length changes with screen size
Speech interfaces may benefit from pronunciation and text-to-speech information

40. Language Elements, cont.
Group Tree
Specify organization of functions
We use n-ary tree with variables or commands at leaves
Need a place for things like “no specific layout information” related work or new high-level spec slide

41. Language Elements, cont.
Dependency Information
Formulas that specify when a variable or command is active in terms of other state variables
Equals, Greater Than, Less Than
Linked with logical operators (AND, OR)
For example,
<and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals> </and>

42. Interface Generators Generators for Two Modalities Graphical Speech
Implemented for PocketPC in Java 1.1
Uses dependency information to generate panel structure of interface
Speech
Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]
Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)

43. Graphical Interface Generator
Focuses on panel structure of user interface
Small groups of controls have basic layouts
Complexity comes from structure of groups
Structure can be inferred from dependency info!
decomposed

44. Inferring Structure
Find sets of variables that are “mutually exclusive”
Every variable in a set will never be active at the same time as a variable in another set
Create structure with sets, using overlapping panels

45. Choosing Panel Types a) b) c)
full screen
tabbed
partial screen

46. Making the Interface Concrete
Finish conceptual layout
Choose controls (decision tree)
Choose row layouts
(one column, two column, etc.)
Allocate space
Examine panel contents and choose sizes
Instantiate and place controls

47. Generating Speech Interfaces
Automatically build USI tree from dependencies
Allows verbal navigation of functional groups
Automatically generate grammar for parser
Phrases for query and control
“What is playmode?”
“Set playmode to play”
“play”
Automatically generate language model and pronunciation for recognizer