1. Lecture 14: Model-based tools: Creating the UI Automatically
Brad Myers
Advanced User Interface Software
Spring 2017
© Brad Myers

2. Model-Based Tools Overview
Programmer describes the operation of the system or the user interface in a specification language = the "model".
model is a high-level description
usually declarative (listing parts and relationships)
System automatically creates the interface
Uses a low-level toolkit for the widgets
© Brad Myers

3. Goals:
High-level description of an interface is easier to write than low-level toolkit code
Automatic generation may produce better UIs than programmers
Allow separation of UI design (embodied in rules) from UI contents (supplied by the programmer)
Support dynamic creation of objects
define templates or prototypes
Increase re-use since design rules shared by multiple applications
Tools can reason over the specification to produce extra stuff:
Automatic generation of help, undo, etc.
Transform interface into different but functionally equivalent interface
Enabling and disabling of widgets
Enforcement or checking of design guidelines- consistency, completeness
Enforces consistency since rules will pick similar objects for similar situations
Automatic adjustment to different screen sizes, etc., since rules can take this into account
Automatic analysis for quality
NGOMSL analysis : GLEAN (Kieras, UIST'95)
© Brad Myers

4. Overview, cont.
Related to the "Declarative" approach discussed in previous lecture
but here system has some intelligence or knowledge so less has to be specified by the programmer.
Different types:
Dialog box creators: Mickey, DON, Jade (lots of others)
Representations of the full UI: ITS, UIDE, Humanoid, MasterMind,
Later: (XML-based) ConcurTaskTrees, Xweb, ICrafter, Supple, PUC
© Brad Myers

5. Dialog Box Creators Easiest part of the UI to create
Given a list of the contents, automatically:
choose widgets:
specify type of desired input: string = text input field number = slider one-of-many = radio buttons or pop-up options many-of-many = check boxes or checks in a menu commands = menu
© Brad Myers

6. Dialog Box Creators, cont.
arrange widgets
based on look-and-feel guidelines
where OK goes
which commands go in which menus
based on good graphic design principles.
set variables
to reduce the number of callbacks necessary
© Brad Myers

7. Example: Mickey
Programmer specifies UI by putting special comments in a Pascal file.
Uses the Apple Macintosh guidelines
Pre-processor to parse the Pascal code and generate the Macintosh resources.
Maps Procedures into Menu items.
If parameter is one of a standard set, pops up appropriate dialog box or waits for input
File to be read, file to be written
New point, line or rectangle
© Brad Myers

8. Mickey, cont. Variables:
Enumerated types mapped to check lists separated by lines. Sets the variables when changed.
Enumerated types with 2 choices mapped to name changes
Booleans: single checked items
© Brad Myers

9. Jade
Brad Vander Zanden and Brad A. Myers,  "Automatic, Look-and-Feel Independent Dialog Creation for Graphical User Interfaces," Proceedings SIGCHI'90: Human Factors in Computing Systems.  Seattle, WA, April 1-5, pp ACM DL Reference
"Judgment-based Automatic Dialog Editor"
Given a textual specification of just the contents and their types, creates a dialog box
Separately specify which look-and-feel (not part of the specification)
Defines mapping from types to widget selection
Graphic design rules for "nice" layout
© Brad Myers

10. DON (Won Chul Kim & Foley, InterCHI'93, pp. 430-437)
ACM DL Reference
Ultimate in dialog box layout
Kim's PhD thesis
Works with OpenLook and devGuide
Allows interactive designer guidance (preferences) on sizes, layout, widget choice, etc.
Can also choose among proposed layouts
Sophisticated 2-D layout
Tries to balance dialog box
Groupings of related items
Effective use of white space (even margins, minimize wasted space)
Generates multiple designs and uses an evaluation metric to choose.
© Brad Myers

11. Don, pictures
PDF, with other pictures
© Brad Myers

12. Generating Full UI
These next tools require a specification of the full UI
Usually have rule-based components
Specifications are in a special language
© Brad Myers

13. Interactive Transaction System (ITS)
Bennett, et.al., UIST'89 pp
Wiecha, et.al. CHI'89, pp
Wiecha, et.al., ACM TOIS, 8(3), Jul'90, pp
Goal: capture designers knowledge as style rules
So unlike other systems, designer is required to edit the rules, not just the specification
All of UI must be created by editing the rules
no interactive editing of generated interface (since then the knowledge about why the generated interface wasn't good enough would be lost)
Like dialog-box systems, separate specification of content and style
Style-independent tags associated with content
"Style expert" programs the style for each tag
Styles include both output (display) and input (interaction techniques) specifications
© Brad Myers

14. ITS, cont.
Can handle dialog boxes, forms, node-link diagrams, kiosk frames, etc.
Used for a number of internal IBM applications
Used for all the information services at Expo'90 in Spain
Information, maps, restaurant reservations, etc.
IBM researchers and content experts were in Spain for months
Evaluation
+ Full representation of design may increase re-use
- Design specification ends up containing many specific "hacks" used to achieve specific effects in single interfaces
- Complex formal language for specification and rules
Pictures from ITS, TOIS, 8(3), pp. 213, 215, 217
© Brad Myers

15. ITS, pictures
PDF
© Brad Myers

16. The User Interface Design Environment (UIDE)
Foley, et. al. CHI'88, pp
Foley, et. al. IEEE Software, Jan'89, 25-32;
Sukaviriya, et. al. InterCHI'93, pp
Long-term project of Foley's at George Washington and Georgia Tech
Ended about 1994 when Foley left
A loose collection of separate implementations:
IDL's transformations
DON dialog boxes (described above)
Sukaviriya's animated help
Martin Frank's work (EET in Event-Based lecture)
- etc.
© Brad Myers

17. UIDE
Programmer defines Knowledge-base "schemas" describing parts of the interface:
Objects: in a class, sub-class inheritance hierarchy
Actions: what can be done in the interface
Pre-conditions and post-conditions are in a very limited language
counting, Booleans, simple tests
used for testing enabled and explaining why
Transformations change among equivalent UIs:
e.g. Currently-selected obj <=> currently-selected cmd
performed based on pre-, post-conditions
© Brad Myers

18. Pictures from UIDE
© Brad Myers

19. UIDE, cont. Automatic generate help for why commands are not available
Sukaviriya, et. al. AVI’94, Pages:   
Animated help provides animations as a tutorial
Determines what needs to be done to demonstrate action
Sequence of actions
E.g. bring windows to the front, create an object, etc.
© Brad Myers

20. UIDE Evaluation + Support for more than dialog boxes
- Pre and post condition language is weak
can't express the test "if the selected object is a polygon..."
- Model language is a different, difficult language to learn
© Brad Myers

21. Humanoid Szekely, et. al. UIST'90, pp. 1-9
Szekely, et. al. CHI'92, pp
Szekely, et. al. InterCHI'93, pp
High-level UIMS for Manufacturing Applications Needing Organized Iterative Development
Model application data and interaction similar to UIDE
Model whole application: semantics + interface
© Brad Myers

22. Humanoid, cont.
System picks generic interaction techniques immediately using "templates"
Designer can refine interface iteratively by creating more specific sub-classes:
Single-Command-Input-With-Alternatives
Single-Command-Input-With-Few-Alternatives
Allows exploration with incomplete designs
Evaluation
+ Much richer specification language than UIDE
- More complex to define interfaces (more to learn)
but interactive tools help
© Brad Myers

23. Humanoid Pictures Pictures from Humanoid, CHI'93 pp. 384
© Brad Myers

24. MasterMind
Neches, et. al. ACM 1993 Intelligent User Interfaces Workshop, pp
Models Allowing Shared Tools and Explicit Representations to Make Interfaces Natural to Develop
Idea: combine UIDE and Humanoid
Support entire life-cycle: early conceptual design through maintenance
Knowledge base is shared among all tools
Knowledge base serves as an integrating framework for various tools at design time and run time.
Spent a lot of time negotiating on how to combine models
Lots of different parts to the model
Personnel and coordination problems in doing the research
Used Amulet!
© Brad Myers

25. MasterMind
One of the first system to integrate multiple models together
© Brad Myers

26. Mastermind Pictures
© Brad Myers

27. Research continued… Fragmented into two different approaches
Software engineering approach (early 90’s-)
Very detailed models to improve overall design process
Intelligent design assistants instead of automatic generation
Significant use of task models
Ubiquitous computing approach (2000-)
Tons of “invisible” processors that perform tasks for us
UIs for these processors are presented on other devices (mobile phone, PDA, speech, etc.)
Impossible to manually build user interfaces for every combination
© Brad Myers

28. What are task models, anyway?
Key part of many current HCI approaches
Description of the process a user takes to reach a goal in a specific domain
Typically have hierarchical structure
Number of different task modeling languages
GOMS, CogTool
UAN
ConcurTaskTrees
© Brad Myers

29. Hierarchical Task Model
H. Rex Hartson and Pardha S. Pyla. The UX Book: Process and Guidelines for Ensuring a Quality User Experience. Morgan Kaufmann / Elsevier p. 218
© Brad Myers

30. Software Engineering Approach
Commercial work
“Model-based design”
Example: BPMN “business process modeling notation”
Business experts should be able to author models
Converted into code to support the process (requires people)
Keynote at ICSE’08: Herbert Hanselmann: Challenges in Automotive Software Engineering
“Model-based design (MBD) of functional behaviour has been a big help in the recent past”
*** move earlier ***
© Brad Myers

31. ConcurTaskTrees
Developed by Fabio Paterno et al. for the design of user interfaces
Goals
Graphical for easy interpretation
Concurrent model for representing UI tasks
Different task types
Represent all tasks, including those performed by the system
© Brad Myers

32. Building/Editing Task Models
Tools are available
ConcurTaskTrees Environment
or search for “ConcurTaskTrees”
© Brad Myers

33. Mobi-D Angel Puerta, IUI’97
Set of tools to support a clearly defined development cycle
Uses a series of different models
Explicit relationships that specify how the models are related to each other
Explicit interaction between end users and developers
© Brad Myers

34. XIML eXtensible Interface Markup Language XIML.org
Based on Mobi-D work
Supports full development lifecycle
Used by RedWhale Software to drive their interface consultant business
They developed many tools
Move interaction data to/from XIML
Leverage data in XIML to better understand various interfaces
Automate parts of the interface design process
© Brad Myers

35. Other Systems UIML (http://www.harmonia.com/)
Originally a research project at Virginia Tech, then developed commercially by Harmonia
Goal is platform independent language for describing UI
Early versions were not very platform independent
Later versions using task models to automatically generate parts of the old language that were not platform independent
Teresa (
Transformation Environment for inteRacti
Tool for taking ConcurTaskTrees models, building an abstract interface, and then building a concrete interface on multiple platforms.
USIXML (
Many of the same features of XIML
Novel aspect is the use of graph structure for modeling relations (seems very complex)
© Brad Myers

36. Ubiquitous Computing Approach
“Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is interactively insufficient.
© Brad Myers

37. Systems addressing UI issues
XWeb
Now known as ICE – Interactive Computing Everywhere
ICrafter
A system for integrating user interfaces from multiple devices
Supple
A system for automatically generating interfaces with a focus on customization/personalization.
Personal Universal Controller
Jeff Nichol’s research
© Brad Myers

38. XWeb Work by Dan Olsen and group at BYU E.g. UIST’2000, pp.191 - 200
Premise:
Apply the web metaphor to services in general
Support higher levels of interactivity
© Brad Myers

39. XWeb Example
Interface
© Brad Myers

40. ICrafter
Part of the Interactive Workspaces research project at Stanford
Ponnekanti, et. al. Ubicomp’2001
Main objective:
“to allow users of interactive workspaces to flexibly interact with services”
Contribution
An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.
In practice, much of the interface generation is done by hand though automatic generation is supported.
© Brad Myers

41. iCrafter UI Top: automatic; bottom: custom
people/fox/static/pubs/pdf/c08.pdf
© Brad Myers

42. Supple Support automatic personalization of user interfaces
Treats generation of interfaces as an optimization problem
Can take into account usage patterns in generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
© Brad Myers

43. Modeling Users with Traces
Supple uses traces to keep a usage model
Sequences of events:
<interface element, old value, new value>
Interfaces are rendered taking the traces into account (though traces are not required)
Trails are segmented at interface close or reset
© Brad Myers

44. Generating with Optimization
Uses a branch-and-bound search to explore space of alternatives
Guaranteed to find an optimal solution
© Brad Myers

45. Screenshots
© Brad Myers

46. Personal Universal Controller
Jeff Nichol’s PhD work
Problem:
Appliance interfaces are too complex and too idiosyncratic.
Solution:
Separate the interface from the appliance and use a device with a richer interface to control the appliance:
PDA, mobile phone, etc.
Goal: Generate high-quality UIs
© Brad Myers

47. Approach
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Appliances
Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation
© Brad Myers

48. Properties of PUC Language
State variables & commands
Each can have multiple labels
Useful when not enough room
Typed variables
Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure
Groups
© Brad Myers

49. Dependency Information
Crucial for high-quality interfaces
Expressed as <active-if> clauses
Operations:
Equals, Less-Than, Greater-Than
Combined Logically
AND, OR
Used for:
Dynamic graying out
Layout
Widget selection
© Brad Myers

50. Generating Consistent UIs
Personally consistent
Reduce learning time
Add unique functions
© Brad Myers

51. Generating Combined UIs
For multiple appliances, such as home theaters
Specify content flow
Combined controls
© Brad Myers

52. Summative Study
Compared PUC to manufacturer’s interfaces for HP and Canon printer/fax/copiers
PUC twice as fast, 1/3 the errors
Consistent: another factor of 2 faster
© Brad Myers

53. Model-based systems advantages/disadvantages
+ Separate specification of UI from content
+ Automatic reformatting, retargeting for different platforms, customization to users
+ May allow programmers (non-experts) to write specification and have a good UI automatically created
But this didn’t really work out
– Result is often unpredictable
– Often UI can be worse UI than hand-drawn
– Sometimes model is larger than the code it would replace
– Model often in a different language that must be learned
© Brad Myers