1. Past, Present and Future of User Interface Software Tools
Brad A. Myers, Scott E. Hudson, and Randy Pausch
Human-Computer Interaction Institute
School of Computer Science
Carnegie Mellon University

2. Introduction User Interface Software Tools
Help developers design and implement user interfaces
Focus on Tools, but influenced by future UIs
Today’s tools are highly successful
Window Managers, Toolkits, Interface Builders ubiquitous
Most software built using them
Are based on HCI research Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, pp
Future tools must be different

3. Talk Outline Historical Perspective Future Prospects and Visions
What worked
What didn’t catch on
Why
Lessons Learned
Future Prospects and Visions
UI Trends that will require new tools
Important issues

4. Historical Perspective
Themes
Address the useful & important aspects of UIs
Tools that succeeded helped (just) where needed
Threshold / Ceiling
Threshold = How hard to get started
Ceiling = how much can be achieved
Path of Least Resistance
Tools influence user interfaces created
Predictability
If not predictable, then not accepted by programmers
Moving Targets
Changing user interface styles makes tools obsolete

5. What Worked Window Managers and Toolkits Event Languages
Graphical, Interactive Tools
Component Architectures
Scripting Languages
Hypertext
Object Oriented Programming

6. Window Managers
Multiple (tiled) windows in research systems of 1960’s: NLS, etc.
Overlapping introduced in Alan Kay’s thesis (1969)
Smalltalk, 1974 at Xerox PARC
Successful because multiple windows help users manage scarce resources:
Screen space and input devices
Attention of users
Affordances for reminding and finding other work

7. Toolkits A collection of widgets Toolkits help with programming
Menus, scroll bars, text entry fields, buttons, etc.
Toolkits help with programming
Help maintain consistency among UIs
Key insight of Macintosh toolbox
Path of least resistance translates into getting programmers to do the right thing
Successful partially because address common, low-level features for all UIs
Address the useful & important aspects of UIs

8. Event Languages Create programs by writing event handlers
Many UIMSs used this style
Univ. of Alberta (1985), Sassafras (1986), etc.
Now used by HyperCard, Visual Basic, Lingo, etc.
Toolkits with call-backs or action methods are related
Advantages:
Natural for GUIs since generate discrete events
Flow of control in user’s hands rather than programmer’s
Discourages moded UIs
Won’t work well in future

9. Graphical Interactive Tools
Create parts of user interface by laying out widgets with a mouse
Examples: Menulay (1983), Trillium (1986), Jean-Marie Hullot from INRIA to NeXT
Now: Interface Builders, Visual Basic’s layout editor, resource editors, “constructors”
Advantages:
Graphical parts done in an appropriate, graphical way
Address the useful & important aspects of UIs
Accessible to non-programmers
Low threshold

10. Component Architectures
Create applications out of components which are separately developed and compiled
In UI software, each component controls an area of the screen
Example: drawing component handles picture inside a document
Invented by Andrew research project at CMU (1988)
Now: OLE, OpenDoc, ActiveX, Java Beans
Address the useful & important aspects of UIs
Just the “glue” to hold together components

11. Scripting Languages First GUIs used interpreted languages
Smalltalk, InterLisp
Rapid development, supports prototyping
Low threshold
Then C and C++ became popular
Now, bringing back advantages in scripting languages
tcl/tk, Python, perl
Visual Basic, Javascript
But language must contain general-purpose control structures

12. Hypertext
Ted Nelson named it in 1965 and developed Hypertext system at Brown University
Important systems: NLS (1967), Hyperties (1986)
World-Wide Web
Phenomenal success due to:
Ease of use of Mosaic browser
Support for embedded graphics
Support for easy authoring
Low threshold both for authoring and viewing

13. Object Oriented Programming
Success of OO owes much to UI software field
Popularized by Smalltalk
GUI elements (widgets) seem like objects
Have state, accept events (messages)
Rise parallels GUIs
C++ with Windows 3.1
Java for behaviors in WWW

14. What Hasn’t Caught On User Interface Management Systems
Formal Language-Based Tools
Constraints
Model-Based and Automatic Techniques

15. User Interface Management Systems
Original goal: like databases, provide high-level language that abstracts details of input and output devices
This separation has not worked in practice
Good user interfaces must take into account the pragmatics and detailed behavior of all objects
Standardization of GUI input and output devices has made goal somewhat moot
Doesn’t address the useful & important aspects of UIs

16. Formal Language Based Tools
Early UIMSs used grammars and state-transition diagrams
Focus on dialog management
Moving Targets
Direct manipulation made dialog management less important
Path of Least Resistance
State diagrams afford worse user interfaces
High threshold
Formal languages are often hard to learn

17. Constraints Declare a relationship and system maintains it
Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet (1990), Amulet (1997), SubArctic (1996)
Predictability
Constraint networks can be hard to debug
Especially in multi-way constraints
Programmer must specify (or deduce) solving order
High threshold
Constraints require thinking differently
May be appropriate for graphical layout
Address the useful & important aspects of UIs

18. Model-Based and Automatic Techniques
Automatic techniques for generating UIs from a model or declarative specification of contents
Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993)
Try to separate specification of UI from content
May provide automatic reformating, retargeting, customization to users, etc.
Result is often unpredictable
Often can be worse UI than hand-drawn
Sometimes model is larger than the code it would replace

19. Discussion of Themes Address the useful & important aspects of UIs
Narrower tools have been more successful than ones that try to do “everything”
Do one thing well
Threshold / Ceiling
Research systems often aim for high ceiling
Successful systems seem to instead aim for a low threshold
Impossible to have both?

20. Discussion of Themes, cont.
Path of Least Resistance
Tools should guide implementers into better user interfaces
Goal for the future: do this more?
Predictability
Programmers do not seem willing to release control
Especially when system may do sub-optimal things
Moving Targets
Long stability of Macintosh Desktop paradigm has enabled maturing of tools
We predict a change soon

21. Future Prospects and Visions
Important Trends
Computers becoming a commodity
Ubiquitous Computing
Move to recognition-based interfaces
3-D interfaces
End-user customization and scripting
Violate assumptions of today’s tools
Assumptions limit what designers can do
Often unrecognized
Implications for future tools

22. Computers Becoming a Commodity
There are no longer “high-end” computers
Computer Science research’s trick of buying faster computers to “time travel” may no longer be possible
Moore’s law continues to operate
New opportunities
Quantitative change makes qualitative change possible
Enables the diversity of platforms
UIs more cinematic
Smooth transitions, animation, visual

23. Ubiquitous Computing Computation embedded in many kinds of devices
Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall-size displays, “smart” rooms
Next wave: easy communication with radio
E.g., BlueTooth:
Significant Implications for tools
Tools for coordinating multiple, distributed, communicating devices
“Multi-computer” user interfaces
Moving target problem

24. Varying Input and Output
Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels
Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels
Cell phone to Stanford’s wall (3796 x 1436 pixels)

25. Need New Interaction Techniques
Interaction techniques for desktop will not work
No room on small devices
Can’t reach menubar on wall-size devices
Want to run same application on different devices

26. Need for Prototyping Devices
User interface will be in hardware
Rapid design and prototyping needed for hardware
Pragmatics and usability cannot be evaluated from a simulation on a screen

27. Multiple, Distributed, Communicating
Computers more for communication, not for computation
Already true for WWW, , digital pagers, cell-phones
Computers as intermediaries between people
CSCW
But can’t assume have similar systems
Single person with multiple devices
Room-area networks like BlueTooth or HomeRF
People communicating with themselves
Tools will need to help with data sharing and synchronization

28. Limitations of Today’s Tools for UbiComp
Tools assume a Pointing Device
Hidden reliance on specific characteristics of common devices
Size of display
Many tools cannot handle a different number of mouse buttons
Change to a stylus on a touchpad requires different techniques
Assumptions about the setting
Assume user is sitting and looking at UI
Assume has user’s full attention

29. Move to Recognition-Based Interfaces
Speech, gestures, camera-based vision
Multimodal interaction
User will pick which modality to use
Use multiple modalities at same time
Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc.
Need tools to hide these complexities

30. Fundamental Differences of Recognition-based UIs
Input is uncertain
Recognition can make errors
Requires monitoring, feedback, correction
Interpreting input requires deep knowledge of data
Context of the application
“Move the red truck to here”

31. Implications of Recognition-based UIs
GUI event model no longer works
Do not produce discrete events
Separation of UI from application no longer works
Need a architecture based on accessible application data structures
“Reflection”, “Open Data Model”

32. 3-D Interfaces Difficult to design the right abstractions for tools
Demise of VRML for Web
Need to settle on the 3-D widgets and interaction techniques that will be standard
Requirement for near-real-time interactivity
Need to hide the mathematics

33. End-user Customization and Scripting
Spreadsheet enables end users to specify their own computation
Visual Basic, other “scripting” languages
Needed in all applications
Threshold for programming is too high
Need “gentle slope systems”

34. Gentle Slope Systems Visual Basic Director (v6) HyperCard Goal
Programming in C
Visual Basic
Director (v6)
HyperCard
C Programming
MFC
Difficulty of Use
C Programming
xCmds
Click and Create
Lingo
HyperTalk
Goal
Basic
Sophistication of what can be created

35. More Assumptions of Today’s Tools
Skill and Dexterity of users
Older users
Makes single, fixed library of widgets untenable
Non-overlapping and opaque components
Preclude translucency, magic lens interactions
Fixed libraries of components (widgets)
Creating new widgets is very difficult
New devices will require new interaction techniques
Interactive tools provide freedom of design
Aim for “Mechanism not Policy”

36. Operating Systems Considerations
What is in the OS?
Window Manager? Toolkit? Communication? Scripting facilities?
Need ever increasing services for applications
Need more access to low-level information
E.g., hardware buttons, whether on network
Ideally, API to support competition and research into these components

37. Some Design Guidelines for Future Tools
Many things require further research
Organize around providing rich context
Of application and device state
To inquire about data & methods; “reflection”
Enables EUP, Recognition-Based UIs, data sharing for UbiComp
Rather than event-based

38. More Design Guidelines
Replaceable User Interfaces
Ability to have multiple UIs
Enabled by procedural interface to everything in UI
Enables UbiComp devices, EUP
Aim for low threshold, rather than high ceiling
But cover the right parts of the interface
Predictable for programmers rather than “smart” or automatic
Need for support for evaluation

39. Conclusions
Research in tools necessarily trails innovation in UI design
Due to consolidation on desktop metaphor, significant progress in tools
UI design poised for radical changes
New opportunities and challenges for tools