1. 1 Past, Present and Future of User Interface Software Tools Brad A. Myers, Scott E. Hudson, and Randy Pausch Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University http://www.cs.cmu.edu/~bambam@cs.cmu.edu Brad A. Myers, Scott E. Hudson, and Randy Pausch Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University http://www.cs.cmu.edu/~bambam@cs.cmu.edu

2. 2 Introduction n User Interface Software Tools n Help developers design and implement user interfaces n Focus on Tools, but influenced by future UIs n Today’s tools are highly successful n Window Managers, Toolkits, Interface Builders ubiquitous n Most software built using them n Are based on HCI research Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html n Future tools must be different n User Interface Software Tools n Help developers design and implement user interfaces n Focus on Tools, but influenced by future UIs n Today’s tools are highly successful n Window Managers, Toolkits, Interface Builders ubiquitous n Most software built using them n Are based on HCI research Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html n Future tools must be different

3. 3 Talk Outline n Historical Perspective n What worked n What didn’t catch on n Why n Lessons Learned n Future Prospects and Visions n UI Trends that will require new tools n Important issues n Historical Perspective n What worked n What didn’t catch on n Why n Lessons Learned n Future Prospects and Visions n UI Trends that will require new tools n Important issues

4. 4 Historical Perspective n Themes è Address the useful & important aspects of UIs n Tools that succeeded helped (just) where needed è Threshold / Ceiling n Threshold = How hard to get started n Ceiling = how much can be achieved è Path of Least Resistance n Tools influence user interfaces created è Predictability n If not predictable, then not accepted by programmers è Moving Targets n Changing user interface styles makes tools obsolete n Themes è Address the useful & important aspects of UIs n Tools that succeeded helped (just) where needed è Threshold / Ceiling n Threshold = How hard to get started n Ceiling = how much can be achieved è Path of Least Resistance n Tools influence user interfaces created è Predictability n If not predictable, then not accepted by programmers è Moving Targets n Changing user interface styles makes tools obsolete

5. 5 What Worked n Window Managers and Toolkits n Event Languages n Graphical, Interactive Tools n Component Architectures n Scripting Languages n Hypertext n Object Oriented Programming n Window Managers and Toolkits n Event Languages n Graphical, Interactive Tools n Component Architectures n Scripting Languages n Hypertext n Object Oriented Programming

6. 6 Window Managers n Multiple (tiled) windows in research systems of 1960’s: NLS, etc. n Overlapping introduced in Alan Kay’s thesis (1969) n Smalltalk, 1974 at Xerox PARC n Successful because multiple windows help users manage scarce resources: n Screen space and input devices n Attention of users n Affordances for reminding and finding other work n Multiple (tiled) windows in research systems of 1960’s: NLS, etc. n Overlapping introduced in Alan Kay’s thesis (1969) n Smalltalk, 1974 at Xerox PARC n Successful because multiple windows help users manage scarce resources: n Screen space and input devices n Attention of users n Affordances for reminding and finding other work

7. 7 Toolkits n A collection of widgets n Menus, scroll bars, text entry fields, buttons, etc. n Toolkits help with programming n Help maintain consistency among UIs n Key insight of Macintosh toolbox è Path of least resistance translates into getting programmers to do the right thing n Successful partially because address common, low-level features for all UIs è Address the useful & important aspects of UIs n A collection of widgets n Menus, scroll bars, text entry fields, buttons, etc. n Toolkits help with programming n Help maintain consistency among UIs n Key insight of Macintosh toolbox è Path of least resistance translates into getting programmers to do the right thing n Successful partially because address common, low-level features for all UIs è Address the useful & important aspects of UIs

8. 8 Event Languages n Create programs by writing event handlers n Many UIMSs used this style n Univ. of Alberta (1985), Sassafras (1986), etc. n Now used by HyperCard, Visual Basic, Lingo, etc. n Toolkits with call-backs or action methods are related n Advantages: n Natural for GUIs since generate discrete events n Flow of control in user’s hands rather than programmer’s n Discourages moded UIs n Won’t work well in future n Create programs by writing event handlers n Many UIMSs used this style n Univ. of Alberta (1985), Sassafras (1986), etc. n Now used by HyperCard, Visual Basic, Lingo, etc. n Toolkits with call-backs or action methods are related n Advantages: n Natural for GUIs since generate discrete events n Flow of control in user’s hands rather than programmer’s n Discourages moded UIs n Won’t work well in future

9. 9 Graphical Interactive Tools n Create parts of user interface by laying out widgets with a mouse n Examples: Menulay (1983), Trillium (1986), Jean-Marie Hullot from INRIA to NeXT n Now: Interface Builders, Visual Basic’s layout editor, resource editors, “constructors” n Advantages: n Graphical parts done in an appropriate, graphical way è Address the useful & important aspects of UIs n Accessible to non-programmers è Low threshold n Create parts of user interface by laying out widgets with a mouse n Examples: Menulay (1983), Trillium (1986), Jean-Marie Hullot from INRIA to NeXT n Now: Interface Builders, Visual Basic’s layout editor, resource editors, “constructors” n Advantages: n Graphical parts done in an appropriate, graphical way è Address the useful & important aspects of UIs n Accessible to non-programmers è Low threshold

10. 10 Component Architectures n Create applications out of components which are separately developed and compiled n In UI software, each component controls an area of the screen n Example: drawing component handles picture inside a document n Invented by Andrew research project at CMU (1988) n Now: OLE, OpenDoc, ActiveX, Java Beans è Address the useful & important aspects of UIs n Just the “glue” to hold together components n Create applications out of components which are separately developed and compiled n In UI software, each component controls an area of the screen n Example: drawing component handles picture inside a document n Invented by Andrew research project at CMU (1988) n Now: OLE, OpenDoc, ActiveX, Java Beans è Address the useful & important aspects of UIs n Just the “glue” to hold together components

11. 11 Scripting Languages n First GUIs used interpreted languages n Smalltalk, InterLisp n Rapid development, supports prototyping è Low threshold n Then C and C++ became popular n Now, bringing back advantages in scripting languages n tcl/tk, Python, perl n Visual Basic, Javascript n But language must contain general-purpose control structures n First GUIs used interpreted languages n Smalltalk, InterLisp n Rapid development, supports prototyping è Low threshold n Then C and C++ became popular n Now, bringing back advantages in scripting languages n tcl/tk, Python, perl n Visual Basic, Javascript n But language must contain general-purpose control structures

12. 12 Hypertext n Ted Nelson named it in 1965 and developed Hypertext system at Brown University n Important systems: NLS (1967), Hyperties (1986) n World-Wide Web n Phenomenal success due to: n Ease of use of Mosaic browser n Support for embedded graphics n Support for easy authoring è Low threshold both for authoring and viewing n Ted Nelson named it in 1965 and developed Hypertext system at Brown University n Important systems: NLS (1967), Hyperties (1986) n World-Wide Web n Phenomenal success due to: n Ease of use of Mosaic browser n Support for embedded graphics n Support for easy authoring è Low threshold both for authoring and viewing

13. 13 Object Oriented Programming n Success of OO owes much to UI software field n Popularized by Smalltalk n GUI elements (widgets) seem like objects n Have state, accept events (messages) n Rise parallels GUIs n C++ with Windows 3.1 n Java for behaviors in WWW n Success of OO owes much to UI software field n Popularized by Smalltalk n GUI elements (widgets) seem like objects n Have state, accept events (messages) n Rise parallels GUIs n C++ with Windows 3.1 n Java for behaviors in WWW

14. 14 What Hasn’t Caught On n User Interface Management Systems n Formal Language-Based Tools n Constraints n Model-Based and Automatic Techniques n User Interface Management Systems n Formal Language-Based Tools n Constraints n Model-Based and Automatic Techniques

15. 15 User Interface Management Systems n Original goal: like databases, provide high-level language that abstracts details of input and output devices n This separation has not worked in practice n Good user interfaces must take into account the pragmatics and detailed behavior of all objects n Standardization of GUI input and output devices has made goal somewhat moot è Doesn’t address the useful & important aspects of UIs n Original goal: like databases, provide high-level language that abstracts details of input and output devices n This separation has not worked in practice n Good user interfaces must take into account the pragmatics and detailed behavior of all objects n Standardization of GUI input and output devices has made goal somewhat moot è Doesn’t address the useful & important aspects of UIs

16. 16 Formal Language Based Tools n Early UIMSs used grammars and state-transition diagrams n Focus on dialog management è Moving Targets n Direct manipulation made dialog management less important è Path of Least Resistance n State diagrams afford worse user interfaces è High threshold n Formal languages are often hard to learn n Early UIMSs used grammars and state-transition diagrams n Focus on dialog management è Moving Targets n Direct manipulation made dialog management less important è Path of Least Resistance n State diagrams afford worse user interfaces è High threshold n Formal languages are often hard to learn

17. 17 Constraints n Declare a relationship and system maintains it n Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet (1990), Amulet (1997), SubArctic (1996) è Predictability n Constraint networks can be hard to debug n Especially in multi-way constraints n Programmer must specify (or deduce) solving order è High threshold n Constraints require thinking differently n May be appropriate for graphical layout è Address the useful & important aspects of UIs n Declare a relationship and system maintains it n Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet (1990), Amulet (1997), SubArctic (1996) è Predictability n Constraint networks can be hard to debug n Especially in multi-way constraints n Programmer must specify (or deduce) solving order è High threshold n Constraints require thinking differently n May be appropriate for graphical layout è Address the useful & important aspects of UIs

18. 18 Model-Based and Automatic Techniques n Automatic techniques for generating UIs from a model or declarative specification of contents n Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993) n Try to separate specification of UI from content n May provide automatic reformating, retargeting, customization to users, etc. è Result is often unpredictable n Often can be worse UI than hand-drawn n Sometimes model is larger than the code it would replace n Automatic techniques for generating UIs from a model or declarative specification of contents n Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993) n Try to separate specification of UI from content n May provide automatic reformating, retargeting, customization to users, etc. è Result is often unpredictable n Often can be worse UI than hand-drawn n Sometimes model is larger than the code it would replace

19. 19 Discussion of Themes è Address the useful & important aspects of UIs n Narrower tools have been more successful than ones that try to do “everything” n Do one thing well è Threshold / Ceiling n Research systems often aim for high ceiling n Successful systems seem to instead aim for a low threshold n Impossible to have both? è Address the useful & important aspects of UIs n Narrower tools have been more successful than ones that try to do “everything” n Do one thing well è Threshold / Ceiling n Research systems often aim for high ceiling n Successful systems seem to instead aim for a low threshold n Impossible to have both?

20. 20 Discussion of Themes, cont. è Path of Least Resistance n Tools should guide implementers into better user interfaces n Goal for the future: do this more? è Predictability n Programmers do not seem willing to release control n Especially when system may do sub-optimal things è Moving Targets n Long stability of Macintosh Desktop paradigm has enabled maturing of tools n We predict a change soon è Path of Least Resistance n Tools should guide implementers into better user interfaces n Goal for the future: do this more? è Predictability n Programmers do not seem willing to release control n Especially when system may do sub-optimal things è Moving Targets n Long stability of Macintosh Desktop paradigm has enabled maturing of tools n We predict a change soon

21. 21 Future Prospects and Visions n Important Trends n Computers becoming a commodity n Ubiquitous Computing n Move to recognition-based interfaces n 3-D interfaces n End-user customization and scripting n Violate assumptions of today’s tools n Assumptions limit what designers can do n Often unrecognized n Implications for future tools n Important Trends n Computers becoming a commodity n Ubiquitous Computing n Move to recognition-based interfaces n 3-D interfaces n End-user customization and scripting n Violate assumptions of today’s tools n Assumptions limit what designers can do n Often unrecognized n Implications for future tools

22. 22 Computers Becoming a Commodity n There are no longer “high-end” computers n Computer Science research’s trick of buying faster computers to “time travel” may no longer be possible n Moore’s law continues to operate n New opportunities n Quantitative change makes qualitative change possible n Enables the diversity of platforms n UIs more cinematic n Smooth transitions, animation, visual n There are no longer “high-end” computers n Computer Science research’s trick of buying faster computers to “time travel” may no longer be possible n Moore’s law continues to operate n New opportunities n Quantitative change makes qualitative change possible n Enables the diversity of platforms n UIs more cinematic n Smooth transitions, animation, visual

23. 23 Ubiquitous Computing n Computation embedded in many kinds of devices n Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall-size displays, “smart” rooms n Next wave: easy communication with radio n E.g., BlueTooth: www.bluetooth.com n Significant Implications for tools n Tools for coordinating multiple, distributed, communicating devices n “Multi-computer” user interfaces n Moving target problem n Computation embedded in many kinds of devices n Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall-size displays, “smart” rooms n Next wave: easy communication with radio n E.g., BlueTooth: www.bluetooth.com n Significant Implications for tools n Tools for coordinating multiple, distributed, communicating devices n “Multi-computer” user interfaces n Moving target problem

24. 24 Varying Input and Output n Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels n Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels n Cell phone to Stanford’s wall (3796 x 1436 pixels) n Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels n Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels n Cell phone to Stanford’s wall (3796 x 1436 pixels)

25. 25 Need New Interaction Techniques n Interaction techniques for desktop will not work n No room on small devices n Can’t reach menubar on wall-size devices n Want to run same application on different devices n Interaction techniques for desktop will not work n No room on small devices n Can’t reach menubar on wall-size devices n Want to run same application on different devices

26. 26 Need for Prototyping Devices n User interface will be in hardware n Rapid design and prototyping needed for hardware n Pragmatics and usability cannot be evaluated from a simulation on a screen n User interface will be in hardware n Rapid design and prototyping needed for hardware n Pragmatics and usability cannot be evaluated from a simulation on a screen

27. 27 Multiple, Distributed, Communicating n Computers more for communication, not for computation n Already true for WWW, email, digital pagers, cell-phones n Computers as intermediaries between people n CSCW n But can’t assume have similar systems n Single person with multiple devices n Room-area networks like BlueTooth or HomeRF n People communicating with themselves n Tools will need to help with data sharing and synchronization n Computers more for communication, not for computation n Already true for WWW, email, digital pagers, cell-phones n Computers as intermediaries between people n CSCW n But can’t assume have similar systems n Single person with multiple devices n Room-area networks like BlueTooth or HomeRF n People communicating with themselves n Tools will need to help with data sharing and synchronization

28. 28 Limitations of Today’s Tools for UbiComp n Tools assume a Pointing Device n Hidden reliance on specific characteristics of common devices n Size of display n Many tools cannot handle a different number of mouse buttons n Change to a stylus on a touchpad requires different techniques n Assumptions about the setting n Assume user is sitting and looking at UI n Assume has user’s full attention n Tools assume a Pointing Device n Hidden reliance on specific characteristics of common devices n Size of display n Many tools cannot handle a different number of mouse buttons n Change to a stylus on a touchpad requires different techniques n Assumptions about the setting n Assume user is sitting and looking at UI n Assume has user’s full attention

29. 29 Move to Recognition-Based Interfaces n Speech, gestures, camera-based vision n Multimodal interaction n User will pick which modality to use n Use multiple modalities at same time n Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. n Need tools to hide these complexities n Speech, gestures, camera-based vision n Multimodal interaction n User will pick which modality to use n Use multiple modalities at same time n Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. n Need tools to hide these complexities

30. 30 Fundamental Differences of Recognition-based UIs n Input is uncertain n Recognition can make errors n Requires monitoring, feedback, correction n Interpreting input requires deep knowledge of data n Context of the application n “Move the red truck to here” n Input is uncertain n Recognition can make errors n Requires monitoring, feedback, correction n Interpreting input requires deep knowledge of data n Context of the application n “Move the red truck to here”

31. 31 Implications of Recognition-based UIs n GUI event model no longer works n Do not produce discrete events n Separation of UI from application no longer works n Need a architecture based on accessible application data structures n “Reflection”, “Open Data Model” n GUI event model no longer works n Do not produce discrete events n Separation of UI from application no longer works n Need a architecture based on accessible application data structures n “Reflection”, “Open Data Model”

32. 32 3-D Interfaces n Difficult to design the right abstractions for tools n Demise of VRML for Web n Need to settle on the 3-D widgets and interaction techniques that will be standard n Requirement for near-real-time interactivity n Need to hide the mathematics n Difficult to design the right abstractions for tools n Demise of VRML for Web n Need to settle on the 3-D widgets and interaction techniques that will be standard n Requirement for near-real-time interactivity n Need to hide the mathematics

33. 33 End-user Customization and Scripting n Spreadsheet enables end users to specify their own computation n Visual Basic, other “scripting” languages n Needed in all applications n Threshold for programming is too high n Need “gentle slope systems” n Spreadsheet enables end users to specify their own computation n Visual Basic, other “scripting” languages n Needed in all applications n Threshold for programming is too high n Need “gentle slope systems”

34. 34 Gentle Slope Systems Difficulty of Use Sophistication of what can be created Goal HyperCard Visual Basic Director (v6) HyperTalk xCmds Basic C Programming Lingo C Programming Programming in C MFC Click and Create

35. 35 More Assumptions of Today’s Tools n Skill and Dexterity of users n Older users n Makes single, fixed library of widgets untenable n Non-overlapping and opaque components n Preclude translucency, magic lens interactions n Fixed libraries of components (widgets) n Creating new widgets is very difficult n New devices will require new interaction techniques n Interactive tools provide freedom of design n Aim for “Mechanism not Policy” n Skill and Dexterity of users n Older users n Makes single, fixed library of widgets untenable n Non-overlapping and opaque components n Preclude translucency, magic lens interactions n Fixed libraries of components (widgets) n Creating new widgets is very difficult n New devices will require new interaction techniques n Interactive tools provide freedom of design n Aim for “Mechanism not Policy”

36. 36 Operating Systems Considerations n What is in the OS? n Window Manager? Toolkit? Communication? Scripting facilities? n Need ever increasing services for applications n Need more access to low-level information n E.g., hardware buttons, whether on network n Ideally, API to support competition and research into these components n What is in the OS? n Window Manager? Toolkit? Communication? Scripting facilities? n Need ever increasing services for applications n Need more access to low-level information n E.g., hardware buttons, whether on network n Ideally, API to support competition and research into these components

37. 37 Some Design Guidelines for Future Tools n Many things require further research n Organize around providing rich context n Of application and device state n To inquire about data & methods; “reflection” n Enables EUP, Recognition-Based UIs, data sharing for UbiComp n Rather than event-based n Many things require further research n Organize around providing rich context n Of application and device state n To inquire about data & methods; “reflection” n Enables EUP, Recognition-Based UIs, data sharing for UbiComp n Rather than event-based

38. 38 More Design Guidelines n Replaceable User Interfaces n Ability to have multiple UIs n Enabled by procedural interface to everything in UI n Enables UbiComp devices, EUP n Aim for low threshold, rather than high ceiling n But cover the right parts of the interface n Predictable for programmers rather than “smart” or automatic n Need for support for evaluation n Replaceable User Interfaces n Ability to have multiple UIs n Enabled by procedural interface to everything in UI n Enables UbiComp devices, EUP n Aim for low threshold, rather than high ceiling n But cover the right parts of the interface n Predictable for programmers rather than “smart” or automatic n Need for support for evaluation

39. 39 Conclusions n Research in tools necessarily trails innovation in UI design n Due to consolidation on desktop metaphor, significant progress in tools n UI design poised for radical changes n New opportunities and challenges for tools n Research in tools necessarily trails innovation in UI design n Due to consolidation on desktop metaphor, significant progress in tools n UI design poised for radical changes n New opportunities and challenges for tools