1. 1 The Evolution of Human- Performance Modeling Techniques for Usability Uri Dekel (udekel@cs.cmu.edu)udekel@cs.cmu.edu Presented in “Methods of Software Engineering”, Fall 2004

2. 2 Outline Motivation and scope From early models to GOMS  Stimulus-Response-Controller models  Information Processing models GOMS variants: what to use? SW tools for GOMS Lessons learned…

3. 3 Motivation and Scope

4. 4 Motivation Minor timing differences may have a major economic impact  Consider a call center with 100 employees  Average call length = 1 min  144000 calls per day for entire call center Improvement of 2 seconds per call  80 person hours per day  29200 person hours per year

5. 5 Where can we optimize? Moore’s law works for HW and SW  In the past, system reaction time was slow Databases, networks and GUIs were slow  Now practically instantaneous Moore’s law does not apply to humans…  But usability has significant impact on performance

6. 6 Motivation Problems and solution:  “How to design more usable interfaces?” Partial solution: usability methods and principles  “How to ensure a design can be used effectively?” Inadequate solution: use intuition Inadequate solution: functional prototypes in a design-implement-test-redesign cycle  Expensive and time consuming, especially for hardware Possible solution: paper prototyping complemented by quantitative models for predicting human performance

7. 7 Motivation We need to predict performance on a system which is not yet available Nielsen, 1993:  “A Holy Grail for many usability scientists is the invention of analytic methods that would allow designers to predict the usability of a user interface before it has even been tested.”  “Not only would such a method save us from user testing, it would allow for precise estimates of the trade-offs between different design solutions without having to build them.”  “The only thing that would be better would be a generative theory of usability that could design the user interface based on a description of the usability goals to be achieved.”

8. 8 Cognitive modeling Definition:  producing computational models for how people perform tasks and solve problems, based on psychological principles Uses:  Predicting task duration and error potential  Adapting interfaces by anticipating behavior

9. 9 Outside our Scope Predicting the intent of the user  Model the activities of the user  Relies on AI techniques to make predictions Useful for intelligent and adaptable UIs Improves learning curve

10. 10 Outside our Scope Predicting the intent of the user  Model the activities of the user  Relies on AI techniques to make predictions Useful for intelligent and adaptable UIs Improves learning curve  But not always successful…

11. 11 Scope Predicting the usability of the UI  Qualitative models Will the UI be intuitive and simple to learn? Is the UI aesthetic and consistent? Will the user experience be positive?  Quantitative models How long will it take to become proficient in using the UI? How long will it take a skilled user to accomplish the task?

12. 12 Goal of this talk The goal is NOT:  To introduce you to GOMS and it variants You got that from the reading The goal is:  To provide the theoretical foundation and evolution of models which led to GOMS  To show tools that support GOMS  To understand how it could be useful to you

13. 13 Early models

14. 14 Stimulus-Response-Controller Research predates Computer Science  Attempts to improve usability of interactive electronic systems such as control panels, radar displays, air traffic control, etc. Early models developed by experimental psychology researches in the 1950s  Limited to single short perceptual and motor activities  Based on information and communications theory Human is a simple device which responds to stimuli by carrying out a motor behavior Based on Shannon’s definitions of entropy and channel capacity

15. 15 Information Theory 101: Entropy Entropy of a random event is a measure of its actual randomness  High entropy if unpredictable:

16. 16 Information Theory 101: Entropy Entropy of a random event is a measure of its actual randomness  High entropy if unpredictable: The winning numbers for this week’s lottery  Same probability for all results  Low entropy if predictable:

17. 17 Information Theory 101: Entropy Entropy of a random event is a measure of its actual randomness  High entropy if unpredictable: The winning numbers for this week’s lottery  Same probability for all results  Low entropy if predictable: What lunch will be served at the next seminar?  High probability of Pizza. Low probability for Sushi

18. 18 Information Theory 101: Entropy Entropy can measure amount of information in message Consider a message encoded as a string of bits.  Is the next bit 0 or 1 ? High entropy  What if we add parity bit? Lower entropy for the parity bit  What if we replicate every bit once? Even lower for replicated bits

19. 19 Information Theory 101: Entropy Formally:  Let x be a random event with n possible values  The entropy of X is:

20. 20 Information Theory 101: Channel Capacity Information rate in a perfect channel  n = bits per second  H = entropy per bit  R = nH R=n if entropy is 1 (pure data)  The channel bandwidth curbs the rate

21. 21 Information Theory 101: Channel Capacity Information rate in an analog channel  Curbed by bandwidth and noise  We can fix some errors using different encodings Is there a limit to how much we can transfer?

22. 22 Information Theory 101: Channel Capacity Shannon’s definition of channel capacity  Maximal information rate possible on the channel For every R<C, there is an encoding which allows the message to be sent with no errors Theoretical maximum effectiveness of error correction codes Does not tell us what the code is  Capacity formula: B = bandwidth SNR = Signal-to-noise ratio

23. 23 Fitts’ Law Paul Fitts studied the human limitation in performing different movement tasks Measured difficulty of movement tasks in information-metric bits Movement task is the transmission of information through the “human channel”  But this channel has a capacity

24. 24 Fitts’ Law Fitts’ law [1954] predicts movement time from starting point to specific target area  Difficulty index A = distance to target center, W = target width  Movement time: a = device dependent intercept b = device dependent Index of Performance  The coefficients are measured experimentally  e.g., mouse IP lower than stylus, joystick

25. 25 Fitts’ Law Implications Primary implication:  Big targets at close distance are acquired faster than small targets at long range Used to empirically test certain designs Theoretical rationale for many design principles

26. 26 Fitts’ Law Implications Should buttons on stylus based touch screen (e.g., PDA) be smaller, larger or the same as buttons in a mouse based machine?

27. 27 Fitts’ Law Implications Should buttons on stylus based touch screen (e.g., PDA) be smaller, larger or the same as buttons in a mouse based machine? Answer: larger, because it is more difficult to precisely point the stylus (higher index of performance)

28. 28 Fitts’ Law Implications Why is the context sensitive menu (“right- click menu” in Windows) located close to the mouse cursor?

29. 29 Fitts’ Law Implications Why is the context sensitive menu (“right- click menu” in Windows) located close to the mouse cursor? Answer: mouse needs to travel shorter distance

30. 30 Fitts’ Law Implications Which is better for a context sensitive menu, a pie menu or a linear menu?

31. 31 Fitts’ Law Implications Which is better for a context sensitive menu, a pie menu or a linear menu? Answer: if all options have equal probabilities, a pie menu. If one option is highly dominant, a linear menu

32. 32 Fitts’ Law Implications In Microsoft Windows, why is it easier to close a maximized window than to close a regular window?

33. 33 Fitts’ Law Implications In Microsoft Windows, why is it easier to close a maximized window than to close a regular window? Answer: If the mouse cannot leave the screen, target amplitude is infinite in the corner of the screen where the close box is located

34. 34 Fitts’ Law Implications Why use mouse-gestures to control applications?

35. 35 Fitts’ Law Implications Why use mouse-gestures to control applications? Answer: A mouse gesture starts at the current location and requires limited movement, compared to acquiring the necessary buttons.

36. 36 Fitts’ Law limitations Addresses only target distance and size, ignores other effects Applies only single-dimensional targets  Later research showed extensions to 2D and 3D Considers only human motor activity  Cannot account for software acceleration Does not account for training  Insignificant effect in such low-level operations

37. 37 Fitts’ Law limitations Only supports short paths  Research provided methods for complicated paths, using integration But most importantly:  Operates at a very low level Difficult to extend to complex tasks

38. 38 Hick’s Law Humans have a non-zero reaction time  Situation is perceived, then decision is made Hick’s law predicts decision time as a function of the number of choices  Humans try to subdivide a problem Binary rather than linear search  For equal probabilities: coefficient a measured experimentally  For differing probabilities:

39. 39 Hick’s Law Hick’s law holds only if a selection strategy is possible  e.g., alphabetical listing Intuitive implications:  Split menus into categories and groups An unfamiliar command should be close to related familiar commands

40. 40 Hick’s Law Example The next slide presents a screenshot from Microsoft Word How fast can you locate the toolbar button for the “WordArt character spacing” command?

41. 41 Hick’s Law Example

42. 42 Limitations of the Early Models Developed before interactive computer systems became prevalent  Use metaphors of analog signal processing Human-Computer Interaction is continuous  Cannot be broken down into discrete events Human processing has parallelism

43. 43 Information Processing Models

44. 44 Information Processing Models Developed in the 1960s Combine psychology and computer science Humans performs sequence of operations on symbols Generic structure:

45. 45 Information Processing Models Models from general psychology are fitted to the results of actual experiments  Not predictive for other systems Zero-parameter models can provide predictions for future system  Parameterized only by information from existing systems e.g., typing speed, difficulty index, etc.

46. 46 Model Human Processor [Card, Moral and Newell in 1983] Framework for zero-parameter models of specific tasks  Humans process visual and auditory input  Output is motor activity Unique in decomposition into three systems  Each consists of processor and memory  Can operate serially or in parallel  Each with unique rules of operation

47. 47 Model Human Processor

48. 48 Model Human Processor Properties Processor  Cycle time limits amount of processing Memory  Relatively permanent long term memory  Short term memory Consists of small activated LTM chunks  There are “seven plus minus two” chunks Every memory unit has:  Capacity, decay time, information type

49. 49 Perceptual System Input arrives from perceptual receptors in outside world Placed in visual and auditory stores  Stored close to physical form “bitmap” and “waveforms” rather than symbols Processor encodes symbolically in stores in LTM  Memory and processing limitations lead to memory loss Attention directs items to be saved

50. 50 Cognitive System Responsible for making decision and scheduling motor operations Performs a recognize-act cycle  Uses association to activate LTM chunks  Acts by modifying data in working memory Might require several cycles

51. 51 Motor System Translates thoughts into actions  Uses decisions stored in working memory to plan movements  Different movement options, including speech Actually sequence of micro-movements Motor-memory cache simplifies actions  But not explicitly represented in the MHP

52. 52 Limitations of the MHP Does not address attention Works in a bottom-up manner  Given a movement scenario, can help determine how long it would take  In design, it is preferable to start with the goal and find how it should best be accomplished This is what GOMS tries to accomplish

53. 53 Principles of Operation Problem space principle  Users apply a series of operations to transform an initial state into a goal state Rationality principle  Users will develop effective methods, given a task, its environment, and their limitations. MHP provides the “human hardware”. GOMS models will represent the software.

54. 54 GOMS

55. 55 GOMS Enables description of tasks and the knowledge to perform them  GOMS model can be used to execute tasks Unlike less formal task analysis techniques  Qualitatively used for developing training tools, help systems, etc.  Quantitatively used for predicting performance

56. 56 GOMS Goals  What the user tries to accomplish Hierarchy of subgoals Operators  Atomic operations for accomplishing the goal Perceptual, Cognitive or Motor Methods  Algorithms for accomplishing goals Selection Rules  Knowledge used to select method for task

57. 57 Variants: KLM Keystroke Level Model  Proposed by Card, Moran Newell Predecessor of GOMS  No selection rules  Simple linear sequence of operators Analyst inputs keystrokes and mouse movements Simple heuristics used for placing mental operators Predicted time is sum of operator execution times  Can be used to compare two scenarios of using the same system

58. 58 Variants: KLM Example: Moving text in Word

59. 59 Variants: CMN-GOMS Card Moran Newell GOMS  Original formulation of GOMS, extends KLM Procedural program form  Hierarchy of subgoals  Methods realize subgoals  Selection rules pick method  Variety of cognitive operations Serial processing

60. 60 Variants: CMN-GOMS Sample task: moving text in word

61. 61 Variants: CPM-GOMS Cognitive-Perceptual-Motor GOMS or Critical-Path-Method GOMS  Adds parallelism using the MHP  Created from a CMN model  Results in a PERT chart Execution time is the critical path  Assumes extreme expertise Might underestimate. Demonstrated in the Ernestine project

62. 62 Variants: CPM-GOMS Example: text editing

63. 63 Variants: NGOMSL Natural GOMS Language  Represents models using natural language  Relies on Cognitive Complexity Theory Internal operators for subgoals, memory manipulation Rules-of-thumb for number of steps in method, setting and terminating goals, information that needs to be remembered  Can provide some estimates of learning time and possibility of errors

64. 64 Variants: NGOMSL Example

65. 65 GOMS limitations Tooling  Will be discussed later Valid only for expert users  The primary goal of HCI was to make systems more accessible to novices  Many systems are only used occasionally  Ignores problem-saving nature of many tasks Assumes no errors  Even experts make simple errors  Error recovery may be important factor

66. 66 GOMS limitations Does not address individual users  Relies on statistical averages Provides only one metric  Execution time should not be the only basis for comparing designs Does not utilize recent cognitive theories  Even newer models are “fuzzy” on cognitive aspects.

67. 67 Should GOMS be used? According to John and Kieras (’93), GOMS should be used only if:  It is goal-directed  It is routine and a user can become skilled  It involves user control  We want to analyze procedural properties of the system

68. 68 Which GOMS model to use? Choice based on parallelism and on required information:  Functionality coverage  Functional consistency  Operator sequence  Execution time  Learning time  Error recovery

69. 69 Which GOMS model to use? (Table from John and Kieras)

70. 70 Current Research

71. 71 Rough research evolution Proposal of initial GOMS models  KLM, CMN-GOMS Initial experimental validation Extension for parallelism  CPM-GOMS  Validation, esp. project Ernestine Improved model usability and basis  Kieras’ NGOMSL Still no industrial adoption! Simplified methodologies Tooling

72. 72 Quick (and Dirty) GOMS A very simplified GOMS  Allows rough time estimates  A generic “task” operator  Probabilities instead of selection rules  Model is formed as a tree Aimed for software engineers  No need to learn complex methodologies  Familiar tree paradigm

73. 73 Quick (and Dirty) GOMS DEMO

74. 74 Deriving KLM Models KLM models are simple to construct and understand  Goal: user creates interface mockup, performs activities, gets prediction Critique System at CMU  Uses research-oriented subArctic GUI toolkit

75. 75 Deriving KLM Models CogTools project at CMU  User creates HTML mock-up interface in Macromedia Dreamweaver  User demonstrate tasks in Netscape browser (continued)

76. 76 Deriving KLM Models CogTools project  System generates KLM modem and task prediction  Too many software requirements: Dreamweaver Java Netscape browser ACT-R Allegro Common Lisp  Skipped the demo for obvious reasons…

77. 77 Deriving NGOMSL models GLEAN project  Uses English-like notation  User supplies: NGOMSL model of tasks Associative representation of working memory  Pairs of tags and values  Can be used as method parameters Transition net  Simulates user behavior on simulated device  Difficult to represent interface without implementation

78. 78 Deriving CPM-GOMS models CPM-GOMS models are most powerful  Utilize parallelism of human processing Very difficult to create manually  Given a serial CMN-GOMS model, must find acceptable interleaving on MHP resources  Generate PERT chart

79. 79 Deriving CPM-GOMS models NASA’s Apex Architecture  A framework for creating reactive intelligent agents for complex tasks  Models how humans complex tasks Procedure Definition Language (PDL) for defining tasks and methods AI engine for making selections Interleaves activities into “resources”

80. 80 Deriving CPM-GOMS models PDL example

81. 81 Deriving CPM-GOMS models APEX is used to model CPM-GOMS  User specifies task in PDL  APEX interleaves and creates PERT chart

82. 82 Deriving CPM-GOMS models

83. 83 Lessons Learned

84. 84 When is it important to predict performance? If the system will be used intensively If the tasks are routine enough If performance has significant economic impact  Benefits should outweigh prediction cost

85. 85 Can’t we just rely on our intuition? As SW engineers, we might believe that our intuition is sufficient  Improving qualitative properties is possible by following usability rules of thumb  Quantitative properties are another story… GOMS research shows that intuition is not sufficient  Consider project Ernestine

86. 86 Are the qualitative benefits to GOMS? Analyzing the task helps us focus on the user We obtain the sequence models of contextual design… These models help us:  Realize what the user is trying to achieve  Pinpoint different means for accomplishing the same goal  Discover inconsistencies and opportunities for simplification It’s a good way to create detailed use cases It lays the foundation for training and manuals

87. 87 How much should we invest? Tradeoff between modeling costs and benefits  What is the impact of each improvement?  Is the system mission-critical? Even rough estimates can be useful  Use QGOMS to analyze high level goals  Create KLM models with a spreadsheet to analyze simple scenarios

88. 88 Who should be assigned with GOMS modeling? SW engineers generally avoid learning techniques that don’t help them construct new software  Hire a usability engineer if modelling is critical enough Experts are needed for accurately using the complex GOMS variants  Use QA or Testing engineers before using engineers

89. 89 Can’t SW engineers do the modelling? Accurate GOMS modeling requires knowledge in cognitive psychology SW Engineers will usually avoid tools not integrated with the IDE  Vision: a “human performance profiler” in the IDE, just like the execution profiler  User interacts with designed UI, system suggests what the average execution time will be

90. 90 Finally… If you use GOMS Remember the limitations of GOMS  Don’t rely on it in inappropriate settings. Do not rely on it as your only metric Do not abandon the notion of prototypes

91. 91 Questions?

92. 92 BACKUP SLIDES

93. 93 Information Theory 101: Entropy Entropy measures the “amount of information” An intuitive example: Consider a sequence of n bits which encodes some random data  Option 1: All bits are used to represent a message of length n Caveat: error can destroy the message We got 2^n possible messages, with equal probabilities  We cannot predict or bias what the message Such as message is said to have a high entropy.  Option 2: Use some bits for error correction (e.g., parity) Advantage: we can detect some errors We got less than 2^n possible messages  We can predict one of the bits Such as message is said to have a lower entropy.

94. 94 Information Theory 101: Entropy Other intuitive examples  Tossing a legal coin: entropy of 1 (same probability for each result)  Skewed coin: lower entropy (one result more likely) Consider encryption mechanisms for textual communications  Characters in unencrypted text have non-uniform distribution Low entropy  Bad encryption: add the same value to every character Entropy is still low  Good encryption: increase entropy by a sequence of encryptions and shifting

95. 95

96. 96 Information Theory 101: Entropy Entropy measures the “amount of information” An intuitive example: Consider a sequence of n bits which encodes some random data  Option 1: All bits are used to represent a message of length n Caveat: error can destroy the message We got 2^n possible messages, with equal probabilities  We cannot predict or bias what the message Such as message is said to have a high entropy.  Option 2: Use some bits for error correction (e.g., parity) Advantage: we can detect some errors We got less than 2^n possible messages  We can predict one of the bits Such as message is said to have a lower entropy.

97. 97 Information Theory 101: Entropy Example  E-commerce site located in the US Provides services to USA and N other countries  Assume probability of US customer is p  Equal distribution for other countries  What is the entropy of user location?

98. 98 In a “perfect” analog channel, we could have unlimited bandwidth by increasing frequency and discrete voltage values  Physics curbs frequency. Noise introduces errors Information rate in a noisy channel  Both bandwidth and noise curb information rate  Wrong intuition: double the rate, double the errors