The Growth of Cognitive modeling in Human-Computer Interaction Since GOMS Presented by: Daniel Loewus-Deitch Douglas Grimes

Purpose of Article (1) Evaluate the current status of our methods for modeling HCI cognition and analyzing task performance. Evaluate the current status of our methods for modeling HCI cognition and analyzing task performance. Review the evolution of GOMS and MHP and how subsequent research has supported and extended this framework. Review the evolution of GOMS and MHP and how subsequent research has supported and extended this framework.

Purpose of Article (2) Examine 3 new directions for this framework: Examine 3 new directions for this framework: Study of learning and transfer. Study of learning and transfer. Study of errors. Study of errors. Analysis of parallel processes. Analysis of parallel processes.

Some Central Issues How people transfer between skilled performance and problem solving. How people transfer between skilled performance and problem solving. Designing consistent user interfaces. Designing consistent user interfaces. How people produce and manage errors. How people produce and manage errors. How we interpret visual displays. How we interpret visual displays. When processes are parallel and when they are serial. When processes are parallel and when they are serial.

Usefulness and Goals of Cognitive Modeling Main goal: Main goal: Predicting how users will interact with proposed designs. Predicting how users will interact with proposed designs.

Usefulness and Goals of Cognitive Modeling Useful in: Useful in: Initially constraining the design space. Initially constraining the design space. Answering specific design decisions, including various tradeoffs. Answering specific design decisions, including various tradeoffs. Estimating the total time for task performance. Estimating the total time for task performance. Estimate training time and guide training documentation. Estimate training time and guide training documentation. Discovering the most resource-intensive and error-prone stages of an activity. Discovering the most resource-intensive and error-prone stages of an activity.

GOMS and MHP Card et al. proposed their framework to help system designers Card et al. proposed their framework to help system designers Gather detailed knowledge about the processes of perception to action. Gather detailed knowledge about the processes of perception to action. Generate predictions about human behavior and task performance in “real, naturalistic settings.” Generate predictions about human behavior and task performance in “real, naturalistic settings.”

GOMS and MHP Two key components: Two key components: MHP (Model Human Processor) – a general characterization of the human as an information-processing system, including a MHP (Model Human Processor) – a general characterization of the human as an information-processing system, including a System architecture System architecture Library of quantitative parameters that breaks down task performance into specific components. Library of quantitative parameters that breaks down task performance into specific components.

GOMS and MHP Two key components: Two key components: GOMS (Goals, Operators, Methods, and Selecction rules) – A family of models that describes what the user needs to know in order to perform computer-based tasks. GOMS (Goals, Operators, Methods, and Selecction rules) – A family of models that describes what the user needs to know in order to perform computer-based tasks.

GOMS and MHP Steps of theoretical process: Steps of theoretical process: User perceives activity on screen. User perceives activity on screen. Evaluates whether it is what is expected. Evaluates whether it is what is expected. Sets up an intention of the next step (goals). Sets up an intention of the next step (goals). Retrieves way to enact this intent on the system. Retrieves way to enact this intent on the system. Executes appropriate motor movements. Executes appropriate motor movements. Repeat process. Repeat process.

Critical Steps of User Activities

Assumptions Routine cognitive skills can be described as a serial sequence of cognitive operations and motor activities. Routine cognitive skills can be described as a serial sequence of cognitive operations and motor activities. Each time parameter is independent of context (the same in any task). Each time parameter is independent of context (the same in any task). Empirical data derived from use of text editors, graphic systems, etc. is generalizable. Empirical data derived from use of text editors, graphic systems, etc. is generalizable.

Strengths Can accurately predict the time it takes a skilled user to execute a task based on a set of composite actions. Can accurately predict the time it takes a skilled user to execute a task based on a set of composite actions. Certain component parameters have been found to be very consistent across various tasks and stable across repeated experiments. Certain component parameters have been found to be very consistent across various tasks and stable across repeated experiments. Keystroke Keystroke Pointing Pointing Moving hands Moving hands Retrieving a chunk of info. from LTM. Retrieving a chunk of info. from LTM.

Strengths Allows comparisons of different design alternatives. Allows comparisons of different design alternatives.

Limitations Generalizable to new domains? Generalizable to new domains? Applied only to skilled users. Applied only to skilled users. No account of learning or delayed recall. No account of learning or delayed recall. No account of errors (predicts perfect performance). No account of errors (predicts perfect performance). Cognitive processes treated in same manner as perceptual and motor activities. Cognitive processes treated in same manner as perceptual and motor activities. No consideration of parallel processes. No consideration of parallel processes.

Limitations No consideration of mental workload. No consideration of mental workload. Does not determine functionality. Does not determine functionality. Does not address user fatigue. Does not address user fatigue. No account for individual differences. No account for individual differences. Does not predict user satisfaction or acceptance. Does not predict user satisfaction or acceptance. Does not consider CSCW issues. Does not consider CSCW issues.

Advances in Modeling Specific Serial Components (1) Direct tests of two assumptions: Direct tests of two assumptions: Serial processing Serial processing Consistency of time parameters across tasks Consistency of time parameters across tasks Helped define and expand the MHP. Helped define and expand the MHP. Determined by studies involving entering editor commands with keyboard commands and entering formulas in spreadsheets. Determined by studies involving entering editor commands with keyboard commands and entering formulas in spreadsheets.

Advances in Modeling Specific Serial Components (2) Parameters are grouped into 3 classes: Parameters are grouped into 3 classes: Motor movement Motor movement Perception Perception Memory and cognition Memory and cognition Parameters can be mapped onto the critical steps of the user activity process (previous figure). Parameters can be mapped onto the critical steps of the user activity process (previous figure).

Motor Movements Keying Keying 230 msec 230 msec Can vary with skill level of typist, frequency of particular key use, and predictability of text being typed. Can vary with skill level of typist, frequency of particular key use, and predictability of text being typed. Moving a mouse Moving a mouse 1100 msec per selection 1100 msec per selection Varies with distance of movement and size of target (Fitt’s Law). Varies with distance of movement and size of target (Fitt’s Law). Always a constant time of 1 s to began moving. Always a constant time of 1 s to began moving.

Motor Movements Hand Movements Hand Movements Move from keyboard to pointing device. Move from keyboard to pointing device. Large-muscle movement. Large-muscle movement. Variation among different pointing devices. Variation among different pointing devices.

Perception Perceptual processor is estimated to be 100 msec. Perceptual processor is estimated to be 100 msec. Saccade is estimated at 230 msec. Saccade is estimated at 230 msec. “Scanning” parameter, identified by Olson and Nielson, found in experiment to be 2300 msec. “Scanning” parameter, identified by Olson and Nielson, found in experiment to be 2300 msec. User scans spreadsheet screen, looking for cell addresses. User scans spreadsheet screen, looking for cell addresses. Composite of scanning, storing, and retrieving. Composite of scanning, storing, and retrieving. Roughly supports previously identified time parameter estimate when broken down into components. Roughly supports previously identified time parameter estimate when broken down into components.

Memory and Cognitive Processes Memory retrieval Memory retrieval M (“mental”) = time to retrieve the next unit of information from LTM. M (“mental”) = time to retrieve the next unit of information from LTM. Estimated at 1350 msec. Estimated at 1350 msec. On repeated trials, retrieval times drop 50% and remain flat, but keying times remain the same. On repeated trials, retrieval times drop 50% and remain flat, but keying times remain the same.

Memory and Cognitive Processes Executing steps in a task Executing steps in a task GOMS provides an explicit representation of the mental steps involved. GOMS provides an explicit representation of the mental steps involved. Kieras and Polson programmed the procedures in production system formalism. Kieras and Polson programmed the procedures in production system formalism.

Memory and Cognitive Processes Choosing among methods Choosing among methods MHP assumes that more choices for a response lead to longer response times. MHP assumes that more choices for a response lead to longer response times. Research suggests that a choice between methods is a complex cognitive task, requiring a number of cognitive steps. Research suggests that a choice between methods is a complex cognitive task, requiring a number of cognitive steps.

Summary of cognitive engineering parameters derived: Enter a keystroke 230 msec Point with a mouse 1500 msec Move hands to mouse 360 msec Perceive 100 msec Make a saccade 230 msec Retrieve from memory 1200 msec Execute a mental step 70 msec Choose among methods 1250 msec

Example of Applied Use of GOMS Walker et al. demonstrated how GOMS can help a designer narrow down his search. Walker et al. demonstrated how GOMS can help a designer narrow down his search. Goal was to shorten menu-selection time with nested menus. Goal was to shorten menu-selection time with nested menus.

Example of Applied Use of GOMS 3 adjustments were made and tested using GOMS techniques: 3 adjustments were made and tested using GOMS techniques: Pop up submenus on right rather than bottom (shorten total distance user much move cursor to make a selection). Pop up submenus on right rather than bottom (shorten total distance user much move cursor to make a selection). Target size grows as the distance from the cursor increases (Fittsized menus). Target size grows as the distance from the cursor increases (Fittsized menus). Add a virtual border around the pop-up menu (increase target size). Add a virtual border around the pop-up menu (increase target size).

Accuracy in Predicting Composite Performance Young and MacLean measured the times of people entering a block of values in a spreadsheet using 2 different methods: Young and MacLean measured the times of people entering a block of values in a spreadsheet using 2 different methods: Mouse method Mouse method Menu method Menu method Showed that using these parameters can provide sufficient accuracy for this level of analysis (14% error range in this case). Showed that using these parameters can provide sufficient accuracy for this level of analysis (14% error range in this case).

Extensions of Basic Framework Task-Action Grammar (TAG) Task-Action Grammar (TAG) Knowledge a user must have in order to translate from goals to actions in a particular system. Knowledge a user must have in order to translate from goals to actions in a particular system. Predicts learning with the relationship between system features and natural world associations. Predicts learning with the relationship between system features and natural world associations. Consists of commands, features of goal, dictionary of tasks, and rules that translate goals into actions. Consists of commands, features of goal, dictionary of tasks, and rules that translate goals into actions.

Extensions of Basic Framework Production systems Production systems Represent GOMS structure and aspects of MHP. Represent GOMS structure and aspects of MHP. Makes underlying knowledge much more explicit. Makes underlying knowledge much more explicit.

Learning and Transfer Reaction to GOMS’ narrow focus on skilled performance. Reaction to GOMS’ narrow focus on skilled performance. Time to learn has been advanced by Time to learn has been advanced by Cognitive Complexity Theory Cognitive Complexity Theory Payne and Green’s research on grammar rules. Payne and Green’s research on grammar rules. Transfer of training has been advanced by Transfer of training has been advanced by Kieras and Polson’s production system models Kieras and Polson’s production system models

Kieras and Polson’s Cognitive Complexity Theory (1) Made advances by focusing on Made advances by focusing on Time to learn new procedures. Time to learn new procedures. Transfer of training between procedures. Transfer of training between procedures. Used specialized language called NGOMSL to facilitate the programming of production system representations. Used specialized language called NGOMSL to facilitate the programming of production system representations.

Kieras and Polson’s Cognitive Complexity Theory (2) First determined number of steps in procedure and then assessed the time it takes a person to learn the procedure. First determined number of steps in procedure and then assessed the time it takes a person to learn the procedure. Can’t generalize quantified learning times to naturalistic settings. Can’t generalize quantified learning times to naturalistic settings.

Payne and Green Number of rules determines ease of learning. Number of rules determines ease of learning. More often a rule can be used, the more consistent a system is. More often a rule can be used, the more consistent a system is. More critical is whether features of rules follow real-world rules already familiar to the user. More critical is whether features of rules follow real-world rules already familiar to the user.

Kieras and Polson’s Production System Models Makes explicit exactly what it is that a person has to learn to use a new system. Makes explicit exactly what it is that a person has to learn to use a new system. Productions = units of learning. Productions = units of learning. Number of productions the two systems share is a good prediction of transfer. Number of productions the two systems share is a good prediction of transfer. Makes consistency of design across systems explicit and quantifiable. Makes consistency of design across systems explicit and quantifiable. TAG has some similar potential. TAG has some similar potential.

(Time to Learn Scatter Plot – Poulson)

Analysis of Errors Hypothesis: When working memory is overloaded, errors will increases. Hypothesis: When working memory is overloaded, errors will increases. Test: Calculate Profit = Revenues – Costs, using Lotus & IFPS Test: Calculate Profit = Revenues – Costs, using Lotus & IFPS Result: IFPS  burdens working memory  fewer errors Result: IFPS  burdens working memory  fewer errors Similar studies on SQL queries. Similar studies on SQL queries.

Parallel Processes Earlier cognitive studies assumed serial processing. Earlier cognitive studies assumed serial processing. Skilled motor and perceptual action is highly parallel: Skilled motor and perceptual action is highly parallel: Expert typist, musician, athlete, driver, … Expert typist, musician, athlete, driver, … Serial models overestimate time for parallel tasks Serial models overestimate time for parallel tasks Skilled action is more parallel; novice action more serial. Skilled action is more parallel; novice action more serial.

Critical Path Analysis: Parallel + Sequential Theory: 3 parallel processors: perceptual, cognitive, and motor Theory: 3 parallel processors: perceptual, cognitive, and motor Studies of skilled typing and skilled menu use. (Author’s interpretation, not speaker’s): Studies of skilled typing and skilled menu use. (Author’s interpretation, not speaker’s): Expert (~33 wpm) limited by cognitive processor (e.g., reading speed for typist) Expert (~33 wpm) limited by cognitive processor (e.g., reading speed for typist) Novice (~6 wpm) limited by motor speed. Novice (~6 wpm) limited by motor speed.

Q’s on Critical Path Analysis If cognitive processor is the bottleneck for expert typist, why can expert typists read much faster than they type? Do we “chunk” letters together in preconscious processing, then “unchunk” them to type? If cognitive processor is the bottleneck for expert typist, why can expert typists read much faster than they type? Do we “chunk” letters together in preconscious processing, then “unchunk” them to type? Where do we draw the line between perception and cognition? Example shows reading is one word at a time with constant time per word, but recognizing words is a cognitive process, whether preconscious or not. Also ocular-motor skill. Where do we draw the line between perception and cognition? Example shows reading is one word at a time with constant time per word, but recognizing words is a cognitive process, whether preconscious or not. Also ocular-motor skill. Doesn’t address looping/cascading. Doesn’t address looping/cascading. In spite of Q’s  more accurate predictions for some tasks than purely serial models. In spite of Q’s  more accurate predictions for some tasks than purely serial models.

What cognitive modeling (serial & parallel) has not revealed How we move from skilled to unskilled tasks smoothly throughout the day. How we move from skilled to unskilled tasks smoothly throughout the day. Learning: “Athough we know that consistency in a system may make that system easier to learn … and easier to operate…, we do not know much more than that.” Learning: “Athough we know that consistency in a system may make that system easier to learn … and easier to operate…, we do not know much more than that.” Errors: More than overloading working memory Errors: More than overloading working memory Individual differences in HCI Individual differences in HCI Fatigue and other stress factors: machines don’t tire or get stressed out; people do. Fatigue and other stress factors: machines don’t tire or get stressed out; people do.

From GOMS to Games Prior studies based on static visual display of text and numbers. Prior studies based on static visual display of text and numbers. Bonnie John & Alonso Vera extended GOMS model to more interactive HCI’s -- video games, call handling, & browsing. Bonnie John & Alonso Vera extended GOMS model to more interactive HCI’s -- video games, call handling, & browsing. Subject: 9 year-old expert Subject: 9 year-old expert Result – 2 levels: Result – 2 levels: GOMS predicts functional interaction well GOMS predicts functional interaction well Keystroke accuracy ~ 50% Keystroke accuracy ~ 50% Program (HI-SOAR) learns ~ humans Program (HI-SOAR) learns ~ humans

Conclusion GOMS and its successors are useful for many repetitive HCI tasks – bank tellers, ATM’s, airline reservations, bookkeeping… Cognitive modeling saves $ + time GOMS and its successors are useful for many repetitive HCI tasks – bank tellers, ATM’s, airline reservations, bookkeeping… Cognitive modeling saves $ + time  less need to build and test prototypes on users when models can predict results. User testing still needed for reality check – “situated cognition” = understanding tasks in their human and environmental context. User testing still needed for reality check – “situated cognition” = understanding tasks in their human and environmental context.

(Start of extra slides) (Start of extra slides)

Learning Computer Tasks Limitation of GOMS: Restricted to skilled performance. Didn’t address learning. Limitation of GOMS: Restricted to skilled performance. Didn’t address learning.  Cognitive Complexity Theory (Kieras & Polson)  Cognitive Complexity Theory (Kieras & Polson) 2 aspects of their learning model: 2 aspects of their learning model: Time to learn a new system or task Time to learn a new system or task Transferring knowledge between tasks Transferring knowledge between tasks

Time to Learn Kieras developed a high-level programming language to describe learning simple computer tasks, like data entry or writing an SQL query. Kieras developed a high-level programming language to describe learning simple computer tasks, like data entry or writing an SQL query. Keiras & Polson found learning each step of a task takes ~ 30 s. Other studies found shorter times. Keiras & Polson found learning each step of a task takes ~ 30 s. Other studies found shorter times. Current best guess ~ 25 s./step for learning. Current best guess ~ 25 s./step for learning.

Learning & Complexity Number of rules to learn is not as important how well the rules follow real- world features the learner already knows. Number of rules to learn is not as important how well the rules follow real- world features the learner already knows. Implication: Metaphors work, as long as they are consistent. Implication: Metaphors work, as long as they are consistent.

Learning Computer Tasks (1): Theory K & P’s production rules make explicit exactly what a person has to learn to master a new system. K & P’s production rules make explicit exactly what a person has to learn to master a new system. Hypothesis: If # of productions (~steps) is a measure of how long it takes to learn, then 2 systems with same # of productions should take same time to learn. Hypothesis: If # of productions (~steps) is a measure of how long it takes to learn, then 2 systems with same # of productions should take same time to learn.

Learning Computer Tasks (2): Studies Example: Compare time it takes to learn: Example: Compare time it takes to learn: How to copy a floppy How to copy a floppy How to print a document How to print a document Result: Learning times are consistent across systems. Result: Learning times are consistent across systems. Value: When designing systems, favor alternatives with simpler production rules Value: When designing systems, favor alternatives with simpler production rules (easier to learn!) (easier to learn!)

Philosophical Critique of GOMS Does it reduce human to machine? Does it reduce human to machine? Answer: Not intended to address philosophy. Only intended to aid in system design. Answer: Not intended to address philosophy. Only intended to aid in system design. Bottom line: Predictions average within 14% of observed values for a narrow range of serial tasks. Bottom line: Predictions average within 14% of observed values for a narrow range of serial tasks.

Practical Critiques of GOMS Restricted to skilled performance Restricted to skilled performance Doesn’t predict learning Doesn’t predict learning Doesn’t apply to parallel tasks Doesn’t apply to parallel tasks

Extensions to Basic GOMS Framework Purpose: Less restrictive framework for understanding HCI. Purpose: Less restrictive framework for understanding HCI. 3 areas of analysis since GOMS: 3 areas of analysis since GOMS: 1. Learning 2. Errors 3. Parallel processes

Cognitive Engineering Grammars Purpose: Define cognitive tasks in terms of countable components for production rules. Purpose: Define cognitive tasks in terms of countable components for production rules. Example: Task-Action Grammar (TAG) Example: Task-Action Grammar (TAG) Commands, e.g. “Left-click,” “press ” Commands, e.g. “Left-click,” “press ” Features of goal, e.g. direction of cursor Features of goal, e.g. direction of cursor Tasks, e.g., “Add 3 numbers.” Tasks, e.g., “Add 3 numbers.” Rules to translate goals into action. Rules to translate goals into action.