The Growth Of Cognitive Modeling in Human Computer Interaction Since GOMS Judith Reitman Olson and Gary M. Olson The University of Michigan Presenters: Tosin Aiyelokun and Norman Makoto Su

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  Modeling Specific Components  GOMS Limitations  Summary

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  Modeling Specific Components  GOMS Limitations  Summary

Cognitive Modeling: Definition  A theory that produces a computational model of how people perform tasks and solve problems by using psychological principles and empirical studies.

Cognitive Modeling: Research Methods Empirical Methods Methods ProgrammingTechniques Formal Analysis Philosophy, Logic, Linguistics, Mathematics, Computer Science Experimental Psychology, Neuroscience Artificial Intelligence

Cognitive Modeling: Role  Limits the design space  Answers specific design decisions  Estimates total task time  Estimates training time  Identifies complex, error-prone stages of the design  A means of testing current psychological theories

Cognitive Modeling: Human Information Processor (HIP) Receptors(perception)Effectors (motor actions) Processor Memory External World HIP

The Human Processor Model Perceptual Processor -sensory input (audio & visual) -code info symbolically -output into audio and visual image storage (WM buffer) Cognitive Processor -input from sensory buffers -access LTM to determine response -output response into WM Motor Processor -input response from WM -carry out response

Cognitive Modeling: Applications  GOMS Today ’ s presentation  Soar Integrated architecture for knowledge-based problem solving, learning and interacting with external environments.  ACT-R Atomic Components of Thoughts - Rational

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  Modeling Specific Components  GOMS Limitations  Summary

GOMS: Overview  Formal representation of routine cognitive skill.  A description of knowledge required by an expert user to perform a specific task.  Provides a description of what the user must learn.

GOMS: Classification  Provides a predictive, descriptive and prescriptive model Predictive  Predicts the time it will take user to perform the tasks under analysis Descriptive  Represents the way a user performs tasks on a system Prescriptive  Guides the development of training programs and help systems

GOMS: Definition  GOMS models user ’ s behavior in terms of: G oals  What the user wants to do. O perators  Specific steps a user is able to take and assigned a specific execution time. M ethods  Well-learned sequences of subgoals and operators that can accomplish a goal. S election Rules  Guidelines for deciding between multiple methods.

GOMS: A Family of Models  Keystroke-Level Model (KLM)  Card, Moran, and Newell (CMN-GOMS)  Natural GOMS Language (NGOMSL)  Cognitive-Perceptual-Motor GOMS (CPM-GOMS)

GOMS: Keystroke-Level Model (KLM)  Simplest GOMS technique The basis for all other GOMS techniques Predicts execution time  Requires analyst-supplied methods  Assumes that routine cognitive skills can be decomposed into a serial sequence of basic cognitive operations and motor activities, which are: K: A keystroke (280 msec) M: A single mental operator (1350 msec) P: Pointing to a target on a small display (1100 msec) H: Moving hands from the keyboard to a mouse (400 msec)

Top-level Goal:Edit Manuscript (move “quick brown” to before “fox”) Subgoal: Highlight text Operators: Move-mouse Click mouse-button Type characters (keyboard shortcuts) Methods: 1. Delete-word-and-retype (retype method) 2. Cut-and-paste-using-keyboard-shortcuts (shortcuts method) 3. Cut-and-paste-using menus (menus method) Selection Rules:If the text to be moved is one or two characters long, use retype method Else, if remember shortcuts, use shortcuts method Else, use the menus method KLM Example

DescriptionOperatorDuration (sec) Mentally PrepareM1.35 Move cursor to “ quick ” P1.10 Double-click mouse buttonK0.40 Move cursor to “ brown ” P1.10 Shift-click mouse buttonK0.40 Mentally PrepareM1.35 Move cursor to Edit MenuP1.10 Click mouse buttonK0.20 Move cursor to Cut menu itemP1.10 Click mouse buttonK0.20 Mentally PrepareM1.35 Move cursor to before “ fox ” P1.10 Click mouse buttonK0.20 Mentally PrepareM1.35 Move cursor to Edit menuP1.10 Click mouse buttonK0.20 Move cursor to Paste menu itemP1.10 Click mouse buttonK0.20 TOTAL PREDICTED TIME Method Used Cut-and-paste-using-menus M=1.35 P=1.10 K=0.20

GOMS: Card, Moran, and Newell (CMN-GOMS)  Subgoal invocations and method selection are predicted by the model given the task situation  In program form – analysis is general and executable  Predicts operator sequence and execution time  Based directly on the Model Human Processor

DescriptionDuration (sec) GOAL: MOVE-TEXT …….GOAL: CUT-TEXT ……………GOAL: HIGHLIGHT-TEXT ………………..MOVE-CURSOR-TO-BEGINNING1.10 ………………..CLICK-MOUSE-BUTTON0.20 ………………..MOVE-CURSOR-TO-END1.10 ………………..SHIFT-CLICK-MOUSE-BUTTON0.48 ………………..VERIFY-HIGHLIGHT1.35 ……… GOAL: ISSUE-CUT-COMMAND …………………MOVE-CURSOR-TO-EDIT-MENU …………………PRESS-MOUSE-BUTTON0.10 …………………MOVE-MOUSE-TO-CUT-ITEM1.10 …………………VERIFY-HIGHLIGHT1.35 …………………RELEASE-MOUSE-BUTTON0.10 ………GOAL: PASTE-TEXT …………….GOAL: POSITION-CURSOR-AT-INSERTION-POINT …………………MOVE-CURSOR-TO-INSERTION-POINT1.10 …………………CLICK-MOUSE-BUTTON0.20 …………………VERIFY-POSITION1.35 …………….GOAL: ISSUE-PASTE-COMMAND …………………MOVE-CURSOR-TO-EDIT-MENU …………………PRESS-MOUSE-BUTTON0.10 …………………MOVE-MOUSE-TO-PASTE-ITEM …………………VERIFY-HIGHLIGHT1.35 …………………RELEASE-MOUSE-BUTTON0.10 TOTAL PREDICTED TIME CMN-GOMS

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  Modeling Specific Components  GOMS Limitations  Summary

Extending GOMS: Grammars  Explicitly represent knowledge a user needs to translate from goals to actions.  Task-Action-Grammar (TAG) by Payne and Green  Model of content knowledge rather than a full system to generate user performance estimation. However, we can measure by number of rules.

Extending GOMS: Grammars  TAG Example for EMACS: Task[Direction, Unit]  Symbol[Direction] + Letter[unit] Symbol[forward]  “cntl” Symbol[backward]  “meta” Letter[word]  “W” Letter[character]  “C”  Task: Move one word forward. Task[forward, word]  Symbol[forward] + Letter[word]  “cntl” + “W”  “cntl-W”

Extending GOMS: Production Systems  Like grammars but models a goal stack and working memory.  Tedious to write but can be fed into a program to automatically check for completeness and accuracy.  Can predict errors and learning time behavior.

Extending GOMS: Production Systems  Production to see if a closing JOIN statement is needed: Rule 1:(StartUp.SeeifJoinNeeded IF ((GOAL SeeIfJoinNeeded) (NOT(NOTE SeeingIfJoinNeeded TRUE)) THEN ((Add NOTE SeeingIfJoinNeeded TRUE) (Add STEP CountTables))) Rule 2: (CountTables ((DoTask Count NumberOfTables *NumberOfTables) (Add NOTE NumberOfTables *NumberOfTables) (Delete STEP CountTables) (Add Step AddJoinNote))) Insert into Working Memory Delete from Working Memory

Extending GOMS: Learning  How to estimate time to learn?  One solution: Soar (UMICH) From the FAQ: “ Soar has also been used for modeling learning in many of these tasks; however, learning adds significant complexity to the structuring of the task…”

Extending GOMS: Natural GOMS Language (NGOMSL)  Structured natural language notation  Based directly on the Cognitive Complexity Theory (Kieras and Polson) Allows GOMS to model working memory (WM) and setup subgoals Unlike CMN-GOMS, provides quantitative predictions about time to learn each new piece of a task.

Description Method for goal: Move text Step 1. Accomplish goal: Cut text Step 2. Accomplish goal: Paste text Step 3: Return with goal accomplished Method for goal: Cut text Step 1. Accomplish goal: Highlight text Step 2. Retain that the command is CUT, and accomplish goal: Issue a command Step 3: Return with goal accomplished Method for goal: Paste text Step 1. Accomplish goal: Position goal at insertion point Step 2. Retain that the command is PASTE, and accomplish goal: Issue a command Step 3: Return with goal accomplished Selection rule set for goal: Highlight text If text-is word, then accomplish the goal: Highlight word If text-is arbitrary, then accomplish goal: Highlight arbitrary text Return with goal accomplished Method for goal: Highlight word Step 1. Determine position of middle of word Step 2. Move cursor to middle of word Step 3. Double-click mouse button Step 4. Verify that correct text is selected Step 5. Return with goal accomplished DescriptionDuration (sec) Method for goal: Highlight arbitrary text Step 1. Determine position of beginning of text1.20 Step 2: Move cursor to beginning of text1.10 Step 3: Click mouse button0.20 Step 4: Determine position of end of text.0.00 Step 5. Move cursor to end of text1.10 Step 6. Shift-click mouse button0.48 Step 7. Verify that correct text is highlighted1.20 Step 8: Return with goal accomplished Method for goal: Position cursor at insertion text Step 1. Determine position of insertion point1.20 Step 2. Move cursor to insertion point1.10 Step 3. Click mouse button0.20 Step 4. Verify that correct point is flashing1.20 Step 5. Return with goal accomplished Method for goal: Issue a command Step 1. Recall command name and retrieve from LTM the menu name for it, and retain the menu name Step 2. Recall the menu name, and move cursor to it on Menu bar 1.10 Step 3: Press mouse button down0.10 Step 4: Recall command name, and move cursor to it1.10 Step 4: Recall command name, and verify that it is selected 1.20 Step 5: Release mouse button0.10 Step 6: Forget menu name, forget command name and return with goal accomplished TOTAL PREDICTED TIME 16.38

Extending GOMS: Parallel Processes  Cognitive processes are not always sequential Clerks imprinting checks often realize an error two checks past When typing, you often realize an error while typing the next sentence or letters

Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS)  Predicts a substantially shorter execution time than the other models. Allocates less time for “ prepare for action ” type operations. Allow parallel processes.  Requires analyst-supplied methods.  Uses Critical Path Analysis to investigate parallel processes

Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS)  Collect-call example 1, operator hits a “ collect-call ” key and says “ Thank you ” to customer:  You can save time by repositioning the key for faster access in the sequential example, but not in the parallel example. 1 Courtesy of Newman, Lemming ’ s TAO (Toll & Assistance Operator) study

Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS)  Critical Path: a connected sequence that represents the greatest total time and therefore determines the overall time for a task.  Critical Path 1 below is = 2.96 seconds 1 Courtesy of Newman, Lemming ’ s TAO (Toll & Assistance Operator) study

GOMS Family: Summary KLMCMNNGOMSLCPM Architectural Basis Simple Cognitive Architecture Model Human Processor Cognitive Complexity Theory Model Human Processor, assume expertise in use Goal HierarchyImplicitExplicitImplicit Models Learning/TransferNo YesNo Models Parallel ProcessesNo Yes Assigned Mental Time Yes, use operator M NoYes Yes, but very short for expert users Notation Used Primitive Operators Programming Language Natural Language Schedule/PERT chart

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  Modeling Specific Components  GOMS Limitations  Summary

Modeling Specific Components  3 general classes Memory & Cognition Motor movements Perception

Goals IntentionEvaluation expectation Execution Mental Activity Physical Activity Perception Interpretation 7 steps 1 of user activities involved in computer-based tasks Action Specification 1 Norman, D. (1986)

Action Specification Goals IntentionEvaluation MEMORY: Retrieve a unit from long term memory expectation COGNITION: Execute a mental step Choose among methods Mental Activity

Memory & Cognition: Memory Retrieval  GOMS provides modeling of Memory Retrieval Time to retrieve next unit of information  Moving information from long-term memory into working memory 1 2 ≈ 1350 … D12)

Memory & Cognition: Execution of a Mental Step  GOMS allows explicit representation of mental steps of a task (the “ Cognitive Processor ” ): Retrieval of goal Find the max value in a column Select a method to achieve the goal Retrieval of motor movements necessary to execute the command Execution of each of the chosen commands Use the “ MAX ” formula Type the formula ≈ 70 msec

Memory & Cognition: Method Decision  Hick ’ s Law T = k log 2 (n+1), k ~ 150 msec n = # of choices Time to make a decision is roughly proportional to the log of the number of choices  However, determining T is problematic Spreadsheet task where parameters in a formula are can be indicated via numerous methods  Hick ’ s law predicts 200 msec  Real time is 2 seconds  → Order of magnitude difference!

Memory & Cognition: Method Decision  Choice is a complex task that requires many cognitive steps  Steps differ task to task

Action Specification Goals IntentionEvaluation MEMORY: Retrieve a unit from long term memory expectation COGNITION: Execute a mental step Choose among methods Execution Mental Activity Physical Activity MOTOR MOVEMENTS: Keystroke Point Move hands

Motor: Key Input  Parameters of keyboard input based on Skill of the typist  Best Typist (120 wpm): 80 msec  Worst Typist: 1200 msec Predictability & continuity of the text to be typed  Typing random letters: 500 msec

Motor: Mouse Movement  Fitts ’ s Law is a robust predictor of mouse movement  Sometimes distance metric is not clear- cut Nested menus

Motor: Applying Fitts ’ s Law  Fitts ’ s law recommends Larger target sizes Smaller distances to targets Usage of corners and edges (they have “ infinite ” height and width)  Macintosh menus are faster than Windows/Unix style menus because they lie on the screen edge

Motor: Applying Fitts ’ s Law Target size grows as distance from cursor ’ s position increases Borders for shorter selection time Fittsized Menus

Motor: Fisheye Model  Provide local context against a global context  Focuses on screen space versus user ’ s attention  3 properties Focal point Distance from focus, D Level of detail, LOD  Degree of Interest Function to determine whether to display an item or not and its size

Motor: Fisheye Menu  Good for browsing tasks  Allows one to present entire menu without having to use hierarchies or scrolling  Longer learning curve  /hcil/fisheyemenu/fishe yemenu-demo.shtml

Motor: Hand Movements  Switching between keyboard and mouse ≈ 360 msec  Differences in times due to distance from home position on keyboard and the size of the targets Joystick ≈ 260 msec Arrow keys ≈ 210 msec

Action Specification Goals IntentionEvaluation MEMORY: Retrieve a unit from long term memory expectation COGNITION: Execute a mental step Choose among methods Execution Mental Activity Physical Activity MOTOR MOVEMENTS: Keystroke Point Move hands Perception Interpretation PERCEPTION: Perceive Saccade

Perception  Recognition or perception Measure the time to respond to stimuli  Responding to lights  Recognizing words  Saccade: fast movement of eye, head, etc. Measure the time to move and take in information in each jump  Eye jerking around, scanning or moving to the next location

Perception  An example: spreadsheet perception Looking for cell addresses and retrieving data 230 msec 130 msec, store row label 230 msec 130 msec, store col label 230 msec 1350 msec, retrieve row & col label Total: 2300 msec

Summary of Cognitive Parameters Retrieve from memory1200 msec Execute a mental step70 msec Choose among methods1250 msec Enter a keystroke230 msec Point with a mouse1500 msec Move hands to mouse360 msec Perceive100 msec Make a saccade230 msec

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  Modeling Specific Components  GOMS Limitations  Summary

GOMS Limitations: User Skill Level  Nonskilled or casual users Current GOMS is best applied towards skilled users.  Transfer and training from one system to another. Example: Transfer GUI operational skills between MacOS, Windows and KDE.

GOMS Limitations: Errors  Even skilled performers make mistakes  Errors are probably caused by an overload of Working Memory (WM) or the goal stack.  The higher the WM load, the more errors.  How to model users adapting to errors arising from interface design? Users write down critical cell in a spreadsheet so that it can be scanned quickly the next time. LotusIFPS WM Load1914 Error (%)146

GOMS Limitations: Parallel Processes  Must be careful of over simplifying assumptions: “A character on the same hand cannot be initiated with a cognitive operator until the motor processor execution of the previous character is complete” – John (1998)  Psych literature doesn’t support the above!  Critical path method requires fine grained characterizations of task dependencies and parameters.

GOMS Limitations: Cognitive Processes  Cognitive processes are unclear Gestalt principles exist, but many more factors exist which influence user ’ s interpretation of screen

GOMS Limitations: Other Problems  Usability  Functionality  Fatigue  Mental Workload  Individual differences

Some Work Has Addressed This Topic Straightforward Extension Seems Possible Cognitive Science Does Not Inform Us Requires Another Kind of Modeling Nonskilled users−X−− LearningXX−− ErrorsXX−− Cognitive Processes XXX− Parallel Processes XXX− Mental Workload−X−− Functionality−−−X Fatigue−−X− Individual Differences X−−− Acceptance−−−X Fit to organizational life −−−X

Outline  Cognitive Modeling  Introduction to GOMS  GOMS Extensions  GOMS Limitations  Modeling Specific Components  Summary

Summary  In HCI, GOMS is, by far, the most detail oriented modeling method for user activities. Great for getting quantitative and qualitative metrics. Easy to explain results. Once constructed, easy to modify in later design iterations.  Readily available tools are scarce, those that do exist have a high learning curve (e.g. CPM-GOMS)  Not as easy as heuristic analysis, walkthroughs or guidelines.  Only works for goal-directed tasks.  However, GOMS has been highly successful in applications where human interaction performance is of utmost importance.

Summary  Simulations of airplanes and helicopters in simulated theatres of war (STOWs) with SOARS  Sun ’ s webpage, CAD, word processors, mobile phone input methods, etc.  Project Ernestine: Adding new, “ improved ” workstations for Telephone Operators CPM-GOMS revealed that the new workstations would have cost an additional $2 million a year to operate!

*Figures from The Precis of Project Ernestine (CHI ’ 92) by Wayne D. Gray et al.

Summary  GOMS model is a predictive, descriptive and prescriptive model 1. Predicts the time it will take user to perform a task 2. Describes the way a user performs tasks on a system 3. Prescribes ways to develop of training programs and help systems

Questions or Comments?