1 George Mason University Human Factors & Applied Cognitive Program NGOMSL Modeling Psyc Week 10 Wayne D. Gray and Thomas Mayfield

2 George Mason University Human Factors & Applied Cognitive Program Next Week -- TUESDAY!! NGOMSL#1  Produce NGOMSL methods from the trace of the VCR (my trace not yours)  Are no pops in NGOMSL. Verifies may be suitable replacements. Yours to decide. NGOMSL#2  Create NGOMSL methods for the ManTel task  Should be one set of methods that can be applied to both interfaces and to 1 or 2 queries  Obviously, not all methods need to be used in each analysis  Produce an NGOMSL trace for Dialog-Box 1 query and PopUp 2 queries Readings  Paul Green’s 1999 Human Factors Ergonomics Society conference article

3 George Mason University Human Factors & Applied Cognitive Program Psyc Reprise of GOMS lecture 1 Levels of analysis based on Newell’s Time Scale of Human Action  Important to concept is that behavior at one level can be decomposed into behavior at next lower level Show what is added at various levels  Important to the argument that I am trying to make to convince you that analyses at increasing lower levels entail decreasing degrees of generality (or increasing degrees of specificity)  Programming VCR is a generally understood concept, but programming my VCR is very different than programming George Lukas’ VCR. Setting Tom’s watch or Mary Ann’s watch is very different than setting the clock in my car Implications?  Behavior is not as unconstrained as we would like to believe. As we get more and more specific in our description of behavior we find more and more constraints on behavior.

4 George Mason University Human Factors & Applied Cognitive Program Psyc645 - Overview of GOMS lecture 2 NGOMSL analyses  Unlike KLM, NGOMSL is a serious notation for discussing interactive behavior at levels lower than the unit task  Not simply a tool for analyzing design but a vehicle for representing low-level performance  Today we will work through several examples

5 George Mason University Human Factors & Applied Cognitive Program NGOMSL Natural Language GOMS  based on structured natural language notation and a procedure for constructing them  models are in program form Control Structure: Hierarchical goal stack Serial or parallel: Serial Level of Analysis:  In principle, as necessary for your design question

6 George Mason University Human Factors & Applied Cognitive Program NGOMSL - why? More powerful than KLM. Much more useful for analyzing large systems More built-in cognitive theory Provides predictions of operator sequence, execution time, and time to learn the methods

7 George Mason University Human Factors & Applied Cognitive Program NGOMSL - Overall Approach Step 1: Perform goal/subgoal decomposition Step 2: Develop a method to accomplish each goal  List the actions/steps the user has to do (at as general and high-level as possible for the current level of analysis)  Identify similar methods/collapse where appropriate Step 3: Add flow of control (decides) Step 4: Add verifies Step 5: Add perceptuals, etc. Step 6: Add mentals for retrieves, forgets, recalls Step 7: Add times for each step Step 8: Calculate total time

8 George Mason University Human Factors & Applied Cognitive Program NGOMSL - Example Car clock Presetting radio stations  simple  perverse

9 George Mason University Human Factors & Applied Cognitive Program Provides predictions of methods and operators used to complete a task. If you provide estimates of operator-duration, you can get predictions of error-free expert performance time. NGOMSL--Car Radio example (1)

10 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (2) Goal/subgoal hierarchy

11 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (3) Assumptions  User's hands are NOT on the buttons at the beginning.  Time to move hand & arm to time change level is estimated by 2ft "reach" from Barnes (1963): 410 msec.  D = device time, time for clock to move forward one number, = 500 msec (we estimate this on the next slide)  I am assuming a 12-hr clock. Seems good assumption for a car clock.  Radio is off at beginning and end.  Begin knowing the time you want to set the clock to.

12 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (4)  D = device time, time for clock to move forward one number, = 500 msec

13 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (2) (Duplicate slide)(Duplicate slide) Note that method for accomplishing the top level goal will have three subgoals and one “release” physical operatorNote that method for accomplishing the top level goal will have three subgoals and one “release” physical operator

14 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (5) Need methods for each of these three subgoals

15 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (6)

16 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (7)

17 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (8)

18 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (1)

19 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (2) Note differences between this and previous slide.

20 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (3)

21 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (4)

22 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (1)

23 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (2)

24 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (3)

25 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (4)

26 George Mason University Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (5)

27 George Mason University Human Factors & Applied Cognitive Program NGOMSL --comparison of optimal and perverse designs

28 George Mason University Human Factors & Applied Cognitive Program Cost-of-Knowledge Characteristic Function Card, S. K., Pirolli, P., & Mackinlay, J. D. (1994). The cost-of-knowledge characteristic function: Display evaluation for direct-walk dynamic information visualizations. In B. Adelson, S. Dumais, & J. Olson (Eds.), ACM CHI'94 Conference on Human Factors in Computing Systems (Vol. 1, pp ). New York: ACM Press.

29 George Mason University Human Factors & Applied Cognitive Program

32 George Mason University Human Factors & Applied Cognitive Program

33 George Mason University Human Factors & Applied Cognitive Program But, Card, Pirolli, & MacKinley did not do a complete GOMS analysis Only did enough of an analysis to determine that the “volume part” of the task was the repeated accesses of menus for each component (century, decade, year, month, week, day)

34 George Mason University Human Factors & Applied Cognitive Program Cost-of-Knowledge Function Small amounts of knowledge can be accessed quickly with access costs increasing as amount of knowledge accessed increases  ≈6.9 s for 1 day  ≈10.4 s for 7 days  ≈14 s for 30 days  ≈17.5 s for 365 days  ≈21 s for 3,562 days  ≈24.6 s for 36,525 days

A = Spiral Calendar as is B = SC w/ 2 s cycle time (was 3.5) C = Eliminate need to select “week” can do that when select day from the month calendar D = Eliminate “week” & improve cycle time

36 George Mason University Human Factors & Applied Cognitive Program Design Improvements? Speed up system response time (SRT)  Time to access = * Ncycles  Imply in text that it takes 1 s to point to item and 2.5 s for system to bring up item Eliminate some of the cycles -- seem redundant  Get rid of the week cycle  Have users bring up month and directly select day from the month calendar rather than first selecting week and then month Issue --  Implicit estimate of 2.5 s srt seems tediously long  Suggest need for a full NGOMSL analysis

37 George Mason University Human Factors & Applied Cognitive Program NGOMSL Methods for Spiral Calendar Top method and one subordinate

38 George Mason University Human Factors & Applied Cognitive Program NGOMSL Methods (cont’)

40 George Mason University Human Factors & Applied Cognitive Program How well does NGOMSL fit the data? Pretty well by r 2

41 George Mason University Human Factors & Applied Cognitive Program How well does NGOMSL fit the data? Pretty well compared to the equation they derived from the data

42 George Mason University Human Factors & Applied Cognitive Program How well does NGOMSL fit the data? But not too well when I assume a 2.5 sec system response time

43 George Mason University Human Factors & Applied Cognitive Program What Happened? Did the modeler cheat? Let’s examine the assumptions of the model Critical assumptions concern the time per cycle and the elements that compose the cycle Card et al., empirically derived 3.5 s per cycle The NGOMSL model, which assumed a 1 s SRT, predicted 3.1 s per cycle

3.1 s

45 George Mason University Human Factors & Applied Cognitive Program Where is the problem? For NGOMSL to accept 2.5 s SRT means either  accepting a very bad fit, or  changing the model But, what can be changed? My guess is that they never measured the SRT of their system In any event it is incredible that they assumed that all of the cycle time not devoted to pointing was attributable to SRT

46 George Mason University Human Factors & Applied Cognitive Program Two Morals to this Story? First  Even good researchers can be careless when they do not check their assumptions against a model  (& yes, Card, Pirolli, and Mackinlay are VERY good researchers) Second  Before investing lots of effort into jazzing up system response time, do a profile analysis with a complete GOMS model

47 George Mason University Human Factors & Applied Cognitive Program Next Week -- TUESDAY!! NGOMSL#1  Produce NGOMSL methods from the trace of the VCR (my trace not yours)  Are no pops in NGOMSL. Verifies may be suitable replacements. Yours to decide. NGOMSL#2  Create NGOMSL methods for the ManTel task  Should be one set of methods that can be applied to both interfaces and to 1 or 2 queries  Obviously, not all methods need to be used in each analysis  Produce an NGOMSL trace for Dialog-Box 1 query and PopUp 2 queries Readings  Paul Green’s 1999 Human Factors Ergonomics Society conference article

48 George Mason University Human Factors & Applied Cognitive Program End of NGOMSL