1. Learning Where to Look: An ACT-R/PM Model
Brian D. Ehret
ARCH Laboratory
Human Factors and Applied Cognition
George Mason University
ACT-R Workshop - August 7, 1999
Today I am going to present research conducted as a part of my dissertation, which as the title suggests, is focused on learning locations. Specifically the learning of interface objects such as menu items and buttons

2. Overview
HCI research has demonstrated that users learn the locations of interface objects
However, not much known about mechanisms underlying location learning
Systematically vary conditions under which location learning may occur in order to
Infer characteristics of location learning mechanisms
Embed these characteristics into a computational cognitive model using ACT-R/PM (Byrne & Anderson, 1998)
HCI research has demonstrated that users learn the locations of items on screen and use this information to improve performance.
Not much is known however, about HOW this learning takes place - that is the underlying mechanisms
The approach of my research has been to systematically vary conditions in which location learning takes place, use the resulting data to infer characteristics of the learning mechanisms, and them incorporate these characteristics into a computational cognitive model, built using ACT-R/PM, which learns locations, presumably in a manner similar to users
To date, I have conducted two experiments and built the model. Today I am going to describe the second of the two experiments and the accompanying model..
The experiment participants and model perform the same basic task, which I will demonstrate next.

3. Participants were shown a colored rectangle in the center of the screen containing a number of white x’s. The rectangle was one of 12 colors. Participants were instructed to click on the rectangle

4. When they did, 12 buttons appeared in a circle surrounding it
When they did, 12 buttons appeared in a circle surrounding it. The goal was to find and click the button which would make the white x’s the same color as the rectangle. There was one color associated with each of the 12 buttons.

5. To assist them, tooltips were always available
To assist them, tooltips were always available. If the participant held the cursor over a button for more than one second, a tip would appear.
When the correct button was clicked, the rectangle appeared solid and after a short delay a new color was shown.
Buttons remained in the same locations throughout the experiment.
There were four different label conditions, this one is called the no-label condition.
It is a PURE location learning condition in the sense that the only way to improve performance is to learn the association between the color and the location of the correct button.

6. Color-Match Condition
This is the color-match condition. It takes advantage of a phenomenon known in the perception literature as the pop-out effect. That is, to people with normal color vision, the correct button ‘pops-out’ and thus can be very quickly and easily located.
To the extent that the participants could rely on this effect, this condition provides no incentive to learn location. Thus, it lies at the opposite extreme from the no-label condition, in which participants would HAVE to learn locations.

7. Meaningful Text Condition
This is the meaningful text condition, in which the buttons were labeled with the appropriate color names. This condition is representative of a typical interface.

8. Arbitrary Icon Condition
Finally, this is the arbitrary icon condition. Here the buttons were labeled with icons that had no inherent relationship to the colors. This is arguably also representative of a typical interface.

9. Performance Time Results
Trial Time (sec)
This graph plots trial completion times over the 16 blocks.
As you can see, the four groups performed quite differently
The color-match group starts off fast and stays fast over the 16 blocks, suggesting a reliance on the pop-out effect throughout the experiment.
The meaningful text group is a bit slower initially, but shows rapid improvement and soon equals the performance of the color-match group
The arbitrary icon and no-label groups start off equivalently, but soon deviate from each other as the arbitrary group improves at a faster rate. The no-label group shows rather gradual improvement and doesn’t match the times of the other groups until block 11.
These trial time data clearly show large performance differences between these groups. The extent to which these differences are attributable to location learning will be explored via the eye data, which I will describe next.
So here is the fit of the performance data with the model.
Blocks

10. Finally, our participant has not only learned where the correct button is located but now can recognize it as such without accessing a tooltip

11. Overview of Model 35 rules for all four conditions
20% of rules common to all four conditions
Overlap between conditions ranges from
23% between Color-Match and Arbitrary to
86% between Arbitrary and No-Label
Interacts directly with software via ACT-R/PM
Relies on search vs. retrieve paradigm
Akin to compute vs. retrieve (arithmetic facts)
Preference for less costly strategy
Now that I’ve gone through the data I want to talk about the model. The model is comprised of 35 rules with a subset of the rules used to model the four conditions. About 1/5th of the rules are common to all four conditions - these rules mostly have to do with things like mouse movements and clicks recognizing when a trial has begun or has finished.
The color-match condition, which uses the fewest rules has about 23% overlap with the arbitrary condition which has the most - and in contrast, the arbitrary and no-label conditions share most of their rules.
The model, written using ACT-R/PM, interacts directly with the experimental software - sending and receiving screen events.
In the way of an overview, the model relies on a search vs retrieve paradigm similar to the compute versus retrieve paradigm used in modeling the learning of arithmetic facts.

12. Overview of Model (cont’d)
Declarative structures
Buttons
Locations
Labels
Colors
Learning Parameters
Base level learning (d=0.3)
Merging and retrieval
Associative strength learning (:al=1)
Source spread
There are three key chunk types in the model. The button chunks are goals - so there is a goal to find a button that applies a given color - each time the model locates and clicks the correct button and the goal is popped, if the button had been encoded before the goal will be merged with an existing goal - as a result there is one canonical chunk representing each button.
The visual location chunks are created by ACT-R/PM whenever a find-location command is executed - these are later retrieved by the model
Finally, there are label chunks which are encoded as the model attends to labeled buttons.
Base-level and associative activation learning were enabled with the decay parameter set slightly below its default value of .5 and with associative learning parameter set to a default value of one. In terms of other parameters, activation noise was set at .7 and the retrieval threshold was set to 2.3.

13. Search Cost and Strategies
Search Cost - number of buttons evaluated per trial

14. Search Cost - Buttons Evaluated
Number of Buttons Evaluated
The analysis of the eye data is structured around two sub-components of this task. The first is search cost. Search cost is simply the number of buttons the participant must attend to before locating the correct button.
This graph plots the average number of buttons attended per trial over blocks.
The color-match group attends to a low and constant number of buttons per trial over the 16 blocks whereas the other groups begin at chance levels and gradually work their way down.
The differences between the meaningful, arbitrary, and no-label groups are not statistically significant indicating that these three label conditions are equivalent in terms of their search cost.
The number of buttons attended is also a measure of location learning. Thus, these data also indicate that high search cost groups learned locations at the same rate over blocks.
Blocks

15. Search Phase Summary Pre-attentive Search Controlled Search
Search for location of button with correct color
ACT-R/PM find-location command
Controlled Search
Attempt to retrieve a past use of the correct button and its associated location chunk
If retrievals fail, attend to random button
Retrieval attempted first (search vs. retrieve)
Recall that the search phase involves determining which button to evaluate first.
The color match condition uses ACT-R/PM find-location command to conduct a pre-attentive search for the correct button - it just searches for a button with the correct color-feature
Each time the model attends to a button the button chunk and the location chunk receive a base-level activation boost - this is an effect of the merging process
In the controlled search conditions the model first attempts to retrieve the button chunk representing the currently needed button as well as the location of the button - if it succeeds then attention is directed to the location of the button - if it fails then attention is moved the the location of a randomly selected button
During this process, base-level activations will receive a boost - they will receive a larger boost of the chunks were successfully retrieved. If retrieval succeeds, then interassociative strength from the activation sources, here the chunk representing the rectangle color and the label chunk (if retrieved), to the retrieved chunks increases.
With all of this strengthening of activation retrieval of the relevant chunks eventually succeeds.
So lets see how the model does in accounting for the buttons attended data

16. Evaluation Cost and Strategies
Evaluation Cost - time required to determine if currently attended button is currently needed

17. Evaluation Cost - Time Per Button
Average Time Per Button (sec)
The other sub-component of this task is the evaluation cost, which is simply how long it takes to determine if the currently attended button is the one being sought after.
This graph plots average time spent per button over the 16 blocks.
As you can see the meaningful text and color-match groups take an equivalent and relatively constant amount of time evaluating the buttons, whereas the arbitrary and no-label groups start off more slowly and improve over blocks.
There is a large difference between the arbitrary and no-label groups. The arbitrary groups shows somewhat rapid improvement whereas the improvement for the no-label group is much more gradual. This is attributable to a lingering reliance on tooltips on the part of the no-label group.
Blocks

18. Evaluation Phase Summary
Color-Match and Meaningful always label match
Arbitrary label matches only if label retrieved in search phase
No-Label and Arbitrary (if label not retrieved) try location assessment
Attempt to retrieve a past use of the currently attended button and its associated location chunk
If retrievals fail, then wait for ToolTip
Retrieval attempted first (wait vs. retrieve)
The evaluation phase entails determining if the currently attended button is the right one. For the color-match condition, this is a entails encoding the color of the current button and comparing that color to the rectangle color
For the meaningful text condition the model encodes the current label and compares that to label chunk it retrieved back in the search phase. Retrieval of the label chunk always succeeds - the assumption being that the name of the color is highly associated with the perceptual experience of the color.

19. Implications Locations encoded as a by-product of attention
Default ACT-R/PM behavior
Location knowledge, once encoded into memory, is like any other knowledge
Subject to the same learning mechanisms
Task performance characterized as rational
Attempt less costly strategies first
Pre-attentive < retrieve location < search [search phase]
Label-match < location assessment < tip [evaluation phase]
Results in differential location learning
One of the theoretical controversies in the spatial memory literature is whether or not spatial information is automatically encoded, as was claimed by Hasher and Zacks in 79. Tests of this claim usually involve a single presentation of a stimulus and then a recognition or recall test for the accuracy of location memory. What this method taps is the ability to retrieve the location from memory, not whether or not the location information was encoded.
From an ACT-R perspective, the distinction is clear - ACT-R/PM encodes the location chunks automatically as a by product of attention. The ability to reliably retrieve an encoded location chunk is contingent upon a gradual increase in activation levels.
Which leads into the next point - once the location chunk has been encoded in memory, it is treated like any other declarative memory structure - its parameters must be learned via the same means that parameters are learned for other chunks. Thus location learning requires repetition or explicit rehearsal strategies.
These implications aren’t derived from the model per se but rather from the ACT-R/PM architecture - encoding is automatic via RPM and declarative learning is responsible for increasing retrieval probabilities.