Modeling Space Fortress Abraham Anderson John Anderson Shawn Betts Dan Bothell Jennifer Ferris Jon Fincham Michelle Moon Ben Poole Yulin Qin

The Decomposition Hypothesis Performance on complex tasks can be decomposed into primitive sets of mental operations of the sort revealed in simple laboratory experiments. This assumption underlies much of the work on latency in basic neuroimaging analyses. Nonetheless there has been very mixed evidence on part-to-whole training. This is generally thought to reflect that fact that one cannot use “pure insertion” because the one task will change when you insert another. Most sophisticated analyses of reaction time and imaging rely on some sort of “additive factors” where you keep all the processes but just make them more or less difficult. Still within the context of a cognitive model one ought to be able to decide when pure insertion will work and when it will not.

Pure Insertion in Space Fortress Insert Mines or Fortress into game and check brain pattern Naïve Insertion: Full Game = Fortress+Mines However, this ignores the fact that we are really getting a double count of navigation and visual processing on the right hand side. -Mines+Mines -FortressOrbitMines +FortressFortressBoth Sophisticated Insertion: Orbit+Both=Mines+Fortress However, this ignores that there are points in time when one cannot do both fortress & mines in the both condition. One needs a model that will perform all of the conditions and look at its predictions for these conditions.

Game structure Played 16 games per session – Four of each condition 10 sessions 2/day Sessions 2 and 10 with fMRI – Analyzed the 20s “Maybe mine” segments Minute 13 IFF lettersConditionGame StartsMaybe Mine Minute 2Maybe Mine Minute 3Maybe Mine Minute 4Maybe MineGame EndsFeedbackFixation

Changes from “regular” SF Created the separate conditions Use a keyboard instead of joystick Fixed timing for mine onsets No need to choose between points and shots Fortress vulnerable while mine on screen

Points and Point Additivity The major point growth is in the fortress conditions There is a necessary sub-additivity in points

Modeling Plan Model the well practiced performance Consider learning later Use participants fortress condition for sessions 7-9 as the basis for the model’s navigating Run model through all conditions Compare it to session 9 behavioral data and session 10 fMRI data

Flight data

Keypresses per game

Keypress duration percentages

Scores Mine + Fortress = > Orbit + Full =

Model total scores

Changes to ACT-R Special device for interfacing with the game Using the Temporal module Modified – Motor module – Visual module – Imaginal module

Motor module changes Set motor feature prep time to 0ms – Suggested by Wayne based on Kieras’s 2009 ICCM paper Added mostly independent hands – Two execution paths Added individual finger queries ?finger-check> left-index busy Added new motor actions – Hold-key – Release-key – Release-all-keys – Delayed-punch

Delayed-punch Holds a finger down for a prespecified time +manual> isa delayed-punch hand left-or-right finger finger-name {delay time} Substitute for a strong visual-motor interface For the model (mostly) only a few times used – Orbiting 90ms, 125ms, 160ms – Firing 70ms, 90ms, 125ms – IFF 125ms, 160ms – Mine aiming special

Visual features Standard text items – foes, conditions, bonus symbols, vlnr count, iff letter “Featureless” indicators – no information other than presence or absence – fortress, fortress hit, fortress explosion, ship explosion, mine, mine explosion, trial over Ship – Contains most of the critical information

Ship chunk MY-SHIP ISA SHIP SCREEN-POS, VALUE, STATUS, COLOR, HEIGHT, WIDTH, X, Y ORIENTATION DIST VEL ANGLE VDIR HEX-HIT MINE-DIST SHOOT-AT-MINE HIT-FORTRESS-FIRST

New visual request Attend-and-track – Combines the separate move-attention and start- tracking requests into one action

New visual buffer Visual-search – A search buffer – Makes visicon chunks available for searching in production matching Visual-location information only – Similar to the old LHS !find-location! Less flexible than a visual-location request – No :attended, :nearest, or current Two primary reasons for adding it – Buffer stuffing not available due to tracking – Allows for modeling control over “stuffing” Utility can decide which feature is more important when co-occurrences (p start-playing =goal> isa goal state start-playing =visual-search> isa sf-visual-location kind ship ?visual> state free ==> +visual> isa attend-and-track screen-pos =visual-search … )

Using the visual-search buffer (p detect-bonus =goal> isa goal check-bonus nil =visual-search> isa sf-visual-location kind new-bonus =imaginal> isa game-state ==> =imaginal> +goal> check-bonus check) (p detect-hit =visual-search> isa sf-visual-location kind fortress-hit ?imaginal> state free =imaginal> isa game-state shot-count =count < shot-count 10 counted nil ==> !bind! =new-count (1+ =count) +imaginal> shot-count =new-count counted t) (p see-mine-onset =goal> isa goal state play =visual-search> isa visual-location kind mine =visual> isa ship ?imaginal> state free … ==> +imaginal-action> isa generic-action action predict-mine-shooting …)

Special Imaginal request Prediction for shooting the mine – 250ms, slightly more than a standard imaginal request – Uses existing imaginal-action buffer +imaginal-action> isa generic-action action predict-mine-shooting Determines through simulation how to best shoot at the mine given current ship details MINE-PREDICT ISA MINE-HEADING ACTION TURN FINGER RING DELAY DIST HIT-DIST

Mine aiming productions (p adjust-mine-heading-with-fortress =goal> isa goal cond2 "+fort“ =imaginal> isa mine-heading action turn finger =finger delay =delay dist =dist =visual> isa ship <= mine-dist =dist =visual-search> isa visual-location kind mine … ==> +manual> isa delayed-punch hand left finger =finger delay =delay …) (p reassess-mine =goal> isa goal =visual-search> isa visual-location kind mine =imaginal> isa mine-heading action drift hit-dist =delay =visual> isa ship < mine-dist =delay … ==> +imaginal-action> Isa generic-action action predict-mine-shooting …)

Model overview 77 productions covering eight basic tasks – Encoding pre-trial info and rehearsal – Attend ship then initial thrust and turn to start orbiting – Orbiting the center hex – Shoot at fortress and count hits – Encode and respond to bonuses – Identify mine, aim, and shoot at it – Correct orbiting problems (hit hex or too far out) – Detect ship destruction or trial over

Orbiting productions Model tries to stay aimed at the center and orbit clockwise about 95 pixels out Broken into groups by ship speed < 1.0 tries to speed up 1.0 – 1.7 maintains the normal orbiting > 1.7 tries to slow down

Normal speed (p right-norm-short =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 >= angle 5 <= angle 10 > vdir 92 ?manual> preparation free ?finger-check> left-index free ==> +manual> isa delayed-punch hand left finger index delay fast) (p right-norm-default =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 >= angle 5 <= angle 20 > vdir 88 ?manual> preparation free ?finger-check> left-index free ==> +manual> isa delayed-punch hand left finger index) (p right-norm-long =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 >= angle 15 <= angle 30 ?manual> preparation free ?finger-check> left-index free ==> +manual> isa delayed-punch hand left finger index delay slow) (p thrust-norm-default =goal> isa goal state play =visual> isa ship <= vel 1.7 >= vel 1.0 > dist 93 >= vdir 92 <= angle 2 >= angle -4 ?manual> preparation free ?finger-check> left-middle free left-index free left-ring free ==> +manual> isa delayed-punch hand left finger middle) (p left-norm-default =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 < angle -14 ?manual> preparation free ?finger-check> left-ring free left-index free left-middle free ==> +manual> isa delayed-punch hand left finger ring)

Consider the BOLD response Use the BOLD prediction tools built into ACT-R to determine activity in the modules Few changes needed – Made all goal and imaginal modifications in the model +’s so they’re counted – Added new buffers to track left and right hand actions separately Only tracks the execution – Modified visual tracking so that it is only periodically busy instead of constantly One attention shift per 2 seconds

Model’s BOLD motor mismatch

Curious…

Future work Address the issues with the model’s BOLD results in the motor system Model the performance on the early days and the learning that takes place over days