12/6/2018 8:38:35 AM An IMPRINT Model of the Digit Data Entry Task MURI Annual Meeting 9/27/06 Carolyn Buck-Gengler and William Raymond Department of Psychology and Center for Research on Training University of Colorado at Boulder

12/6/2018 8:38:35 AM A basic data entry experiment: Healy, Kole, Buck-Gengler, & Bourne (2004) 10 blocks of 64 #s Numeral format only NO intertrial pause Keypad only Exp 1 (modeled in ACT-R) Exp 2 (modeled in IMPRINT) Repetition of numbers each # repeated 5 times (different sets each half) all unique #s Use of non-dominant (left) hand Left hand only Both hands used (R-R, R-L, L-R, L-L) Break between halves 0s, 60s, 300s, self-paced 300s Description of the study and the particular experiment we are currently modeling-- Coty is modeling Experiment 1; we are starting with Experiment 2 but plan to go to Experiment 1 after modeling all aspects of Experiment 2 needed to compare with Experiment 1 Describing both to highlight some of the manipulations that differ between the two Fatigue: Two aspects of the basic DDE experiment were changed or manipulated in this experiment to try to induce fatigue: 1) No intertrial pause: Earlier experiments had an intertrial pause of 500 ms. In these experiment there was no intertrial pause 2) use of left (non-dominant) hand -- manipulated in E2; only left hand in E1. Earlier experiments used right (dominant) hand Experiment 1: each of the 64 numbers was repeated 5 times; different set sof 64 numbers were used in each half December 6, 2018

Results from Healy et al. (2004, Experiment 2) to be fit 12/6/2018 8:38:35 AM Results from Healy et al. (2004, Experiment 2) to be fit Speed improvement Total RT (TRT) decreases across first half RTs increase across second half (for some keystrokes) Digit entry times: Keystroke 3 is slower than keystrokes 2 or 4 (and Enter Key) Effect of hand Left hand slower than right hand Accuracy decline Speed/accuracy tradeoff - errors increase across blocks Types of errors (by output length) We usually refer to K3 slower than K2, K4, enter as “chunking” Also looking at serial position of errors, RT for error trials December 6, 2018

access representation 12/6/2018 8:38:35 AM A cognitive model of digit data entry  1395 Language/reading Symbolic representation Motor planning Fine motor manipulation read digit represent digit access representation plan response access plan execute plan  1395 December 6, 2018

Modeling Approach Initial goal: fit results from this experiment 12/6/2018 8:38:35 AM Modeling Approach Initial goal: fit results from this experiment i.e., effects of skill practice, hand manipulation Both speed and accuracy effects for all response measures Add one manipulation at a time to expand the model From here move to other similar experiments At each step, verify previous measures still replicated Model the cognitive processes underlying the task Decompose task into cognitive components (mappable to MURI task taxons) Model individual variation (not aggregate data) Granularity of processing is digit (not trial or block) Probabilistic simulation We started modeling with this experiment because it has the fewest manipulations -- model the very basic task Next three slides will expand on these three points: • high level summary of the results to be fitted by the model • sequence of steps in expanding the model • cognitive model of the task December 6, 2018

Speed/Accuracy tradeoff 12/6/2018 8:38:35 AM Modeling Sequence Right hand only, first 5 blocks, No repetition Speed/accuracy tradeoff Other manipulations predictions Add Left hand, first 5 blocks, No repetition Effect of hand on speed Left hand only, Repetition Repetitious practice ALL experiments currently modeled or in immediate future involve keypad only (typing location) and numeral presentation only. Perhaps further out we will look at row (typing location) and word/other presentation formats. First effort: just RTs: Speed improvement due to skill practice only Adding errors: Speed/accuracy tradeoff Possible predictions: R hand with repetition (keypad, numeral presentation) Other manipulations: Typing Location Feedback Presentation format (word/ math problems, etc) Add 2nd half No repetition Speed/Accuracy tradeoff and effect of hands over longer time December 6, 2018

The IMPRINT model Main network: the “computer” 12/6/2018 8:38:35 AM The IMPRINT model Main network: the “computer” Bookkeeping Set up subject-specific parameters and variables “Present” each trial Goal network: the “subject” Set up trial-specific parameters and variables Main subnetwork: one trial Components (“tasks”) for each cognitive process that goes on in a trial In Imprint, There are two networks running in parallel. One represents the “computer” presenting the data -- which really is used for bookkeeping at the experimental and subject level, and to keep the simulation running. Each “run” of the IMPRINT model corresponds to a unique subject. The other represents the subject. It is only the lowest level of the subject network that is of primary interest today. This network represents the simulation of ONE trial. It is executed over and over again to simulate an entire experimental run of one subject, and that larger network is, in turn, run over and over again to simulate multiple subjects. Subject characteristics, such as an average speed or hand used, are set at the outermost (experiment) level. The basic unit in IMPRINT is the “task”. This sometimes corresponds 1 to 1 with the idea of component, and sometimes “component” encompasses more than one task. The term component will be used for both. December 6, 2018

“Subject” Network: Do one trial 12/6/2018 8:38:35 AM “Subject” Network: Do one trial read represent access rep plan resp access plan execute plan read plan motor This network shows the embodiment of the cognitive model into the IMPRINT network. At a high level, one trial of DDE involves the following: ** read/“represent” the 4-digit number, ** plan the motor response, and ** type it, followed by typing the enter key to end the trial. In our model, we have broken these three steps into 2 or more tasks each, and each subset of tasks is executed for each digit (the network loops over the pair once for each digit) or chunk of digits in the typing part. A separate task is placed in the network for the enter key. Flow -- based on “typical” trial with chunking into two 2-digit chunks: ** First, the trial is initialized in Initialize Trial (setting various counters to 0, updating the trial counters, determining error status of trial, and determining response times for each component). ** Next are the two subcomponents of reading a digit, ** Read 1 digit and ** Represent one digit. ** This pair is looped 4 times, as the enter key does not need to be read. ** After reading, the motor task is accessed for each digit. This is also broken down into two tasks, ** Access representation of 1 keystroke and ** Plan Motor response for 1 keystroke. This pair of tasks is looped 4 times. ** Finally, the digits are typed. Based on data collected in many experiments, we have assumed chunking, but have designed the network to allow for chunks of different sizes. A chunk size of two is the basic assumption. In general, the two steps of typing a key are ** Access Motor Plan for 1 chunk (executed once for each chunk, which is 4/chunksize) and ** Execute 1 keystroke, which is executed once for each digit. Type a chunk and Keystroke gatekeeper are there for bookkeeping and logic purposes. ** A final task Plan and Execute Enter embodies typing the enter key, followed by the computer/dummy task Cleanup after trial. At this point, the “goal” is finished, and control returns to the computer’s network, which then presents the next trial. type December 6, 2018

Mapping tasks to keystrokes 12/6/2018 8:38:35 AM Mapping tasks to keystrokes Init (K1) The time response data from the DDE experiments are as follows: Init time (= time since stimulus onset at which the first digit is typed), times for the 2nd, 3rd, and 4th digits (K2, K3, K4), and the time to the Enter key. [In some versions of the experiment, times for any keys beyond the 4th are also collected.] ** Init time is usually very long in comparison to the other digit typing times; ** K2 and K4 are very similar in time and shortest, and ** K3 is somewhat longer. ** Enter key time is also relatively very short. It is assumed theoretically that all the reading and most of the encoding and motor planning is captured in the Init time, with some small amount of planning/re-accessing done before the 3rd keystroke is executed (and thus being captured in the time for K3), leading to the chunking concept, that the four digits are grouped into smaller chunks of two digits for the purpose of remembering or entering them better. Thus, the times for all aspects of reading/encoding the digits and accessing the motor plan, as well as typing the first digit, must be included in the init time, and the extra time for K3 must be expressed in the network as something that only happens for the 3rd digit (and 1st) and not for the 2nd and 4th digits (nor the enter key). Thus, in the model, any tasks outside of actual typing for reading/planning/encoding/accessing, except those that could also contribute to K3, must take place before the first time the Execute 1 keystroke task is encountered. K3 K2,K4 Enter December 6, 2018

Implementing skill learning (RTs) 12/6/2018 8:38:35 AM Implementing skill learning (RTs) Main concept to model: Learning of the skill of typing 4-digit numbers, without repetition Two types of skill learning: Cognitive: Applying only to K1 and K3; independent of hand – applied in Access motor plan for 1 chunk Physical: Applying only to Execute 1 keystroke and Plan and execute Enter; more learning possible for L hand than for R hand Using power function Assumption: Learning occurs only on correct trials Here we are talking only about the response times, not accuracy, and the improvement of response times (learning) over time due only to skill/practice and not to repetition of numbers. The Model does not actually LEARN. Learning is SIMULATED by using a multiplier derived from an exponent, a parameter set in the model, applied to a count of trials, either good trials so far for cognitive learning, or all trials so far for physical learning. Each type has its own exponent, and when the left hand is used, the physical learning exponent is adjusted to allow for more learning for the left hand than the right hand. More learning for L than for R based on data Power function: decreases, approaches asymptote, assumed for human learning Assumption: Learning occurs only on correct trials (exponent applied only to a count of correct trials so far) Cognitive learning occurs only on correct trials (exponent applied only to a count of correct trials so far) Physical learning occurs on all trials December 6, 2018

Caveat Not yet modeling fatigue 12/6/2018 8:38:35 AM Caveat Not yet modeling fatigue Seems to counteract learning – at least for K1 – in second half Modulated by repetition as well, which will be addressed when we model Experiment 1 Role of fatigue in the first half, if it exists, is swamped by learning --> only evident in second half December 6, 2018

RT Assumptions Subjects’ TRTs from normal distribution 12/6/2018 8:38:35 AM RT Assumptions Subjects’ TRTs from normal distribution A TRT chosen for each, then used as the base for each of the trials for that subject A specific set of proportions of TRT assigned to each task Independence of tasks – time for each drawn separately from a skewed distribution around the proportion of the total time allotted to that task Penalty for the Left hand on the execute keystroke tasks (Execute 1 keystroke and Plan and execute Enter) These assumptions are BASED ON published results and other analyses of experimental data December 6, 2018

Total Response Time: Overall TRT 12/6/2018 8:38:35 AM Total Response Time: Overall TRT RESULTS compared to model, overall Overall Improvement across session halves Overall improvement across blocks in first half but not second -- 2nd half part not yet modeled, because we don’t have something for “fatigue” Note: whether overall mean increases or decreases between blocks 5 and 6 depends on how individual Ss randomly assigned to Hand and Switch groups Improvement across session halves – depended on combination of hands used in each half – could be explained by L being slower than R LR best improvement between halves RL got worse LL and RR intermediate improvement However, improvement of no switch and switch groups similar, so improvement was cognitive, not just motoric improvement of LL+RR similar to RL+LR, so improvement was cognitive, not motoric Always faster on L than on R Improvement is cognitive December 6, 2018

Total Response Time by groups 12/6/2018 8:38:35 AM Total Response Time by groups Improvement across session halves – depended on combination of hands used in each half – could be explained by L being slower than R LR best improvement between halves RL got worse LL and RR intermediate improvement However, improvement of no switch and switch groups similar, so improvement was cognitive, not just motoric improvement of LL+RR similar to RL+LR, so improvement was cognitive, not motoric Always faster on R than on L Improvement is cognitive Notes: Solid lines = Experimental data, dashed = model output Each color represents a different hand/switch combination December 6, 2018

12/6/2018 8:38:35 AM All keystrokes December 6, 2018

Init (K1) Init (K1): decrease in first half, increase in second half 12/6/2018 8:38:35 AM Init (K1) Init (K1): decrease in first half, increase in second half Cognitive learning; offset by fatigue in second half Hand not a factor Model: decrease in first half, fatigue not modeled yet so don’t have increase in second; Relative position of data due to our statistical Ss being drawn from a similar “population” but not being identical with the experimental Ss December 6, 2018

Execution time (average of K2, K3, K4): Overall RTs 12/6/2018 8:38:35 AM Execution time (average of K2, K3, K4): Overall RTs First, overall: Looks like 2nd half lower than first, less change within block (just as with experiment data) December 6, 2018

Execution time (average of K2, K3, K4): by groups 12/6/2018 8:38:35 AM Execution time (average of K2, K3, K4): by groups Improvement across session halves – depended on combination of hands used in each half – could be explained by L being slower than R LR best improvement between halves RL got worse LL and RR intermediate improvement However, improvement of no switch and switch groups similar, so improvement was cognitive, not just motoric improvement of LL+RR similar to RL+LR, so improvement was cognitive, not motoric Always faster on R than on L Improvement is cognitive Notes: Solid lines = Experimental data, dashed = model output Each color represents a different hand/switch combination December 6, 2018

Conclusion time (Enter key) 12/6/2018 8:38:35 AM Conclusion time (Enter key) Model maybe not declining as much as experimental data in second half December 6, 2018

Accuracy Overall accuracy: is trial correct or not 12/6/2018 8:38:35 AM Accuracy Overall accuracy: is trial correct or not Output length analysis: how many digits typed in error trial? Missed trial: responded with Enter key only Missed digit: typed fewer than 4 digits Wrong digit: typed 4 digits, but at least one wrong Extra digit: typed more than 4 digits Also will be looking at Location of error(s) (serial position) Are also splitting errors into Planning vs. execution errors, per Bill Raymond’s separate analysis December 6, 2018

Accuracy (errors): how modeled 12/6/2018 8:38:35 AM Accuracy (errors): how modeled At beginning of trial network, trial randomly assigned a status of either “correct” or “incorrect” Proportional to data Output length randomly selected from distribution Gives missed trial, missed digit, wrong digit, and extra digit trials Technical details of modeling Chance of randomly being assigned to “incorrect” grows with block number to model decrease in accuracy over the course of the experiment. Reflecting data from various experiments, the distribution of output length ensures most error trials have length 4; of the remainder, most are 3 or 5, and very very few are 0, 1, 2, or 6 or more. Errors are distributed between lengths 0 and 8. December 6, 2018

Accuracy – overall Overall accuracy: 12/6/2018 8:38:35 AM Accuracy – overall Experiment Results: Overall accuracy: - Errors increased from first to second half (directional test) • Errors modeled as increasing so that error rate increases overall when second half modeled - Error rate increase did not depend on which hand was in use • Must be some cognitive component, rather than limited to motoric processes • Comparison to similar analysis in E1 suggests that motoric processes may also still be involved (Motoric also involved because increase in errors between halves significant only by directional test and because no effect of block ) Model follows data fairly well December 6, 2018

Accuracy – output length 12/6/2018 8:38:35 AM Accuracy – output length RESULTS Output length: - Missed trial and missed digit trials very few, and not increasing across blocks: • Subjects did not type fewer digits as the experiment progressed and any fatigue increased MODEL Pretty good match Red: data from experiment Blue: model December 6, 2018

Insights gained in modeling process 12/6/2018 8:38:35 AM Insights gained in modeling process Where learning occurs Breaking it up into physical and cognitive components, and where to put it Hands not different, non-dominant just slower More complex understanding of the chunking process Very little skill learning? December 6, 2018

Summary Modeled fairly well Taken into account 12/6/2018 8:38:35 AM Summary Modeled fairly well RTs, both TRT and keystrokes, also chunking pattern With exception of effects of fatigue Accuracy findings Taken into account Learning/practice, both cognitive and physical Hand and hand switch December 6, 2018

Next on modeling agenda 12/6/2018 8:38:35 AM Next on modeling agenda Fatigue Buildup, especially with no break and no change of hand Dissipation: Effects of break and different length breaks Relationship with repetition (compare E1, E2) Learning through Repetition (Healy et al., 2004, E1) Test (E1) (repetition priming/item transfer) Feedback (Kole et al., 2006) Presentation format, etc. (other experiments) December 6, 2018

12/6/2018 8:38:35 AM Acknowledgments Personnel at MA&D for IMPRINT assistance, access, and facilitation Especially Bob Sargent and Ron Laughery December 6, 2018