1. ACT-PRO Action Protocol Tracer A Tool for Analyzing Simple, Rule- based Tasks Wai-Tat Fu & Wayne D. Gray ARCH Lab George Mason University

2. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks2 Overview Problem What is ACT-PRO? Utility of ACT-PRO How it works

3. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks3 *where correct = behavior that the model knows about Problem Model tracing correct* behavior –Successfully developed and used to shadow students as they perform a task –Usually combined with tutors that keep students from straying off the correct path Can model tracing techniques be applied after-the-fact to large quantities of action protocol data? Model tracing errors, error detection, and error correction –Usually done “by hand” or not at all Can the tracing of errors, error detection, and error correction be handled as routinely as the tracing of correct behavior is handled by existing tutoring systems?

4. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks4 ACT-PRO Developed to handle a class of interactive tasks –Programming VCR, flight computer, ATM, etc Requires task analysis of interactive behavior –Format of task analysis is irrelevant; e.g., ACT-R, GOMS, Soar, or Hierarchical Task Analysis Currently implemented in visual basic for applications

5. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks5 The Task -- Overview ACT-PRO has been applied to action protocols collected from Ss programming a simulated VCR The VCR is representative of a large class of human- computer interaction tasks (e.g., ATMs, flight computers, and so on) The ACT-PRO approach should work with this class of interactive tasks

6. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks6 Level of Effort How does ACT-PRO compare to alternative methods of classifying action protocols? ≈ 12 days to program ≈ 12 min to setup and run ≈ 12 hr to verify results

7. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks7 Overview of Rest of Talk The task in more detail ACT-PRO -- how it works

8. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks8 The Task A simple rule-based task - programming a simulated commercial VCR

9. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks9 Device-Task Rule-Hierarchy Example: Start Time: 7:00; End Time: 8:00; Dow: Sat Channel: 11

10. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks10 Focus of the Study Focus of the study is on the explanation of human errors in terms of the model’s control structure By comparing the human data with the model’s prediction, we can characterize human behavior as matching or violating the model’s behavior ACT-PRO is needed to correctly categorize the correct behavior and to classify errors, error detection, and error correction

11. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks11 ACT-PRO

12. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks12 Matching Protocols to Model HOUR 10MIN MIN Push-goal-SET-StartHour Click On HOUR Pop-goal-SET-StartHour Push-goal-SET-Start10Min Click On 10MIN Pop-goal-SET-Start10Min Push-goal-SET-StartMin Click On MIN Pop-goal-SET-StartMin Button Clicks from Ss Model

13. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks13 ACT-PRO Two levels: –Keypress level Classify each button click as matching or violating the model –Goal level Parse button clicks into goal pushes and pops Classify each goal push and pop as matching or violating the model

14. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks14 Keypress Level Button clicks are classified as a type of hit or error* A transition matrix is used to parse the button clicks Within context defined by prior button clicks, clicks are unique to a certain goal * see Gray, W. D. (in press). The nature and processing of errors in interactive behavior. Cognitive Science... Btn 2 Btn 3 - hit Btn 1 - error..

15. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks15 Goal Level Parse button clicks into goal pushes and pops Classified each push and pop as a hit or error –Hits are labeled according to the ideal model (i.e., ACT-R productions or other task analysis) –Errors are further classified into: Push-error: Ss attempted to do something that the model would not do. Pop-error: Ss stopped working on a goal either before or after the model would stop.

16. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks16 Example 1 - Push-error Show Time End Time: 20:00 ButtonDisplay END-MODEE:EE HOUR19:00 HOUR20:00 10MIN20:10 10MIN20:20 10MIN20:30 10MIN20:40 10MIN20:50 10MIN20:00 DOW20:00 Error Hit Error Push-error Error detection and correction

17. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks17 Example 2 - Pop-error Show Time Start Time 21:15 ButtonDisplay START-MODEE:EE HOUR21:34 10MIN21:44 MIN21:45 10MIN21:55 10MIN21:05 10MIN21:15 Pop-error Goal not accomplished yet. Error detection and correction

18. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks18 Results KeypressGoal push/pop Total 95456750 Correct Behavior80.1%94.81% Errors, Error Detection, or Error Correction 19.9%5.19% Correctly Classified by ACT-PRO 98.95%99.30% VCR 2.0 -- Condition A: Ubiquitous Information

19. Fu & Gray ACT-PRO -- A tool for simple, rule-based tasks19 Conclusions ACT-PRO works very well Involves several orders of magnitude less effort than hand encoding Well suited to modeling behavior from a large class of interactive devices Next challenge: –Condition B - where target settings are covered by gray boxes and Ss need to click on the gray box to access the information –Condition C - Same as Cond B, but Ss are forced to memorize the target settings before the experiment