1. Scaled Worlds: Tractable, Realistic, and Engaging? Wayne D. Gray, Ph.D. George Mason University Human Factors & Applied Cognition gray@gmu.edu

2. gray@gmu.eduTractable, Realistic, & Engaging?Slide 2 Motivation (1) Outstanding in My Field –Ft. Hunter-Ligget, NTC, Ft. Knox, Ft. Benning, Ft. Hood, Schofield Barracks, W. Germany The Unfinished Revolution –Problem Finding (Newell & Simon, 1972) Protocol Analyses Real-World Experts Real-World Novices –Unified Theories of Cognition (Newell, 1973)

3. gray@gmu.eduTractable, Realistic, & Engaging?Slide 3 Motivation (2) Resistance from Academic Psychology? –Traditional Experimental Psychology slowing yielding (work well-accepted by Cognitive Science Society types) Resistance from Human Factors Psychology? –Traditional HF Psychology -- Suspicious of Theory –Ethnographic HF Psychology -- Suspicious of Abstractions of the Task Environment

4. gray@gmu.eduTractable, Realistic, & Engaging?Slide 4 Topic Simulations of the Task Environment How do various Task Environment Simulations differ? What inferences do different types of Task Environment Simulations permit us to draw?

5. gray@gmu.eduTractable, Realistic, & Engaging?Slide 5 Outline The Problem Distinctions, Dimensions, Definitions Examples Scaled Worlds

6. gray@gmu.eduTractable, Realistic, & Engaging?Slide 6 The Problem In field research there is too much complexity to allow for definite conclusions; whereas, in laboratory research there is too little complexity to allow for interesting conclusions. Brehmer, B. & Dörner, D. (1993). Experiments with computer-simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Computers in Human Behavior, 9(2-3), 171-184.

7. Three Dimensions of Simulated Task Environments Researcher Real-World Task Participant

8. gray@gmu.eduTractable, Realistic, & Engaging?Slide 8 Definitions: Scaled World Beginning with a Complex, Real-World Task –A scaled world abstracts certain functional relationships, while paring away others –Can be multiple scaled worlds of the same complex, real-world task that differ on which functional relationships are abstracted and which are pared away –The nature of the research question determines what is abstracted and what is pared away

9. gray@gmu.eduTractable, Realistic, & Engaging?Slide 9 Definitions: Tractable The complexity issue –Can the researcher pursue the question of interest? Includes issues such as –Data collection (the right data, at the right grain size, with the right timestamp) –Learnability issues –Usability issues –Computational cognitive modeling issues -- can the model interact with the system?

10. gray@gmu.eduTractable, Realistic, & Engaging?Slide 10 Definitions: Realism The simulated task environment is realistic to the situation to the degree to which experiences encountered in the environment will occur in the real- world

11. gray@gmu.eduTractable, Realistic, & Engaging?Slide 11 Definitions: Engagement The simulated task environment is engaging to the degree to which it involves and affects the participants; that is, the degree to which they take it seriously

12. gray@gmu.eduTractable, Realistic, & Engaging?Slide 12 Dimension

13. Dimensions

14. Dimension

15. gray@gmu.eduTractable, Realistic, & Engaging?Slide 15 Dimension

16. gray@gmu.eduTractable, Realistic, & Engaging?Slide 16 How Simulated Task Environments vary on Tractability, Realism and Engagement Four examples –Flight Simulator –VCR –Argus –Ned Each is followed by a discussion of whether it is or is not a scaled world

17. gray@gmu.eduTractable, Realistic, & Engaging?Slide 17 Example: Flight Simulator As High Fidelity as $$$ and current technology can provide Is it a scaled world? NO

18. gray@gmu.eduTractable, Realistic, & Engaging?Slide 18 Example: VCR -- A High Fidelity Simulation of a Very Simple Task

19. gray@gmu.eduTractable, Realistic, & Engaging?Slide 19 Gray, W. D. (in press). The nature and processing of errors in interactive behavior. Cognitive Science. http://hfac.gmu.edu/Publications/pubs.html Is the VCR a Scaled World? No -- more like a high fidelity simulation of a low complexity task

20. gray@gmu.eduTractable, Realistic, & Engaging?Slide 20 Example: Argus

21. gray@gmu.eduTractable, Realistic, & Engaging?Slide 21 Is Argus a Scaled World? No -- Lacks the relation to a complex, real-world task that defines a scaled world It is a Synthetic Environment –Focus is on the study of interactive behavior under conditions of high workload

22. gray@gmu.eduTractable, Realistic, & Engaging?Slide 22 Example: Ned

23. gray@gmu.eduTractable, Realistic, & Engaging?Slide 23 Is Ned a Scaled World? Yes

24. gray@gmu.eduTractable, Realistic, & Engaging?Slide 24 Design Goals for Ned Has been carefully designed to abstract certain functional relationships of a complex, real-world task The functional relationships abstracted were defined by the goals of the study Most other functional relationships have been pared away Specifications for Ned arose from –Studying and listening to submarine Approach Officers

25. gray@gmu.eduTractable, Realistic, & Engaging?Slide 25 Research Goals for Ned (1) Functional Relationships –Project goals entail the description and simulation of the cognitive processes and memory structures used by Approach Officers in locating enemy submarines Impact on Cognitive Theory –To be able to say something new and different about the expertise required for performing a complex, event-driven task

26. gray@gmu.eduTractable, Realistic, & Engaging?Slide 26 Research Goals for Ned (2) Generalizability –Our goal is to be able to generalize the data and models of Ned to a new, unknown submarine, with reduced manning, and new instrumentation Caveat: generalizability may be hindered by excessive attention to mundane realism (DiFonzo, Hantula, and Bordia 1998)

27. gray@gmu.eduTractable, Realistic, & Engaging?Slide 27 Ned on the Tractability Dimension Ten displays with no overlapping information Each object clicked is timestamped to nearest tick (60hz) and saved to a log file Researchers can play back Approach Officer scenarios Computational cognitive models written in ACT-R can use Ned in same manner that our Approach Officers do.

28. gray@gmu.eduTractable, Realistic, & Engaging?Slide 28 Ned on the Realism Dimension We decreased realism –by paring away the functional relationships provided by the Approach Officers interactions with his crew and boat But, we maintained an information environment in which the functional relationships essential to our goals were preserved Used scenarios that represented situations familiar to the Approach Officer

29. gray@gmu.eduTractable, Realistic, & Engaging?Slide 29 Ned on the Engagement Dimension Ned maintains a reasonable level of engagement –but only for those with much prior submarine experience -- i.e., not suitable for college sophomores (or for this audience!!) The engagement arose from presenting Approach Officers with scenarios, information types, and displays that were recognizable as similar to those encountered in real missions

30. Dimensions Revisited

31. gray@gmu.eduTractable, Realistic, & Engaging?Slide 31 Summary: Tractability Tractability –Microworld & Synthetic Environments Scaled World > High Fidelity Simulation –Tractability is defined by the needs of the research question

32. gray@gmu.eduTractable, Realistic, & Engaging?Slide 32 Summary: Realism Realism for scaled worlds –Hi-Fi simulation > Scaled World > Microworlds or Synthetic Environments –The realism of scaled worlds is a focused subset of a more complex, real-world task. –Focus is on a limited set of functional relationships with a limited set contextual elements (e.g., scenarios, information types, and displays)

33. gray@gmu.eduTractable, Realistic, & Engaging?Slide 33 Summary: Engagement Engagement for scaled worlds –Scaled worlds present schematics of the real world situations –Participants supply the engagement via their motivation and knowledge of the task and domain –Will not be more engaging than the real task

34. gray@gmu.eduTractable, Realistic, & Engaging?Slide 34 The End http://hfac.gmu.edu