1. Is There a “Fast-track” Into the Black Box?: The Cognitive Models Procedure Robert R. Hoffman, Ph.D. John W. Coffey, Ed.D. Mary Jo Carnot, M.A. Institute for Human & Machine Cognition University of West Florida Poster presented at the 41st Annual Meeting Psychonomics Society New Orleans 16 Nov 2000

2. Abstract The Cognitive Models Procedure supports experts in crafting a model of their own reasoning, by relying on the presentation of "bogus" models. The goal is to have the expert re-construct a better model, which ideally will converge on the “modal model” of expert reasoning which includes the Duncker refinement cycle, recognition-priming, and situation awareness. The models are then cross-validated by: (1) field study of the observable behaviors that are entailed by models (e.g., the expert asserts that he always first inspects a certain data type), and (2). having a group of experts attempt to determine which of the models characterizes each of the experts in the team (or organization). This poster reports an application of this interview technique in a project on the reasoning and knowledge of expert weather forecasters.

3. Application Domain Meteorology and Oceanography Training Facility, Pensacola Naval Air Station Pool of forecasters spanning Apprentice- Journeyman-Expert-Senior Expert levels of proficiency

7. Domain-appropriate Proficiency Scale

8. Knowledge Modeling Utilized the following methods: –Structured Interviews –Protocol Analysis –Knowledge Audit –Critical Decision Method –Concept-Mapping

11. The Procedure Phase 1 - Choice among “bogus” models Phase 2 - Refinement Phase 3 - Guessing Game Phase 4 - Verification via direct observations

12. Base Model of Expertise

13. “Bogus” Models used in the CMP

14. Phase 1-2 Results The strategy of providing the Participant with the "Bogus model" guidance in crafting a representation of their own reasoning strategy seems to have been successful. Each Participant balked at the bogus models, but then went on to craft a model that they felt comfortable with.

15. Example Models from Phase 1, 2

19. Phase 3 Results Most participants adopted a "divide-and-conquer" strategy of first trying to identify models of senior experts or forecasters with whom they were more familiar, and identifying last the models that they thought were bogus and the models they thought were those of Apprentices. Of all of the identification judgments (N = 60), 13 or 22 percent were correct identifications.  Expert models were sometimes incorrectly identified as being models of Apprentices and bogus models.  Participants found the task to be an interesting challenge.  Participants' comments during Phase 3 were revealing of the extent to which they have opportunities to become familiar with one another's strategies.  Participants may have opportunities to see what one another does, but do not share much information about their actual strategies for data search, mental model formation and hypothesis testing.

20. Phase 4 Observation of forecaster behavior when they first came on watch after a period of days when they had not been on the watchbill. Allowed probe questions:  Understanding of the current weather situation? (e.g., "Is what you're seeing fit with persistence?" "Are the models agreeing?")  Skywatching (e.g. " Did you look at the Weather Channel before you came in?")  What are you going to do now? (e.g., "Are you going to look at the models?)

21. Example Phase 4 Results

27. Conclusions Results from the procedure included models of the reasoning of seven forecasters, affirmed in a second phase, and then (for five of them), affirmed with modifications based on observations of actual forecasting behavior. For the total task time of 364 minutes, the procedure would average out to 52 minutes task time to develop a model. This figure is without doubt considerably less that than time takes in traditional experiments (i.e., think aloud problem solving with protocol analysis) to reveal and verify reasoning models. The CMP holds promise as a "fast track into the black box," allowing the development of reasoning models and the testing of hypotheses concerning reasoning models in less time than taken by traditional experimentation.

28. References Hoffman, R. R., & Markman, A. M. (Eds.) (in press). Human factors in the interpretation of remote sensing imagery. NY: Lewis. Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). A case study in cognitive task analysis methodology: The Critical Decision Method for the elicitation of expert knowledge. Human Factors, 40, 254-276. Hoffman, R. R., Shadbolt, N., Burton, A. M., & Klein, G. A. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129-158. Hoffman, R. R., Detweiler, M. A., Lipton, K., & Conway, J. A. (1993). Considerations in the use of color in meteorological displays. Weather and Forecasting, 8, 505-518. Hoffman, R. R. (1991). Human factors psychology in the support of forecasting: The design of advanced meteorological workstations. Weather and Forecasting, 6, 98-110.