Building an Aviation Corpus Conclusions & Future Work BUILDING AN LSA-SIMULATED REPRESENTATION OF PILOT AVIATION KNOWLEDGE Accelerating Development of NextGen with Affordable Task Analysis Tools Marilyn Blackmon (Ph.D.), Peter Polson (Ph.D.), Lance Sherry (Ph.D.), & Michael Feary (Ph.D.) University of Colorado, George Mason University, NASA – Ames Research Center Accurately predict probability-of-failure-to-complete for both routine and infrequently performed, safety-critical tasks; also predict depth/familiarity of underlying concepts Test whether infrequently performed, safety-critical tasks will be semantically salient and intuitive to prevent errors Identify usability problems in a design early in the design cycle Can also be used to evaluate training designs Context Research Approach LSA-simulated Pilot Aviation Knowledge Problem Statement Affordable Operator Performance Models require a representation of the aviation knowledge of pilots — an Aviation Corpus — to evaluate proposed systems being incorporated into NextGen automation An Aviation Corpus is a collection of all the texts that pilots have read about aviation at a particular level of pilot expertise. It can be “read” by Latent Semantic Analysis (LSA) or other language models (e.g., PMI-IR) Requirements: – –Combined with the performance model, the corpus can be used to simulate pilots doing any specific task, predicting whether pilots will do the task correctly, rapidly, and safely or be prone to errors – –The simulation of pilot aviation knowledge can be incorporated into other task analysis tools (ADEPT, HCIPA, CogTool, SNIF-ACT) Latent Semantic Analysis (LSA) “reads” the Aviation Corpus — text files of all key documents pilots have read — and builds a valid computer-simulated representation of pilots’ aviation knowledge — expert aviation knowledge pilots gained from reading these texts LSA is a well-established computational model of language that can represent any knowledge domain LSA provides automated, objective measures of semantic similarity, word frequency, and concept depth/familiarity – –Blackmon, Polson, et al. (2005, 2007) developed an automated usability evaluation tool called CWW that synthesizes multiple measures — semantic similarity, concept familiarity, attention — to accurately predict task performance of a particular user group – –CWW is especially accurate at identifying and repairing serious usability problems — where probability-of-failure-to-complete a task is 25% to 75% or more Generalize lessons learned from CWW research to evaluating usability of NextGen automation By analogy to CWW, we will use the LSA-simulated pilot knowledge to make accurate predictions synthesizing three measures: Will the pilot focus on the correct subregion of the display? Will the correct action be the most salient option, the one most semantically similar to the goal and thus most apt to be chosen? Will selection of the correct action be less likely due to conceptual unfamiliarity (insufficient knowledge or infrequent use)? Join computational model that represents pilots’ aviation knowledge with proven tool called CWW Figure at left: Blackmon et al. (2007) verified that using the CWW tool with LSA accurately predicts which tasks will be easy, medium, and hard and successfully repairs tasks that are hard. The Aviation Knowledge Corpus, once built, will have many potential uses for a wide variety of aviation research projects LSA-simulated Pilot Aviation Knowledge