CRESST ONR/NETC Meetings, July July, 2003 ONR Advanced Distributed Learning Greg ChungAllen Munro Bill BewleyQuentin Pizzini Girlie DelacruzUSC/BTL Terry Vendlinski Zenaida Aguirre-Munoz UCLA/CRESST USN EDO Application  2003 Regents of the University of California

CRESST ONR/NETC Meetings, July CRESST/BTL/USN EDO Collaboration Research to develop new kinds of assessment and instructional tools to support USN EDO school Initial domain: Risk management within the integrated case study Tool prototypes: –Knowledge mapping (assessment) –Decision-analysis (instruction)

CRESST ONR/NETC Meetings, July Assessment Focus Knowledge mapping as a way to measure problem solving –Examine students’ understanding of the decision analysis process –Examine the extent to which domain-independent links can be used to measure problem solving processes

CRESST ONR/NETC Meetings, July Context Integrated Case Study (ICS) –Problem-based learning –Teams are given broad requirements –Deliverable is a professional- level briefing Problem solving, teamwork, communication, prior job experience –Concepts covered in instruction are put into practice in ICS

CRESST ONR/NETC Meetings, July Domain Risk Management –Complex — multiple variables to consider, broad requirements, imperfect information, changing conditions –No “correct” answer — trade-offs among cost, schedule, and performance variables –Reasoning and analytical processes are critical to success

CRESST ONR/NETC Meetings, July The Tools Knowledge maps –Students depict their understanding of the decision process underlying the refueling at sea (RAS) “hiccup” –Students’ maps can be compared to an expert’s map for assessment purposes Decision-analysis –Tool permits quantitative or qualitative views of problem solution alternatives –Instruction can be delivered in the context of the tool

CRESST ONR/NETC Meetings, July Decision Analysis Template Options Analysis Selection Domain Variables Environment

CRESST ONR/NETC Meetings, July Risk Management Environment Selection Options Analysis Cost Schedule Performance

CRESST ONR/NETC Meetings, July Risk Management: Refueling at Sea Hiccup Existing system Recompete Full inventory buy Technical data plan buy Full + TDP buy Selection Analysis Cost Schedule Performance No. ships Time constraints Config. Mgt Manufacturing Contracting Supportability T&E Options Environment Requirements Regulations Legal issues

CRESST ONR/NETC Meetings, July Assessment Tools How do we measure an EDO’s problem solving processes within the risk management domain? –Knowledge maps –Online processes and decisions with decision-analysis simulation tool

CRESST ONR/NETC Meetings, July Refueling At Sea (RAS) Scenario

CRESST ONR/NETC Meetings, July Refueling At Sea (RAS) Scenario Options Analyze Select Schedule Cost Performance Environment

CRESST ONR/NETC Meetings, July Refueling At Sea (RAS) Scenario

CRESST ONR/NETC Meetings, July Current Status Instructor consensus knowledge map created for RAS scenario Domain-independent links –Harder for processes (vs. declarative) –Level of abstraction issues Domain-independent: context Domain-dependent: requirement, legal, regulation

CRESST ONR/NETC Meetings, July Next Steps Pilot test at EDO with EDO instructor and EDO class (July/Aug) Evaluate and revise for next class

CRESST ONR/NETC Meetings, July Decision Aiding Tool

CRESST ONR/NETC Meetings, July Decision Making and Risk Management A major aspect of EDO education The Refueling at Sea (RAS) problem –Evaluating options systematically A tool that promotes systematic assessments of risks and remediations A tool that teaches about itself and about risk and decision-making

CRESST ONR/NETC Meetings, July Decision Aiding Tool Demo TDP = Technical Data Package T&E = Test and Evaluation RAS = Refueling at Sea system Decision Thresholds Controller Decision Thresholds Controller User created nodes have arbitrary names. User can set estimated probabilities, utilities. RAS = Refueling at Sea system

CRESST ONR/NETC Meetings, July Decision Aiding Tool Instruction

CRESST ONR/NETC Meetings, July Analyzing the User Application of Chung’s ‘Click- through’ interface.

CRESST ONR/NETC Meetings, July General Application Instructional approach can provide opportunities for students to engage in systems engineering thinking Assessment approach can provide information on what people know and understand about processes