1 JUP Conference, 10-11 Jan 2002 Hosted by Princeton University Quantitative Experimental Results: Automation to Support Time- Critical Replanning Decisions.

Slides:



Advertisements
Similar presentations
Victorian Curriculum and Assessment Authority
Advertisements

Modeling Conflicts of Multiple Independent Alerting Systems Lixia Song James K. Kuchar Massachusetts Institute of Technology.
EXPERIMENTAL DESIGN STATEMENT OF PROBLEM Not a yes or no answered question Problem should be clearly testable and specific to your investigation.
Jennifer Siegel. Statistical background Z-Test T-Test Anovas.
The General Linear Model Or, What the Hell’s Going on During Estimation?
Linear Programming?!?! Sec Linear Programming In management science, it is often required to maximize or minimize a linear function called an objective.
Introduction to Factorial ANOVA Designs
Abstract Background Results Olivia Augustin, Lyddia Petrofsky, Kathryn Lyman, & Brittany Weiler Faculty Advisor: Dr. Don Bredle Department of Kinesiology,
Resource Management of Highly Configurable Tasks April 26, 2004 Jeffery P. HansenSourav Ghosh Raj RajkumarJohn P. Lehoczky Carnegie Mellon University.
Quantifying Generalization from Trial-by-Trial Behavior in Reaching Movement Dan Liu Natural Computation Group Cognitive Science Department, UCSD March,
Laboratory Overview MIT Humans and Automation Lab February 2006.
Report on Intrusion Detection and Data Fusion By Ganesh Godavari.
PROXY FOR CONNECTIVITY We consider the k shortest edge disjoint paths between a pair of nodes and define a hyperlink, whose ‘connectivity’ is defined as:
Mary (Missy) Cummings Humans & Automation Lab
Multirobot Coordination in USAR Katia Sycara The Robotics Institute
SQM - 1DCS - ANULECTURE Software Quality Management Software Quality Management Processes V & V of Critical Software & Systems Ian Hirst.
RESEARCH STATISTICS Jobayer Hossain Larry Holmes, Jr November 6, 2008 Examining Relationship of Variables.
Classification and Prediction: Regression Analysis
Tianfeng Chai 1,2, Alice Crawford 1,2, Barbara Stunder 1, Roland Draxler 1, Michael J. Pavolonis 3, Ariel Stein 1 1.NOAA Air Resources Laboratory, College.
Marshall University School of Medicine Department of Biochemistry and Microbiology BMS 617 Lecture 12: Multiple and Logistic Regression Marshall University.
Example of Simple and Multiple Regression
Chapter 5 Product Specifications. Learning Objectives How to translate subjective customer needs into precise target specs? How could the team resolve.
1 Security Risk Analysis of Computer Networks: Techniques and Challenges Anoop Singhal Computer Security Division National Institute of Standards and Technology.
Issues in Experimental Design Reliability and ‘Error’
Integration of AIRLINE Online into Syllabus
1 “Experimental Study of Automation to Support Time-Critical Replanning Decisions” Kip E. Johnson, MIT Dept. of Aero & Astro Engineering Advisors Jim Kuchar.
Statistics for Education Research Lecture 5 Tests on Two Means: Two Independent Samples Independent-Sample t Tests Instructor: Dr. Tung-hsien He
SynGenics Corporation 72 E Granville Road Worthington OH ©2003 SynGenics Corporation. All rights reserved. The Use.
Chapter 8 Experimental Design: Dependent Groups and Mixed Groups Designs.
PHYSICS WORKSHOP Demystifying 9188/4 Yours truly T.V Madziva or
MODELING SIMPLE AEROSPACE DYNAMICS USING CBDS Elijah Johnson, COMP 522, 12/15/2008.
Systems Engineering for the Transportation Critical Infrastructure The Development of a Methodology and Mathematical Model for Assessing the Impacts of.
L 1 Chapter 12 Correlational Designs EDUC 640 Dr. William M. Bauer.
Statistics for clinicians Biostatistics course by Kevin E. Kip, Ph.D., FAHA Professor and Executive Director, Research Center University of South Florida,
ANTs PI Meeting, Nov. 29, 2000W. Zhang, Washington University1 Flexible Methods for Multi-agent distributed resource Allocation by Exploiting Phase Transitions.
Report on Intrusion Detection and Data Fusion By Ganesh Godavari.
Optimal Feedback Control of the Magneto-hydrodynamic Generator for a Hypersonic Vehicle Nilesh V. Kulkarni Advisors: Prof. Minh Q. Phan Dartmouth College.
Correlation Analysis. A measure of association between two or more numerical variables. For examples height & weight relationship price and demand relationship.
Highway Risk Mitigation through Systems Engineering.
Linear Programming with Excel Solver.  Use Excel’s Solver as a tool to assist the decision maker in identifying the optimal solution for a business decision.
Polymorphous Computing Architectures Run-time Environment And Design Application for Polymorphous Technology Verification & Validation (READAPT V&V) Lockheed.
For ABA Importance of Individual Subjects Enables applied behavior analysts to discover and refine effective interventions for socially significant behaviors.
Introduction to Correlation and Regression Introduction to Correlation and Regression Rahul Jain.
Regression Lesson 11. The General Linear Model n Relationship b/n predictor & outcome variables form straight line l Correlation, regression, t-tests,
Military Rank And Obesity: A Cross-Sectional Study of Military Dependents Cared for at MAMC Objective This study examines BMI data of military spouses.
AMSc Research Methods Research approach IV: Experimental [1] Jane Reid
Notes. Science is… The gathering of information by methods including observation and experimentation Systematic Repeatable Testable.
Advanced Statistical Methods: Continuous Variables REVIEW Dr. Irina Tomescu-Dubrow.
Highway Risk Mitigation through Systems Engineering.
1 JUP Conference, Jan 2002 Hosted by Princeton University Experimental Study of Automation to Support Time- Critical Replanning Decisions Kip Johnson,
ALI SALMAN1 LECTURE - 05 ASST PROF. ENGR ALI SALMAN ceme.nust.edu.pk DEPARTMENT OF ENGINEERING MANAGEMENT COLLEGE OF E & ME, NUST DEPARTMENT.
MGT 301 Chapter 1: Introduction to Human Resource Management FEIHAN AHSAN BRAC University Sep 15th, 2013.
SME Loan Decision Making Process: A Declining Role of Human Capital NATIONAL CHENG KUNG UNIVERSITY – INSTITUTE OF CREATIVE INDUSTRY DESIGN Ottavia, Shao-Chi.
Chapter 11 REGRESSION Multiple Regression  Uses  Explanation  Prediction.
Algebra I Exponential Functions: The Marvel of Medicine Irina Keith.
Deploying an Intelligent Pairing Assistant for Air Operation Centers Jeremy Ludwig, Ph.D. June 21, Distribution A: Approved for public release.
Correlation and Linear Regression
Management & Planning Tools
UNIT – V BUSINESS ANALYTICS
Using Generalizability Theory to Examine the Air Reserve Component Augmentation (ARC) Request Process The 4th Untested Ideas (UI) International Research.
Two Way ANOVAs Factorial Designs.
Management of Multiple Dynamic Human Supervisory Control Tasks
DMAIC Analyze, Improve, Control
Scientific Method Steps
One way ANOVA One way Analysis of Variance (ANOVA) is used to test the significance difference of mean of one dependent variable across more than two.
Knowing When to Stop: An Examination of Methods to Minimize the False Negative Risk of Automated Abort Triggers RAM XI Training Summit October 2018 Patrick.
Analysis of Variance: repeated measures
Experimental Evaluation
Chapter 10 Content Analysis
Establishing a Framework for Intelligence Education and Training
Presentation transcript:

1 JUP Conference, Jan 2002 Hosted by Princeton University Quantitative Experimental Results: Automation to Support Time- Critical Replanning Decisions Kip Johnson, MIT Dept. of Aero/Astro Eng Advisor Jim Kuchar, along with Dr. Charles Oman Sponsored by Office of Naval Research Disclaimer: The views expressed in this presentation are those of the authors and do not reflect the official policy or position of the U.S. Air Force, Department of Defense, or the U.S. Government.

2 Research Context Time-Critical Decision-Making ie. Combat flight route planning Aviation, medicine, chemical and energy production, finances Complex Problem Unstructured aspects Multiple competing interests and goals Time pressures Automation Integration Cannot “see” everything (sensor limitations) Human may not understand basis for automated decisions

3 Experimental Goal Discover the relationship between time pressures, automation assistance, and the resulting decision performance, both quantitatively and subjectively. Decision Making Time Pressure Automation Assistance Problem Complexity Human Solution Quality Rating Quantitative & Subjective Score

4 Replanning Task Description Information Integration Human Auto Solution Hazards Time to Target Fuel Constraint Environment: 1 st Order Effects

5 Experimental Protocol 4. Subject Modifies Flight Route Under Time Pressure  Minimize Threat Exposure and Time to Target Deviation  Meet Time to Target and Fuel Constraints 1. View Preplanned Mission Finish Start Target Rendezvous Hazards 2. Change in Environment Hazard, Time to Target, or Fuel Update 3. Route Suggested with Varying Automation Assistance (BLUE) New Hazard!

6 Independent Variables Automation Categories None: No Automation  original route remains Time/Fuel: Constraint Information Only  meets time to target & fuel constraints Hazard: Hazard Information Only  avoids/minimizes hazard exposure Full: Integration of Constraint + Hazard Information  minimizes hazard exposure + meets constraints

7 Independent Variables (2) Time Pressure 20, 28, 40, 55 seconds (logarithmic) –capture performance change Unlimited time to find individual’s optimal performance

8 Dependent Variables 1. Quantitative human performance measured by route cost at end of time pressure. Cost = Hazard Exposure (linear) + Time to Target Deviation (exponential) Fuel = constraint 2. Subjective evaluations.

9 14 subjects: students, ave age = 25, 3 pilots, 3 females 3 hrs: 2 hr training, 1 hr data collection Test Matrix: Greco-Latin Square Design 4 base maps (M), each rotated for 16 effective scenarios Test Conditions Time Pressure Auto Category

10  Repeated Measures ANOVA  Full auto assistance is sig. best  Hazard auto assistance is sig. better than none Automation Effects Quantitative Analysis Increasing Performance

11 Repeated Measures ANOVA No sig. performance improvement after  28 sec Quantitative Analysis (2) Time Pressure Effects Increasing Performance

12 Quantitative Analysis (3) Only none had sig. improvement with time Hazard better than none and time/fuel < 55 sec, sig. at 40 sec None outperforms hazard and time/fuel at 55 sec, while is the worst at 20 sec Performance decreases at times dependent on auto level Interaction Effects optimal score Increasing Performance

13 Idea of “Characteristic Time” (CT) –Period of beneficial time from having auto assistance Possible linear metric –Similar slopes –Intercept quantifies relative auto benefit CT Full > 55 sec CT Hazard  55 sec CT Time/Fuel  20 sec Temporal Benefit of Automation Increasing Auto Benefit

14 Failure Analysis Most failures with hazard –Contrary to quantitative performance 40 sec had fewest failures –While no quantitative improvement in performance after 28 sec Failures at 55 sec  20 sec A failed scenario had  1 of the following: 1. Hit a brown hazard 2. Arrived target outside of time window 3. Not enough fuel to complete mission Only 1 subject was perfect 14.3% failure rate, 32 of 224

15 Conclusions Automation does assist in time-critical decision making –Auto benefit decreases as available time increases –Auto benefit increases dependent on type and amount of information integrated by automation Moderate amounts of time may actually hinder performance over less time Subject data supported quantitative analysis

16 Future Work Develop a generalized model for decision support automation Identify information support needs of human Follow-on experiment –More specific to flight environment –Data points > 55 seconds to reach characteristic time –Further test interactions between partial and no automation