Presented to: By: Date: Federal Aviation Administration OPERATIONAL SUITABILITY EVALUATIONS Weather-in-the-Cockpit Workshop Robert Ruiz, AFS-430 August 9, 2006

2 Federal Aviation Administration Operational Suitability Evaluation August 9, Introduction Background Activities Objectives Study Approach Evaluation

3 Federal Aviation Administration Operational Suitability Evaluation August 9, Background WX Products emerge from research as technically sound products – Good Science Operational implementation (AFS definition) historically not part of the R&D effort Lack of operational suitability evaluation on newly developed graphical products Some products issued as supplementary with restrictions CIP/FIP 2005 Hazard analysis highlighted safety issues and the need for a structured Safety Assessment of WX products

4 Federal Aviation Administration Operational Suitability Evaluation August 9, FY-06 Implementation Activities AFS-400 able to secure funding to develop and implement an operational suitability evaluation FAA Tech Center Weather Sensors Group was selected to conduct the evaluation

5 Federal Aviation Administration Operational Suitability Evaluation August 9, Definitions – HBAT and AIM Primary Weather Product. –An aviation weather product that meets all the regulatory requirements and safety needs for use in making flight-related, aviation weather decisions. Supplementary Weather Product. –An aviation weather product that may be used for enhanced situational awareness. If used, a supplementary weather product must only be used in conjunction with one or more primary weather products. In addition, the FAA may further restrict the use of supplementary weather products through limitations described in the product label to conduct the evaluation

6 Federal Aviation Administration Operational Suitability Evaluation August 9, Objectives – Operational Suitability Evaluation Determine pilot understanding/intuitiveness of the product Determine that the information presented is not hazardous misleading information Determine the level of safety of the product; provide an equivalent level of safety (Supplementary-Situational Awareness) Determine the level of training required for product use Determine if any training and guidance materials will be needed Provide a roadmap for AVS to move the product from Supplementary to Primary

7 Federal Aviation Administration Operational Suitability Evaluation August 9, Study Outline Study 1: Supplementary Weather Product –Assess CIP operational suitability for pilot use as Supplementary Product with no restrictions –Assess operational suitability using: Situational Awareness CIP Supplementary Weather Product Questionnaire Interview questions Observer ratings Study 2: Primary Weather Product –Identify CIP operational suitability issues for pilot use as Primary Product –Identify issues using: Situational Awareness CIP Primary Weather Product Questionnaire Interview questions Observer ratings

8 Federal Aviation Administration Operational Suitability Evaluation August 9, Overview Evaluate CIP Severity and Probability products Evaluate from pilot perspective Use in flight planning scenarios –Preflight –In-flight* *Note: While “In-Flight” use is being examined, the evaluation will not yield data appropriate for cockpit use.

9 Federal Aviation Administration Operational Suitability Evaluation August 9, Approach Simulate route planning with focus on in-flight icing –Preflight –In-flight Active pilot participants (n=32/48) –General Aviation (Expert and Novice) –Business Jet –Regional Airline

10 Federal Aviation Administration Operational Suitability Evaluation August 9, Approach Flight Planning Scenarios –Archived weather examined to determine flight routes –Flight routes developed so that icing is encountered –Eight GA Routes –Four FAR Part 121 (SkyWest) Routes –Flight route information includes: Departure, Destination, Waypoints ETD ETE Preferred Flight Level Distance Equipage

11 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Conduct Experimental Conditions –Study 1: CIP Supplementary Weather Product AIRMET/SIGMETs (No CIP) AIRMET/SIGMETs (With CIP) –Study 2: CIP Primary Weather Product CIP (No AIRMET/SIGMETs) CIP (With AIRMET/SIGMETs)

12 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Conduct Experimental Conditions –Pilots Expert Novice –Equipage De/anti-icing equipment No de/anti-icing equipment –Icing Severity Trace/Light Moderate or Greater (MOGR)

13 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Conduct Pretest Briefing –Objectives of evaluation –Training CIP information and use Evaluation platform Evaluation data collection tools

14 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Conduct Preflight –Assigned preferred route Trace/Light Icing Moderate or Greater (MOGR) –Standard weather briefing –Depending on experimental condition, provide icing products for Supplementary Study or Primary Study –Pilot decides route/altitude adjustments/no adjustments –Interview questions –Assess Situational Awareness Pilot CFI

15 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Conduct In-flight –Assigned preferred route Trace/Light Icing Moderate or Greater (MOGR) –Perform preflight self brief –Pilot is asked to imagine he/she is about 90 minutes into flight –In-flight weather update provided –Depending on experimental condition, provide icing products for Supplementary Study or Primary Study –Pilot decides route/altitude adjustment/no adjustment –Interview questions –Assess Situation Awareness Pilot CFI

16 Federal Aviation Administration Operational Suitability Evaluation August 9, Data Elements Situation Awareness (SA) rating –Pilot –Observer (CFI) Operational suitability questionnaire responses Interview comments Potential pilot errors/poor decisions General observations

17 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Criteria: Supplementary Statistically significant increase in SA when CIP is used with AIRMETs/SIGMETs Evaluation Criteria: Primary No decline in SA when CIP is used without AIRMET/SIGMETs

18 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Results (Expected) Supplementary Weather Product Evaluation –Operational suitability confirmation data Data to support or not support CIP use as supplementary product without restrictions Primary Weather Product Evaluation –Identify operational suitability issues Issues to be resolved in transitioning CIP to Primary Product Interview responses Pilot Error Situation Awareness Operational Suitability Comments

19 Federal Aviation Administration Operational Suitability Evaluation August 9, Evaluation Timeline Data collection (7/21 – 9/05) Evaluation Reports –Quick Look (9/25/2006); NBAA/FPAW –Final Report (11/03/2006) CIP-Severity and Probability Operational Dec 06.

20 Federal Aviation Administration Operational Suitability Evaluation August 9, Acknowledgement FAA Tech Center Weather Sensors Group in providing assistance in developing some of the slides that were used in the presentation Contact Information Robert M. Ruiz AFS