1. Lockheed Martin Aeronautics Company Superior Products Through Innovation Validation & Verification of Intelligent and Adaptive Control Systems (VVIACS) Jim Buffington

2. Lockheed Martin Aeronautics Company 2 Outline Introduction Motivation Approach Assessment Development Evaluation Summary

3. Lockheed Martin Aeronautics Company 3 Team Vince Crum – AFRL - Government PM Jim Buffington – LM Aero - Contractor PM LM Aero – Greg Tallant, Peter Stanfill LM M&FC - Clinton Plaisted, Barry Frazier, Rich Hull, Guy Rowlands LM SS - Prasanta Bose Carnegie Mellon University - Bruce Krogh General Electric Global Research - Tim Johnson, Hunt Sutherland Scientific Systems Company, Inc - Ravi Prasanth, Sanjeev Seereeram

4. Lockheed Martin Aeronautics Company 4 Scope Safety-Critical Flight Systems Military Certification Advanced Controls System Development All Development Phases Process, Tools, and Methods Verification & Validation

5. Lockheed Martin Aeronautics Company 5 Emerging Software Size and Complexity Projected Exponential Increase in SW Size and Complexity Inter-System Communication & Dependencies F-16 IDF YF-22 F-22 JSF CDA UAV - Projected F-35 – Est Block 60 Increasing system integration requirements and Complexities  Advanced system attributes (on-board intelligence and adaptive control laws) will be required to accommodate emerging functional requirements.  This will increase the size and complexity of control systems beyond the capability of current V&V practices.

6. Lockheed Martin Aeronautics Company 6 Emerging System Costs Size and Complexity Increase Costs  SW size and system complexity lead to growth system development and certification costs  Test automation insufficient for emerging control systems

7. Lockheed Martin Aeronautics Company 7 Compressed Development Schedules Emerging Program Requirements Increase Risk Customers requiring 12 to 36 month development schedules Baseline is 48 months to first flight

8. Lockheed Martin Aeronautics Company 8 Purpose GOAL: Enable affordable development of future safety-critical flight systems with prescribed levels of safety and reliability. OBJECTIVE: Study, develop, and demonstrate effective V&V technologies for advanced safety-critical control system flight certification. APPROACH: Use extensive experience-base and diverse team to develop innovative concepts Evaluate concepts in a realistic framework to maximize transition success Classify emerging safety-critical control systems according to fundamental attributes Develop and demonstrate preliminary V&V strategies that focus on critical flight certification schedule and cost points Identify high-payoff V&V process, tool, and method technologies for further development

9. Lockheed Martin Aeronautics Company 9 Tasks  Task 1  Emerging Control System Study  Task 2  Control Characteristics and V&V Needs Study  Task 3  Innovative Flight Certification Strategies Development  Task 4  Proof of Concept  Task 5  Technology Development Planning and Reporting 1. Assessment 2. Development 3. Evaluation

10. Lockheed Martin Aeronautics Company 10  Emerging Control Systems Study  Develop control system project database  Down select to subset of projects for additional analysis  Control Characteristics and V&V Needs Study  Analysis to define emerging fundamental properties  Identification of V&V drivers  System Development Model  System Development impact assessment Tasks 1 & 2 – Assessment V&V NEEDS

11. Lockheed Martin Aeronautics Company 11 Task 1 – Emerging Control Systems DESCRIPTION Integrated Management, Adaptive Control Intelligent Autonomy Polymorphous Computing Architectures Autonomous Control Bio-inspired Multi-vehicle Control Adaptive Failure Management Optimal Trajectory Generation Intelligent Failure Management Model-based Health Management Distributed Multi-vehicle Control ECS PROJECT AIMSAFE / RESTORE ICARUS PCA LOCAAS Swarm Intelligence XACT Software Enabled Control Autonomous Propulsion System Tech Intelligent Engine PHM Distributed Space Systems Characterized 10 Emerging Control System (ECS) projects 6 ECS projects used to form the Single-Vehicle ECS (AIMSAFE/RESTORE, PCA, LOCAAS, XACT, Autonomous Propulsion System, and Intelligent Engine PHM) 5 ECS projects used to form the Multi-Vehicle ECS (ICARUS, PCA, Swarm Intelligence, Software Enabled Control, and Distributed Space Systems)

12. Lockheed Martin Aeronautics Company 12 Task 2 - Emerging Systems Analysis Extensible Analysis Framework based on Fundamental Properties Identified 100+ fundamental properties (FPs) for the ECS projects Categorized FPs in 5 Views: Requirements, System, Algorithm, SW, HW Identified 28 FPs as Emerging Fundamental Properties (EFPs) Identified 4 primary V&V Drivers Difficulty, Complexity, Unfamiliarity, External factors System development model Based on LM Process Multiple levels Contains over 800 tasks MS Project / Excel

13. Lockheed Martin Aeronautics Company 13 Task 2 - Impact Analysis Results Significant Cost/Schedule Increase Projected Due to Complexity Single-Vehicle ECS Increases Development Costs ~ 50%, V&V Costs ~ 100%, and Critical Path Length ~ 50% Multiple-Vehicle ECS Increases Development Costs ~ 100%, V&V Costs ~ 150%, and Critical Path Length ~ 125% Software: Single-Vehicle 100% Increase and Multiple-Vehicle 200% Increase in V&V Costs Test: Single-Vehicle 150% Increase and Multiple-Vehicle 250% Increase in V&V Costs

14. Lockheed Martin Aeronautics Company 14 Task 2 - Pareto Analysis  Single-Vehicle ECS V&V Cost Impact  80%: TEST, SW  90%: 80% + TTD  TEST largest component (41%) followed by SW (28%)  Single-Vehicle ECS V&V Duration Impact  80%: SW, TEST  90%: 80% + TTD  SW dominates at 53% Pareto Analysis Identifies the Critical Functional Disciplines

15. Lockheed Martin Aeronautics Company 15 Task 3 – Flight Certification Strategies Development  Requirements Development  Requirements/Guidelines  V&V Drivers Mapping  Flight Certification Refinement  Current Process Development  Evolution of Process Development for V&V  V&V Methods Development  Identify Current V&V Methods  Recommend Approaches V&V STRATEGY EVOLUTIONS

16. Lockheed Martin Aeronautics Company 16 Near-Term (1-3 yrs) Evolution: System Model-based design now being implemented  Auto-Code  Auto-Test  Rapid Prototyping  System Model-Based  Automated Verification Management  Simulation-Based design Mid-Term (4-6 yrs) Evolution: Formal Foundations in advanced development Formal Requirements Specs Requirements and Traceability Analysis Formal Methods Computer-Aided System Engineering Far-Term (7-9 yrs) Evolution: V&V Awareness throughout  still in research Run-Time V&V Rigorous Analysis for Test Reduction Requirements & Design Abstraction Probabilistic/Statistical Test Testing Metrics Task 3 – Strategy Evolutions

17. Lockheed Martin Aeronautics Company 17 Task 4 – Advanced Technology Impact to System Development  System Development Cost Reductions:  Baseline: 25%  Single-Vehicle: 33%  Multi-Vehicle: 35%  System Development Critical Path Reductions:  Baseline: 12%  Single-Vehicle: 29%  Multi-Vehicle: 30%

18. Lockheed Martin Aeronautics Company 18 Task 5 – Technology Development Planning and Reporting  Risk Waterfall Planning  ROM Development Cost  EFP Coverage Analysis + Task 4 System Development Impact  Net Benefit  Priority of V&V technologies was established on the basis of a cost-benefit analysis performed on each technology  Cost-Benefit Ratio (CBR) = ROM Development Cost/Net Benefit  All near-term technologies except Automated Verification Management were eliminated from the CBR analysis because these technologies are relatively mature (moderate technical risk) and significant industry investment in these technologies is ongoing and is expected to continue.

19. Lockheed Martin Aeronautics Company 19 Task 5 – Technology Cost / Benefit  ROM Cost – Risk Waterfall Analysis  Net Benefit is defined as the product of the technology’s EFP Coverage Benefit and the System Development Benefit  B NET = B EFP B SD  EFP Coverage Benefit is a measure of the technology’s applicability across the set of EFPs (large B EFP implies good coverage)  System Development Benefit is a measure of the technology’s impact on system development cost and schedule

20. Lockheed Martin Aeronautics Company 20 Task 5 – Final Prioritized Technologies Based on Cost-Benefit Analysis 1.Automated Verification Management 2.Formal Requirements Specifications 3.Requirements and Traceability Analysis 4.Formal Methods 5.Probabilistic / Statistical Test 6.Requirements and Design Abstraction 7.Run-Time V&V 8.Testing Metrics 9.Rigorous Analysis for Test Reduction 10.Computer-Aided System Engineering Increasing Cost-Benefit Ratio Near-Term (1-3 years) Technology Mid-Term (4-6 years) Technology Far-Term (7-9 years) Technology EFP weighting: W EFP = 1 (all equal) ECS weighting: 10% Baseline 50% Single-vehicle 40% Multi-vehicle

21. Lockheed Martin Aeronautics Company 21 Summary  Developed emerging control system (ECS) database and populated with an extensive set of past, present, and future ECS projects.  Assessed impact of future systems on current development process and identified “long-pole” functional disciplines with highest adverse impact (i.e., SW, TEST, TTD).  Identified 15 key V&V technologies to address system development impact and characterized in terms of near- (1-3 years), mid- (4-6 years), and far-term (7-9 years) strategy evolutions.  Demonstrated effectiveness of each strategy evolution using the system development model and the system development impact assessment tool (i.e., cumulative 25% reduction in V&V cost and 12% reduction in V&V effort for all strategies).  Developed technology maturation plans for each V&V technology identified, and prioritized the technologies by performing a cost-benefit analysis.