Growing the SAVI Paradigm Used BEM suggestion to shorten the title. Don Ward, Aerospace Vehicle Systems Institute (AVSI) Steve Helton, Boeing Research and Technology (BR&T) Safe & Secure Systems & Software Symposium June 14, 2011

2011 Safe & Secure Systems & Software Symposium © AVSI SAVI Progress – 2010 2011 Background SAVI is high-value integration paradigm change Progress made in demonstrating feasibility of approach Proof of Concept Project II (Expanded Proof of Concept) Basic Thrusts Example of Results Next Steps Shadow Projects are newly underway Progress to Date Conclusion: All indications green Need more resources Accepted BEM suggestion to change title of this slide. 6/12/2011 2011 Safe & Secure Systems & Software Symposium © AVSI

The Situation The Situation High-level Design RFP Response System Integration Checks High-level Req’s in RFP Req’s Changes Target Completion PDR CDR Trades Req’s Defined Sys Design Detailed Design Sys Re-Design Sys Development Sys Integration V&V Aero Avionics Systems Aero Avionics Systems Suppliers Suppliers Suppliers COST GROWTH The Situation Typical causes of “cost growth” and “schedule delay” (BEM) Requirement errors – tied to lack of architecture uncertainty often leads to delayed detection of inconsistent interfaces and correction of the inconsistencies until physical and/or logical integration Thus, there is a strong need for automated consistency checking Human error in modeling from different linked models (often with differing sets of data) and using manual review processes at PDR, CDR, … Unexpected results from integration SCHEDULE DELAY 6/6/11 2011 Safe & Secure Systems & Software Symposium © AVSI 3

Systems Are Becoming More Complex Software Base Cost COCOMO II 299M $160 B $7.8 B $290 M $81 M $38 M Slope = 0.17718 Intercept = -338.5 Curve implies SLOC doubles about every 4 years 134M 61M Assumed Affordability Limit 27M 8M The line fit is pegged at 27M SLOC because the projected SLOC sizes for 2010 through 2020 are unaffordable. The COCOMO II estimated costs to develop that much software are in excess of $10B. B777: 4M A330/340: 2M B737: 470K A320: 800K B747: 370K A310: 400K B757, B767: 190K A300FF: 40K Systems Are Becoming More Complex SAVI, once developed and implemented, has the potential (we think a very credible one) to bring down both cost growth and schedule delays – reversing the trends – as systems become more complex. The efficiencies that SAVI can bring may well put us back below this assumed ceiling in cost. There are a number of other factors that might help drive costs below this assumed affordability limit (BEM with mods by DTW). Enhanced ability to reuse system components Greater durability of system/aircraft components. More use of COTS components to handle complex functionality. Better software tools driven by the SAVI paradigm. Automated methods springing from the integrated “single-truth” data. Closely coupled requirements and architectures. Greater use of formal methods to reduce verification and validation effort. Translators to facilitate broader use of multiple integrated tool sets in distributed development environments. Standards promoted by the SAVI single-truth reference model in integrated systems/subsystems that have been built separately A300B: 4..6K INS: 0.8K Airbus data source: J.P. Potocki De Montalk, Computer Software in Civil Aircraft, Sixth Annual Conference on Computer Assurance (COMPASS ’91), Gaithersburg, MD, June 24-27, 1991. Boeing data source: John J. Chilenski. 2009. Private email. 4/9/11 2011 Safe & Secure Systems & Software Symposium © AVSI 4

… and constrained by dated SE methods Silo’ed Organizations “pi” 3.141592653589793 3.14 Written Requirements …and constrained by dated SE methods SAVI is designed to replace (BEM): manual methods and disparate services to support requirements, architecture, design, development, testing, certification/qualification, operational environment development/analysis/performance/enforcement, maintenance, technical refresh-ment, replacement, reuse, and existing standards and guidance, ground support systems, tools and management practices. This slide may be overly simplified; might be considered condescending to the audience. Mismatched Assumptions 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 5

Current tools for managing complexity have issues Indeterminate Change Impact Operational Models System Models Component Models Functional/Behavior Model Performance Model Structural/Component Model Cost Model Safety Model Security Model Reliability Model Maintainability Model Structural Model Mass Production Model Manufacturing (Assembly) Models Modeling Domains Ops/Mission Analysis System Design Algorithm Development Hardware Design Software Design Logistics Support Manufacturing Integration & Test Performance Simulation Engineering Analysis Human System Integration System Architecture Model (Integration Framework) Analysis Models Hardware Models Software Models Verification Models MODEL EXPLOSION Multiple Truths Incompatible Abstractions Current means of managing complexity have issues Other considerations worth mentioning (BEM): Inflexibility. Private (proprietary?) tools, processes, environments. Skill levels, funding. Static documents, unmaintained requirements, unmonitored maintenance/ evolution. Configuration drift. Impact on ‘ilities 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 6

Common issues create common goals and suggest a cooperative solution Integration complexity will continue to increase Individual companies cannot solve it alone Industry cannot afford to solve it multiple times We cannot afford “not” to solve it A coordinated, industry-wide effort is needed to solve this issue. Common issues create common goals and suggest a cooperative solution Omits Government considerations (BEM): U.S. government should better support development of cooperative solutions (partnering with European R&D and others). Global coordination is possible. Regulation should enable use of better, more effective support methodologies. BEM suggested changing the words on the slide to … “a coordinated Government/industry-wide effort” DTW will add verbally, that the term “industry-wide” includes business, government, and academia – “industry-wide” is meant to connote this full partnership, not one or two elements and certainly not restricted to solely U.S. industrial organizations. SAVI has successfully recruited and is continuing to pursue this kind of global, all-inclusive partnership. 4/9/11 2011 Safe & Secure Systems & Software Symposium © AVSI 7

The Aerospace Vehicle Systems Institute AVSI is a global cooperative of aerospace companies, government organizations, and academic institutions Past AVSI projects have covered the breadth of aerospace systems and current research includes projects in the areas of reliability, certification, and virtual integration. The System Architecture Virtual Integration program is an AVSI program addressing virtual integration of systems. The Aerospace Vehicle Systems Institute BEM comment: Asian organizations missing – Why? Others? DTW notes that no part of the global “industry” has been ignored by AVSI (we have entertained enquiries from Toyota, JAXA, Thales, Dassault, Rolls Royce, and many others, but so far they have not joined the Cooperative). This chart merely lists all the organizations who have joined AVSI (and who remain involved – Parker, Moog, Cessna, and Gulfstream Aerospace have been members or associate members – but are no longer affiliated with AVSI) 4/9/11 2011 Safe & Secure Systems & Software Symposium © AVSI 8

Boeing brought the issue to AVSI 2005 2006 2007 2008 2009 2010 AFE 32 & 32S1 AFE 57 AFE 58 AFE 59 Plan AFE 59 Boeing Commercial Aircraft Boeing Research & Technology Boeing Goodrich Honeywell Rockwell Collins Airbus BAE Systems Boeing GE Aerospace Honeywell Lockheed Martin Rockwell Collins FAA DoD Army DoD Navy Airbus BAE Systems Boeing GE Aerospace Lockheed Martin Rockwell Collins FAA DoD Army Airbus BAE Systems (?) Boeing EMBRAER (S1) Goodrich Honeywell (S1) Lockheed Martin (not S1) Rockwell Collins FAA DoD Army NASA (?) CMU/SEI (?) Dassault? Goodrich? Honeywell? NASA? Boeing brought the issue to AVSI Acronyms include: BCA – Boeing Commercial Aircraft BR&T – Boeing Research and Technology AFE – Authority for Expenditure CMU/SEI – Carnegie Mellon University/Software Engineering Institute Question marks indicate uncertainty of participation – did not join till later, but were part of some of the deliberations, or were heavily recruited. CMU/SEI CMU/SEI 6 Labor-Yrs (1.5) 9+ Labor-Yrs (1) 16+ Labor-Yrs (2) Labor-Yrs (TBD ?) 6/5/11 2011 Safe & Secure Systems & Software Symposium © AVSI 9

Architecture-Centric Engineering Virtual Integration and Validation of System Architecture Safety and Reliability MTBF FMEA Hazard analysis Cyber Security Availability Authentication Integrity Confidentiality No repudiation Annotated Architecture Model Auto-generated analytical models Data precision/ accuracy Temporal correctness Confidence Data Quality Resource Consumption Bandwidth CPU time Power consumption Architecture-Centric Engineering BEM questioned “auto-generated analytical models”: DTW response is that AADL automatically generates code used in analysis of the architecture’s characteristics and which allow evaluation of changes made to the architectural elements These “analytical” models are part of SAVI’s architecture-centric structure, though SAVI is also “friendly” (accepting of) more detailed analysis tools like CAD, MATLAB/Simulink, Modelica, and a host of other software analysis tools. “Friendly” means that SAVI is at least “tool-neutral” and accepts results from them and can pass information back to them with appropriate interface software. BEM also suggested this chart points to future possibilities, not current SAVI demonstrated capabilities: DTW – true; this chart is an SEI chart describing their concept of architecture-centric characteristics. These capabilities are meant to be part of the AADL structure. SAVI is one implementation of that kind of structure and it has demonstrated some of these capabilities. We are still working on others. BEM also voiced concern that “validation” in the blue rounded rectangle goes beyond virtual integration.: DTW – again, this chart is an SEI chart describing their concept of architecture-centric characteristics. However, in my opinion the development team has to have in mind how the system will be validated (define metrics, specific performance goals, and the like) during the integration – virtual or physical. Real-time Performance Execution time/Deadline Deadlock/starvation Latency source: SEI 6/12/11 2011 Safe & Secure Systems & Software Symposium © AVSI

What are the Core Elements of SAVI? BEM questioned the applicability of tools and analysis results: DTW – Again, my view of this is that SAVI does do analysis of the integration effort; it allows evaluation of changes to the architecture – which is an “analysis result”. This analysis may not be to the same level of detail as that done within the individual tools, but it is analysis related to integration and can be done within the current AADL implementation. The Use Case results show that. BEM also noted that the following are missing from this chart: Architectures Libraries Implemented tools Automated management methods Environment translations Standards Application tailoring Certification/qualification, design assurance, … policy and guidance DTW – I do not think any of these are part of the “core elements” of SAVI, though they are important to the future evolution of the paradigm. These are ultimately “deliverables” (what SAVI should do) for the most part. Dave Redman’s notes: “model bus and data exchange layer (DTW) enable the “single-truth” (DTW) reference model – interfaces also exist to the link model, but not the consistency afforded by a reference model.” 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 11

Virtual Integration for Development Cycle Sensitivity analysis for uncertainty Confidence in implementation Requirements Engineering Top-Level Verification Items Acceptance Test From Prediction to Validation System Design System Test High-level AADL Model Model-driven artifact generation Conformance of models and systems Software Architectural Design Integration Test Detailed AADL Model Virtual Integration for Development Cycle BEM noted that this “vee” chart is for the development life cycle, not the system life cycle: Omits calibration of models against the built aircraft/systems and the reasons for doing so. Omits SAVI growth through continuing development after the SAVI VIP is completed. Omits the benefits of recurring applications of the SAVI paradigm (models mature and can be reused). Component Software Design Unit Test Specify Model- Code Interfaces → generation of test cases ← updating models with actual data Code Development 6/5/11 2011 Safe & Secure Systems & Software Symposium © AVSI 12

Global Team Implementing SAVI Boeing Rockwell Collins Goodrich FAA BAE Systems SEI Airbus Honeywell Worldwide PoC Model Development Lockheed-Martin Subversion Model Repository at AVSI EMBRAER US Army Global Team Implementing SAVI BEM asked: What is the subversion model repository at AVSI? DTW – It is a standard CVS subversion model, I think. Asking Fred. . A distributed, multi-party development team implemented the PoC demo, reflecting current real-world development environments 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 13 13

What Has SAVI Been Doing? Strengthening the Proof of Concept by: Building and Exercising Critical Use Cases “Fit” Use Case Reliability Assessment Use Case Safety Analysis Use Case Behavior Use Case About 10-15% of Total Number of Use Cases Postulated Improving RoI Estimate Iterating to reset the Integrated Program Plan to fit current economic climate What Has SAVI Been Doing? BEM asked: How much of these Use Case developments will be developed to the point of being applicable to real world projects? DTW – All of these and all others that are deemed necessary to reach a TRL of 9 for SAVI Version 3.0. BEM observed: RoI is based solely on “find and fix errors” (requirements based): Hardware estimates are based on software data. Innovative methods/possibilities are identified with no impact on RoI value. (DTW – Why is this impact needed? Current estimates are very positive and wholly support the SAVI planned development without these intangible benefits. Taking this approach is admittedly and on purpose conservative – yet our RoI is not credible in the eyes of SAVI detractors. So how is adding these hard-to-quantify innovations going to add credibility?) Data taken from older, smaller, less complex aircraft developments and financial estimate tools (DTW – true, the data trend line is the basis for doing what we are doing AND it has been verbally corroborated by both major OEMs as accurate for newer projects on which they cannot yet publish the data.) BEM asked: Why are we beginning again to develop an integrated program plan? DTW – I changed the wording to indicate what this is: an iteration to change the previous plan to fit the roadmap that was revised in Seattle at the end of AFE 59. This change is also in response to redirection by the AVSI EB in June 2010. 6/6/11 2011 Safe & Secure Systems & Software Symposium © AVSI 14

Mechatronics Architecture PoC II PoC I Software Functions States Modes … Libraries Modules Messages Protocols Code … Mechanical Systems Kinematics Dynamics Powertrain Thermal Fluids Electric Power … Electronics Interface Displays User Controls Haptics Remote Links … Electronic Control Unit (ECU) Sensors Actuators Mechatronics Architecture BEM asked: Is vibration covered under dynamics? DTW – Structural vibrations are covered under Dynamics on this slide; the Behavior Use Case addresses both dynamical and steady state loadings of the structure. This Use Case also addresses fluid analyses (the hydraulic servomechanical systems and the kinematics of the mechanical components of the control linkages. Weight? - DTW: Mass properties (including weight) was covered and included in the AFE 58 AADL model with extended properties; these properties were also included in the all four Use Cases exercised under AFE 59 Spatial (location, volume relationships)? – DTW: The Fit Use Case is specifically aimed at capturing information from CAD and similar software tools used to describe mechanical design of physical components. The emphasis in this Use Case was on transferring between these kinds of tools and the AADL model these kinds of dimensional and properties. The Use Case does not fit all tools of this type but it did work with a representative set from Rockwell Collins. Aerodynamics? – DTW: The Behavior Use Case utilized crude, but representative aerodynamic models to represent structural loads on a wing-like structure. Like the Fit Use Case, the aerodynamic models used have not been demonstrated for all aerodynamic models, only a representative one. Materials? - DTW: Again, the Behavior Use Case used a finite element model (a fairly simplistic one granted) that does carry materials properties. Moreover, the mass properties elements added to AADL under AFE 58 considered some of the materials properties. Communications Bus 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 15

Expanded Proof of Concept Demonstration Objectives Results Model Data Exchange Layer and Repository Development Collected and prioritized use cases Expand the PoC demo to exercise these, including mechatronic systems Address Questions With Current SAVI Approach Investigating multi-language-model approaches to the Model Repository Developed initial SAVI requirements Improve ROI Analysis Added statistical estimation Showed that RoI estimates are favorable for both Suppliers and System Integrators Expanded Proof of Concept Demonstration BEM asked: Will the Use Cases be refined to come up to level of real world applications? DTW – The shadow projects proposed for AFE 59 Supplement 1 (2011-2012 work) are the first step in this direction (See the revised Roadmap chart). We do have a lot of work to do to completely fill in the Use Case matrix. BEM commented: While the RoI has been evaluated for benefit to suppliers and integrators, this has not resulted in participation costs based on these estimated benefits. Also, not participation of small users or other domains is included. BEM criticized suggesting that we had any kind of Integrated Program Plan. DTW – In my view the original work done at the beginning of AFE 58 (even within AFE 57 and earlier) qualifies as an Integrated Program Plan. We have draft documents laying out an initial program plan (done by BEM) as well as the work done under AFE 58 for the “to be” roadmap that spell out what we intended to do. Now we are trying to bring that up to date with the incremental development approach preferred by the EB and coincident with the reality of commitment of organizations and resources. It is not complete, but it has been started. It is inadequate, but it is not wrong to discuss revisions to those original efforts. The revision is very significant, but it is still a revision – may be even a redirection. Develop SAVI 1.0 Program Plan Initial SAVI Integrated Program Plan is being revised Outreach Expanded interaction with other efforts 6/12/11 2011 Safe & Secure Systems & Software Symposium © AVSI 16

2011 Safe & Secure Systems & Software Symposium © AVSI Define SAVI Use Cases Use cases reflect modes of interaction in SAVI framework Identify initial high-level requirements Help identify technology gaps Use cases used to exercise PoC models Define SAVI Use Cases BEM suggested adding: Identify the tool/process paradigm improvements to be sought (for example, automated management support) specify the SAVI paradigm implementation and follow-on possibilities R&D Tool development Process and standards development Application tailoring Regulatory acceptance – policy and guidance DTW : While I agree that these are subjects to be expanded upon in our overall plan, I do not see adding them here. This is a slide depicting the Use Cases we chose in Toulouse in March 2010 as the priority Use Cases to be addressed in AFE 59. It is not a slide depicting the end game for SAVI and any follow-on projects. 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 17

Mechatronics: Physical System Modeling Applied Load (from MatLab model) Wing Structure Response Mechatronics: Physical System Modeling BEM commented: Great deal of work to be done to implement the mechatronic modeling to meet real world application requirements. DTW – True, but we did a pretty fair job of showing that it could be done and that is what the goal was – to demonstrate the concept is feasible. We are making no claim that it is a finished product. Structural Finite Element Model Mechatronic Actuator Model Architectural model captures and integrates behavior of virtual subsystems 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 18

AADL Error Annex Drives Reliability and Safety Analyses AADL and Error Model Annex Standard For both reliability & safety modeling Assess system reliability and safety from same annotated architecture model Focus application on embedded software system (IMA) Reliability Use Case MTTF for dual redundant flight guidance (FG) and auto pilot (AP) Different deployment configurations on dual and triple redundant HW Consider perfect and imperfect functional & fault management SW Safety Use Case Functional hazard assessment of FG and AP Failure mode and effects analysis (FMEA) for CPU, FG and AP failures AADL Error Annex Drives Reliability & Safety Analyses BEM commented: Impact analysis for tradeoffs (e.g. Safety vs. Security vs. Cost vs. Schedule) DTW –I am not absolutely sure what this comment means, let alone how much of the ability to do this impact analysis was demonstrated by Peter and his team at SEI. 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 19

Safety Analysis Use Case Tier 2 Flight Guidance IMA Architecture Computer Hardware View Embedded Software View Safety Analysis Use Case 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 20

Use Case Demonstrations SAFETY USE CASE DEMO FEM Plug-in demo Use Case Demonstration This is the “placeholder” for the video file to be shown that more completely illustrates what was done in each of the Use Cases. 6/12/11 2011 Safe & Secure Systems & Software Symposium © AVSI 21

Overall average deviation Where We Are Now PoC feasibility is stronger after EPoCD Phase 1. Supporting plans more credible (EPoCD final report) RoI estimates still valid Use Case structure on solid footing. Structured approach to demonstrating details Exercising Use Cases clearly demonstrated Relatively small number exercised AADL is growing (new Annexes, interfaces feasible) SAVI is still looking for critical mass of Number of participants Right skill sets for participants Average RoI for ten Monte Carlo runs Overall average deviation 78.09% 98.33% 115.88% 0.81% 1.05% 1.73% Where We Are Now BEM comments: We are now ready to start planning and implementing the SAVI Virtual Integration Process (VIP). There is opportunity for new participants to get in on the ground floor, affect implementation decisions, grow the skills related to the use of the SAVI paradigm, and get your requirements considered in the development process. All of the benefits of participation and sharing the cost with all other participating members and the results of the development process (hit the ground running with the SAVI paradigm methods of the future) 6/12/11 2011 Safe & Secure Systems & Software Symposium © AVSI 22

2011 Safe & Secure Systems & Software Symposium © AVSI What Is Next? EPoCD Phase 2 focused on “Shadow” projects Parallel SAVI integrations with “real” projects Subsystem level – Goodrich DVMS System level – AMRDEC/Rockwell CH-47 Upgrade Prepare for SAVI 1.0 Multi-language reevaluation (SysML/AADL/?) Priority Set of Use Cases to demonstrate Exercise as many priority Use Cases as possible Look to more realistic projects Integrated Program Plan Detail the SAVI Integrated Program Plan Refine incremental development plans – SAVI 2.0, 3.0 What Is Next? BEM suggested: Remove “Continue to pursue critical mass” and add: Detail the SAVI Implementation Program Plan. Open the SAVI Program to stakeholders, tool vendors, and R&D efforts Consolidate participant needs into this SAVI implementation DTW: I agree with the first bullet and have made that change; I prefer to keep the last two bullets as points to be made verbally. The SAVI program has always and continues to be open to stakeholders, tool vendors, and R&D efforts – this is not a new idea just for the next phase (AFE 59 Supplement 1). I think the last bullet suggestions is better captured in my last bullet. 6/6/11 2011 Safe & Secure Systems & Software Symposium © AVSI 23

2011 Safe & Secure Systems & Software Symposium © AVSI Questions? Contacts: Dr. Don Ward Phone: (254) 842-5021 Mobile: (903) 818-3381 dward@avsi.aero Dr. Dave Redman Office: (979) 862-2316 Mobile: (979) 218-2272 dredman@avsi.aero 3/23/11 2011 Safe & Secure Systems & Software Symposium © AVSI 24

Current Modified Schedule 6/12/11 2011 Safe & Secure Systems & Software Symposium © AVSI 25

Overall average deviation RoI Results Summary RoI results for one major aircraft project with SAVI Annual RoI based on finding 30%, 40%, or 50% of software defects early and correcting them As low as 10% find/fix rate still gives 4% annual RoI Supplier expectations depend on involvement in integration Average RoI for ten Monte Carlo runs Overall average deviation 78.09% 98.33% 115.88% 0.81% 1.05% 1.73% 6/12/11 2011 Safe & Secure Systems & Software Symposium © AVSI 26